Replace the trie index format

BUCK

@@ -381,8 +381,7 @@ cpp_library_wrapper(name="rocksdb_lib", srcs=[
         "utilities/transactions/write_prepared_txn_db.cc",
         "utilities/transactions/write_unprepared_txn.cc",
         "utilities/transactions/write_unprepared_txn_db.cc",
-        "utilities/trie_index/bitvector.cc",
-        "utilities/trie_index/louds_trie.cc",
+        "utilities/trie_index/trie.cc",
         "utilities/trie_index/trie_index_factory.cc",
         "utilities/ttl/db_ttl_impl.cc",
         "utilities/types_util.cc",

CMakeLists.txt

@@ -1107,8 +1107,7 @@ set(SOURCES
         utilities/transactions/write_prepared_txn_db.cc
         utilities/transactions/write_unprepared_txn.cc
         utilities/transactions/write_unprepared_txn_db.cc
-        utilities/trie_index/bitvector.cc
-        utilities/trie_index/louds_trie.cc
+        utilities/trie_index/trie.cc
         utilities/trie_index/trie_index_factory.cc
         utilities/types_util.cc
         utilities/ttl/db_ttl_impl.cc

db_stress_tool/db_stress_tool.cc

@@ -567,15 +567,6 @@ int db_stress_tool(int argc, char** argv) {
     FLAGS_atomic_flush = true;
   }
 
-  // Trie UDI uses zero-copy pointers into block data, which is
-  // incompatible with mmap_read.
-  if (FLAGS_use_trie_index && FLAGS_mmap_read) {
-    return ReturnValidationError(
-        "use_trie_index is incompatible with mmap_read. The trie index uses "
-        "zero-copy pointers into block data which is unsafe with mmap'd "
-        "reads.");
-  }
-
   // TrieIndexFactory requires plain BytewiseComparator, but timestamps use
   // BytewiseComparator.u64ts.
   if (FLAGS_use_trie_index && FLAGS_user_timestamp_size > 0) {

docs/components/index_factory/index.md

@@ -14,7 +14,7 @@ of, the built-in index selected by `BlockBasedTableOptions::index_type`.
 | Mode / read selection options | [`include/rocksdb/table.h`](../../../include/rocksdb/table.h), [`include/rocksdb/options.h`](../../../include/rocksdb/options.h) |
 | Built-in index adapter | `table/block_based/builtin_index_factory.{h,cc}` |
 | Reader wrapper | `table/block_based/user_defined_index_wrapper.h` |
-| Reference implementation | `utilities/trie_index/` (LOUDS trie, registered as `trie_index`) |
+| Reference implementation | `utilities/trie_index/` (registered as `trie_index`) |
 
 ## SPI shape
 
@@ -40,7 +40,7 @@ tiers an SST is written with and which one reads use by default.
 | Mode | Standard index written | Custom index written | Default read target | Missing/unusable custom block |
 |------|------------------------|----------------------|---------------------|-------------------------------|
 | `kStandardOnly` | full | no (factory ignored entirely) | standard | n/a |
-| `kStandardDefault` | full | yes | standard | warn, fall back to standard |
+| `kStandardDefault` | full | yes | standard | log, fall back to standard |
 | `kStandardRequired` | full | yes | standard | hard error at open |
 | `kCustomDefault` | full | yes | custom | hard error at open |
 | `kCustomOnly` | stub only | yes | custom | hard error at open |
@@ -129,12 +129,14 @@ baking a degraded reader into a long-lived `Rep`.
 Custom factories cannot be combined with user-defined timestamps unless
 `kStandardOnly` ignores the factory.
 
-The trie format stores data-block offsets and sizes as 32-bit integers. A block
-with either field above `UINT32_MAX` makes construction return `NotSupported`,
-including in `kStandardDefault`, where trie construction is optional. This can
-fail flushes, compactions, or `SstFileWriter::Finish()`. Total SST size above
-4 GiB alone is not the rejection condition. File-size targets do not split L0
-outputs or `SstFileWriter` outputs; bound those outputs or use `kStandardOnly`.
+`BlockBasedTableFactory::ValidateOptions()` calls the factory's
+`ValidateOptions()` in every mode except `kStandardOnly`. `DB::Open()` and
+`CreateColumnFamily()` run that validation; `SetOptions()` and `SstFileWriter`
+do not, and the table builder does not repeat it. A factory that cannot build
+with some option should therefore also fail `NewBuilder()`. The built-in trie
+rejects comparators other than `BytewiseComparator` in both places, so
+`DB::Open()` and `CreateColumnFamily()` fail up front, and the other paths fail
+at the first table build.
 
 ## Cache and memory accounting
 
@@ -150,7 +152,7 @@ outputs or `SstFileWriter` outputs; bound those outputs or use `kStandardOnly`.
   allocations, excluding the serialized block. A reader whose estimate includes
   the block can override `MemoryUsageIncludesIndexBlock()` to return true; the
   wrapper then subtracts the block size from that estimate. `TrieIndexReader`
-  reports just its auxiliary lookup tables.
+  reads the block in place, so it reports only its own object.
 
 ## Parallel compression
 
@@ -172,9 +174,11 @@ Recommended path: `kStandardDefault` -> `kStandardRequired` -> `kCustomDefault`
 -> `kCustomOnly`, compacting between steps so existing SSTs pick up the new
 layout.
 
-Rolling back from `kCustomDefault` or below is immediate: those files always
-carry a full standard index, so lowering `index_mode` sends reads back to it
-without waiting for compaction.
+Rolling back from `kCustomDefault` or below needs no compaction: those files
+always carry a full standard index. Index routing is fixed when a table reader
+opens, though. Lowering `index_mode` through `SetOptions()` reaches only SSTs
+opened afterwards; SSTs already open keep their routing until they are
+reopened, e.g. by reopening the DB.
 
 `kCustomOnly` is effectively a one-way door. Those SSTs have no usable standard
 index, so before dropping the factory you must move to `kCustomDefault` and
@@ -215,12 +219,16 @@ compact every file, with the factory still attached.
 `ValidateOptions()` (and, for callers such as `SstFileWriter` that never reach
 it, the table builder) rejects:
 
-- a custom factory together with user-defined timestamps;
+- a custom factory together with user-defined timestamps, except in
+  `kStandardOnly`;
 - `kCustomDefault` / `kCustomOnly` with `kTwoLevelIndexSearch` or
   `partition_filters`;
 - `kCustomOnly` with `format_version < 6`;
 - `kCustomDefault` / `kCustomOnly` with no factory.
 
+`ValidateOptions()` alone also rejects anything the factory's own
+`ValidateOptions()` rejects, except in `kStandardOnly`.
+
 ## Observability
 
 - `SST_USER_DEFINED_INDEX_LOAD_FAIL_COUNT` counts SSTs that could not use their

include/rocksdb/index_factory.h

@@ -352,14 +352,21 @@ struct IndexFactoryOptions {
 //
 // Extends Customizable for string-based construction (CreateFromString),
 // options serialization, and Name()-based identification.
+//
+// A factory can override ValidateOptions() to reject options it does not
+// support. BlockBasedTableFactory calls it in every index mode except
+// kStandardOnly, which ignores the factory. It runs when DB::Open() and
+// CreateColumnFamily() validate the options, but not for SetOptions() or
+// SstFileWriter, so a factory that cannot build with an option should also
+// fail NewBuilder().
 // ---------------------------------------------------------------------------
 class IndexFactory : public Customizable {
  public:
   ~IndexFactory() override = default;
 
   static const char* Type() { return "UserDefinedIndexFactory"; }
 
-  // Create an IndexFactory from a string identifier (e.g., "trie").
+  // Create an IndexFactory from a string identifier (e.g., "trie_index").
   static Status CreateFromString(const ConfigOptions& config_options,
                                  const std::string& value,
                                  std::shared_ptr<IndexFactory>* factory);

include/rocksdb/table.h

@@ -660,11 +660,15 @@ struct BlockBasedTableOptions {
   //
   // Rollback:
   //   From kCustomDefault: switch to kStandardDefault or kStandardOnly.
-  //     The standard index is fully populated, so SSTs are immediately
-  //     readable. kStandardOnly silently ignores any factory pointer that
-  //     remains set in the config.
+  //     The standard index is fully populated, so no rewrite is needed.
+  //     kStandardOnly silently ignores any factory pointer that remains set
+  //     in the config.
   //   From kCustomOnly: switch to kCustomDefault and compact to rewrite
   //     all SSTs with both indexes before downgrading further.
+  //   An SST's index routing is fixed when its table reader is opened. A
+  //   change made through SetOptions() applies to SSTs opened afterwards;
+  //   SSTs already open keep their routing until they are reopened, e.g. by
+  //   reopening the DB.
   //
   // Backup/restore and other Options-serialization reopen paths:
   //   user_defined_index_factory (shared_ptr) does not survive Options

src.mk

@@ -352,8 +352,7 @@ LIB_SOURCES =                                                   \
   utilities/transactions/write_unprepared_txn.cc                \
   utilities/transactions/write_unprepared_txn_db.cc             \
   utilities/ttl/db_ttl_impl.cc                                  \
-  utilities/trie_index/bitvector.cc                              \
-  utilities/trie_index/louds_trie.cc                             \
+  utilities/trie_index/trie.cc                                   \
   utilities/trie_index/trie_index_factory.cc                     \
   utilities/types_util.cc                                       \
   utilities/wal_filter.cc                                       \

table/block_based/block_based_table_factory.cc

@@ -421,7 +421,14 @@ static struct BlockBasedTableTypeInfo {
          OptionTypeInfo::AsCustomSharedPtr<IndexFactory>(
              offsetof(struct BlockBasedTableOptions,
                       user_defined_index_factory),
-             OptionVerificationType::kByNameAllowFromNull)},
+             OptionVerificationType::kByNameAllowFromNull)
+             // BlockBasedTableFactory::ValidateOptions() validates the factory
+             // only in the index modes that use it.
+             .SetValidateFunc(
+                 [](const DBOptions& /*db_opts*/,
+                    const ColumnFamilyOptions& /*cf_opts*/,
+                    const std::string& /*name*/,
+                    const void* /*addr*/) { return Status::OK(); })},
         {"whole_key_filtering",
          {offsetof(struct BlockBasedTableOptions, whole_key_filtering),
           OptionType::kBoolean, OptionVerificationType::kNormal}},
@@ -826,6 +833,18 @@ Status BlockBasedTableFactory::ValidateOptions(
         "user_defined_index_factory is incompatible with user-defined "
         "timestamps");
   }
+  // kStandardOnly ignores the factory, so only the other modes are bound by
+  // its requirements. The user_defined_index_factory option's validate func
+  // skips the generic Configurable check so that this one decides.
+  if (table_options_.GetEffectiveIndexMode() !=
+          BlockBasedTableOptions::IndexMode::kStandardOnly &&
+      table_options_.user_defined_index_factory != nullptr) {
+    Status s = table_options_.user_defined_index_factory->ValidateOptions(
+        db_opts, cf_opts);
+    if (!s.ok()) {
+      return s;
+    }
+  }
   if (table_options_.cache_index_and_filter_blocks &&
       table_options_.no_block_cache) {
     return Status::InvalidArgument(

tools/db_bench_tool.cc

@@ -777,9 +777,7 @@ DEFINE_bool(index_with_first_key, false, "Include first key in the index");
 
 DEFINE_bool(use_trie_index, false,
             "Use trie-based user-defined index (UDI) for block-based tables. "
-            "Builds a LOUDS-encoded succinct trie from separator keys, "
-            "providing space reduction compared to the standard index. "
-            "Requires BytewiseComparator.");
+            "Builds a trie from separator keys. Requires BytewiseComparator.");
 
 DEFINE_int32(index_mode,
              static_cast<int32_t>(

tools/db_crashtest.py

@@ -1643,12 +1643,7 @@ def finalize_and_sanitize(src_params):
         dest_params["open_read_fault_one_in"] = 0
         dest_params["sync_fault_injection"] = 0
 
-    # UDI is compatible with all operation types and iteration directions
-    # except mmap_read.
     if dest_params.get("use_trie_index") == 1:
-        # Trie UDI uses zero-copy pointers into block data, which is
-        # incompatible with mmap_read.
-        dest_params["mmap_read"] = 0
         index_mode = dest_params.get("index_mode")
         if index_mode in (2, 3):
             # kCustomDefault/kCustomOnly reject partitioned index
@@ -2291,6 +2286,17 @@ def gen_cmd_params(args, unknown_params=None):
         if v is not None:
             params[k] = v
 
+    # With the default 16 KiB blocks an SST has only a few dozen separators.
+    # Smaller blocks give the trie more buckets and nodes, and user keys whose
+    # versions span several blocks. kStandardOnly (index_mode 0) never builds
+    # the trie, and an explicit --block_size wins.
+    if (
+        getattr(args, "block_size", None) is None
+        and params.get("use_trie_index") == 1
+        and params.get("index_mode") != 0
+    ):
+        params["block_size"] = lambda: random.choice([1024, 4096, 16384])
+
     # Reserve a small fraction of ordinary crash-test invocations for the
     # file-scoped WAL recovery state machine. Do this after merging command
     # line arguments so automatic selection cannot override an explicit opt-out

tools/db_crashtest_test.py

@@ -961,6 +961,33 @@ def test_finalize_keeps_parallel_compression_with_trie_index(self):
                 self.assertEqual(1, finalized["use_trie_index"])
                 self.assertEqual(4, finalized["compression_parallel_threads"])
 
+    def test_trie_block_size_draw(self):
+        db_crashtest = self.load_db_crashtest()
+
+        def block_sizes(**overrides):
+            # auto_readahead_size=0 keeps the unrelated 1 KiB block shrink off.
+            args = self.build_mode_args(
+                test_type="blackbox", use_trie_index=1, auto_readahead_size=0,
+                **overrides
+            )
+            params = db_crashtest.gen_cmd_params(args)
+            params["db"] = self.test_tmpdir
+            return {
+                db_crashtest.finalize_and_sanitize(dict(params))["block_size"]
+                for _ in range(60)
+            }
+
+        # Smaller blocks are drawn for the modes that build the trie.
+        drawn = block_sizes(index_mode=1)
+        self.assertTrue(drawn <= {1024, 4096, 16384}, drawn)
+        self.assertGreater(len(drawn), 1)
+        # An explicit --block_size is kept.
+        self.assertEqual({8192}, block_sizes(index_mode=1, block_size=8192))
+        # kStandardOnly never builds the trie, so it keeps the default.
+        self.assertEqual(
+            {db_crashtest.default_params["block_size"]}, block_sizes(index_mode=0)
+        )
+
     def test_finalize_disables_test_batches_snapshots_when_disable_wal(self):
         db_crashtest = self.load_db_crashtest()
         params = self.build_params(
@@ -1295,7 +1322,7 @@ def test_finalize_trie_index_mode_rules(self):
                 finalized = db_crashtest.finalize_and_sanitize(params)
 
                 self.assertEqual(index_mode, finalized["index_mode"])
-                self.assertEqual(0, finalized["mmap_read"])
+                self.assertEqual(1, finalized["mmap_read"])
                 if index_mode in (2, 3):
                     self.assertIn(finalized["index_type"], [0, 3])
                     self.assertEqual(0, finalized["partition_filters"])

unreleased_history/behavior_changes/trie_index_format.md

@@ -0,0 +1 @@
+The experimental trie index (`TrieIndexFactory`) has a new on-disk format that is smaller, faster to search and supports data blocks past 4 GiB. Trie blocks from earlier releases fail with `NotSupported`: in `kStandardDefault` their SSTs fall back to the standard index, so compact there to rewrite them. SSTs written in `kCustomOnly` have no standard index, so rewrite them in another mode before upgrading. Earlier releases cannot read the new trie blocks. The empty user key no longer fails flushes and compactions as a trie separator, and `DB::Open()` and `CreateColumnFamily()` now reject the trie with a non-bytewise comparator outside `kStandardOnly`.