diff --git a/Rules.md b/Rules.md index b36d8332..6c231d4c 100644 --- a/Rules.md +++ b/Rules.md @@ -16,7 +16,6 @@ * [3.6. Training OPEN versus CLOSED Options](#36-training-open-versus-closed-options) * [4. Validating the Checkpointing Workloads](#4-validating-the-checkpointing-workloads) * [4.1. Checkpointing Sizing Options](#41-checkpointing-sizing-options) - * * [4.2. Checkpointing Generation Options](#42-checkpointing-generation-options) * [4.3. Checkpointing Run Options](#43-checkpointing-run-options) * [4.4. Checkpointing Access Via POSIX API Options](#44-checkpointing-access-via-posix-api-options) @@ -37,7 +36,7 @@ * [6.4. KVCache Access Via POSIX API Options](#64-kvcache-access-via-posix-api-options) * [6.5. KVCache Access Via Object API Options](#65-kvcache-access-via-object-api-options) * [6.6. KVCache OPEN versus CLOSED Options](#66-kvcache-open-versus-closed-options) -# 1. Introduction +# 1. Introduction These are the requirements for the *submission validation checker* for version 2.0 of the MLPerf™ Storage benchmark, but since the `mlpstorage` tool will be responsible for generating the vast majority (if not all) of the contents of a submission, it is also a spec for what `mlpstorage` should generate. @@ -54,9 +53,9 @@ The `mlpstorage` tool must be used to run the benchmarks, submitters are not all 1.1. **mlpstorageGeneratesHierarchy** -- The `mlpstorage` command must obtain (somehow) the pathname of the output file directory hierarchy and directly create and/or append to the files within that hierarchy to successively build out the submission folder. We don't want the submitter to manually create anything in that hierarchy except for the SystemDescription.* files (if we can help it). -# 2. Core/Common Rules for All Submissions +# 2. Core/Common Rules for All Submissions -## 2.1. Core/Common POSIX API Rules +## 2.1. Core/Common POSIX API Rules 2.1.1. **submitterRootDirectory** -- The submission structure must start from a single directory whose name is the name of the submitter. This can be any string, but a blank or any other character in that string that cannot be part of a POSIX filename should be replaced 1-for-1 with a dash character. @@ -72,7 +71,7 @@ The `mlpstorage` tool must be used to run the benchmarks, submitters are not all 2.1.5.b. **requiredSubdirectoriesOpen** -- Within an OPEN submitter directory, there must be exactly two directories: "results" and "systems". These names are case-sensitive. The "code" directory does NOT appear at the OPEN submitter level; instead, a "code" directory is captured at each leaf inside `results/`. The leaf shape is per-benchmark-type: - For "training" and "checkpointing" the leaf is `results////` (one capture per model). -- For "vector_database" the leaf is `results//vector_database//` where `` is the UPPERCASE token (`DISKANN`, `HNSW`, or `AISAQ`) (one capture per index type, because results across index types — e.g. AISAQ vs DISKANN vs HNSW — are not comparable and must live in separate trees). +- For "vector_database" the leaf is `results//vector_database///` where `` is the engine token (`milvus` today) and `` is the UPPERCASE index token (`DISKANN`, `HNSW`, or `AISAQ`) (one capture per engine/index pair, because results across engines or index families are not comparable and must live in separate trees). - For "kv_cache" the leaf is currently `results///` (one capture per type). This is transitional pending finalization of the kv_cache directory structure below the type prefix. See §2.1.6 and §2.1.27. @@ -119,8 +118,6 @@ configuration of storage system and to link together those results with the .pdf 2.1.18. **runTimestampGap** -- The timestamp (the day and time) represented by the name of each *timestamp directory* must be separated by less than the duration of a single *timestamp directory* from it's neighboring *timestamp directories*. Ie: the gap between a consecutive pair of *timestamp directories* must be short enough that we can be sure that there was no benchmark activity between them. -2.1.18a. **runPickingSubsets** -- It is permissible to execute a large number of runs (all consecutive) and then delete the *timestamp directories* for all but 6 consecutive runs in the middle. This differs from "cherry picking" in that these runs must be consecutive, see **runTimestampGap**. - 2.1.19. **runFiles** -- Within each *timestamp directory* within the "run" *phase*, there must exist the following files: "training_run.stdout.log", "training_run.stderr.log" file, "*output.json, "*per_epoch_stats.json", "*summary.json", and "dlio.log", plus a subdirectory named "dlio_config". These names are case-sensitive. 2.1.20. **runDlioConfig** -- The "dlio_config" subdirectory in each *timestamp directory* must contain the following list of files, and nothing else: "config.yaml", "hydra.yaml", and "overrides.yaml". These names are case-sensitive. @@ -133,8 +130,6 @@ configuration of storage system and to link together those results with the .pdf 2.1.24. **checkpointingTimestampGap** -- The timestamp (the day and time) represented by the name of each *timestamp directory* must be separated by less than the duration of a single *timestamp directory* from it's neighboring *timestamp directories*. Ie: the gap between a consecutive pair of *timestamp directories* must be short enough that we can be sure that there was no benchmark activity between them. -2.1.24a. **checkpointingPickingSubsets** -- It is permissible to execute a large set of runs (all consecutive) and then delete all but one run (with 10 checkpoint files in it) in the middle. - 2.1.25. **checkpointingFiles** -- Within the *timestamp directories* within the "checkpointing" directory hierarchy, there must exist the following files: "checkpointing_run.stdout.log", "checkpointing_run.stderr.log" file, "*output.json, "*per_epoch_stats.json", "*summary.json", and "dlio.log", plus a subdirectory named "dlio_config". These names are case-sensitive. 2.1.26. **checkpointingDlioConfig** -- The "dlio_config" subdirectory in each *timestamp directory* must contain the following list of files, and nothing else: "config.yaml", "hydra.yaml", and "overrides.yaml". These names are case-sensitive. @@ -200,36 +195,37 @@ root_folder (or any name you prefer) │ │ │ └── YYYYMMDD_HHmmss │ │ │ └── dlio_config │ │ └── vector_database -| | ├── AISAQ -│ │ | ├── datagen -│ │ | │ └── YYYYMMDD_HHmmss -│ │ | │ └── summary.json -│ │ | └── run -│ │ | ├── YYYYMMDD_HHmmss -│ │ | │ └── summary.json -│ │ | ... (5x Runs total) -│ │ | └── YYYYMMDD_HHmmss -│ │ | └── summary.json -| | ├── DISKANN -│ │ | ├── datagen -│ │ | │ └── YYYYMMDD_HHmmss -│ │ | │ └── summary.json -│ │ | └── run -│ │ | ├── YYYYMMDD_HHmmss -│ │ | │ └── summary.json -│ │ | ... (5x Runs total) -│ │ | └── YYYYMMDD_HHmmss -│ │ | └── summary.json -| | └── HNSW -│ │ ├── datagen -│ │ │ └── YYYYMMDD_HHmmss -│ │ │ └── summary.json -│ │ └── run -│ │ ├── YYYYMMDD_HHmmss -│ │ │ └── summary.json -│ │ ... (5x Runs total) -│ │ └── YYYYMMDD_HHmmss -│ │ └── summary.json +│ │ └── milvus +│ │ ├── AISAQ +│ │ │ ├── datagen +│ │ │ │ └── YYYYMMDD_HHmmss +│ │ │ │ └── summary.json +│ │ │ └── run +│ │ │ ├── YYYYMMDD_HHmmss +│ │ │ │ └── summary.json +│ │ │ ... (5x Runs total) +│ │ │ └── YYYYMMDD_HHmmss +│ │ │ └── summary.json +│ │ ├── DISKANN +│ │ │ ├── datagen +│ │ │ │ └── YYYYMMDD_HHmmss +│ │ │ │ └── summary.json +│ │ │ └── run +│ │ │ ├── YYYYMMDD_HHmmss +│ │ │ │ └── summary.json +│ │ │ ... (5x Runs total) +│ │ │ └── YYYYMMDD_HHmmss +│ │ │ └── summary.json +│ │ └── HNSW +│ │ ├── datagen +│ │ │ └── YYYYMMDD_HHmmss +│ │ │ └── summary.json +│ │ └── run +│ │ ├── YYYYMMDD_HHmmss +│ │ │ └── summary.json +│ │ ... (5x Runs total) +│ │ └── YYYYMMDD_HHmmss +│ │ └── summary.json │ └── systems │ ├──system-name-1.yaml │ ├──system-name-1.pdf @@ -298,39 +294,40 @@ root_folder (or any name you prefer) │ │ └── YYYYMMDD_HHmmss │ │ └── dlio_config │ └── vector_database - | ├── AISAQ - │ | ├── code # captured per-leaf - │ | ├── datagen - │ | │ └── YYYYMMDD_HHmmss - │ | │ └── summary.json - │ | └── run - │ | ├── YYYYMMDD_HHmmss - │ | │ └── summary.json - │ | ... (5x Runs total) - │ | └── YYYYMMDD_HHmmss - │ | └── summary.json - | ├── DISKANN - │ | ├── code # captured per-leaf - │ | ├── datagen - │ | │ └── YYYYMMDD_HHmmss - │ | │ └── summary.json - │ | └── run - │ | ├── YYYYMMDD_HHmmss - │ | │ └── summary.json - │ | ... (5x Runs total) - │ | └── YYYYMMDD_HHmmss - │ | └── summary.json - | └── HNSW - │ ├── code # captured per-leaf - │ ├── datagen - │ │ └── YYYYMMDD_HHmmss - │ │ └── summary.json - │ └── run - │ ├── YYYYMMDD_HHmmss - │ │ └── summary.json - │ ... (5x Runs total) - │ └── YYYYMMDD_HHmmss - │ └── summary.json + │ └── milvus + │ ├── AISAQ + │ │ ├── code # captured per-leaf + │ │ ├── datagen + │ │ │ └── YYYYMMDD_HHmmss + │ │ │ └── summary.json + │ │ └── run + │ │ ├── YYYYMMDD_HHmmss + │ │ │ └── summary.json + │ │ ... (5x Runs total) + │ │ └── YYYYMMDD_HHmmss + │ │ └── summary.json + │ ├── DISKANN + │ │ ├── code # captured per-leaf + │ │ ├── datagen + │ │ │ └── YYYYMMDD_HHmmss + │ │ │ └── summary.json + │ │ └── run + │ │ ├── YYYYMMDD_HHmmss + │ │ │ └── summary.json + │ │ ... (5x Runs total) + │ │ └── YYYYMMDD_HHmmss + │ │ └── summary.json + │ └── HNSW + │ ├── code # captured per-leaf + │ ├── datagen + │ │ └── YYYYMMDD_HHmmss + │ │ └── summary.json + │ └── run + │ ├── YYYYMMDD_HHmmss + │ │ └── summary.json + │ ... (5x Runs total) + │ └── YYYYMMDD_HHmmss + │ └── summary.json └── systems ├──system-name-1.yaml ├──system-name-1.pdf @@ -350,11 +347,11 @@ root_folder (or any name you prefer) └── overrides.yaml ``` -## 2.2. Core/Common Object API Rules +## 2.2. Core/Common Object API Rules -# 3. Validating the Training Workloads +# 3. Validating the Training Workloads -## 3.1. Training Sizing Options +## 3.1. Training Sizing Options 3.1.1. **trainingVerifyDatasizeUsage** -- The *submission validator* must verify that the *datasize* option was used by finding the entry(s) in the log file showing its use. @@ -370,11 +367,11 @@ root_folder (or any name you prefer) * `min_files_size = min_samples * record_length / 1024 / 1024 / 1024` * A minimum of `min_total_files` files are required which will consume `min_files_size` GB of storage. -## 3.2. Training Generation Options +## 3.2. Training Generation Options 3.2.1. **trainingDatagenMinimumSize** -- The amount of data generated during the *datagen* phase must be equal **or larger** -- than the amount of data calculated during the *datasize* phase or the run must be failed. -## 3.3. Training Run Options +## 3.3. Training Run Options 3.3.1. **trainingRunDataMatchesDatasize** -- The amount of data the *run* phase is told to use must be exactly equal to the *datasize* value calculated earlier, but can be less than the value used in the *datagen* phase. To express that, you can run the benchmark on a subset of that dataset by setting `num_files_train` or `num_files_eval` smaller than the number of files available in the dataset folder, but `num_subfolders_train` and `num_subfolders_eval` must be to be equal to the actual number of subfolders inside the dataset folder in order to generate valid results. @@ -393,15 +390,15 @@ root_folder (or any name you prefer) 3.3.7. **trainingNodeCapabilityConsistency** -- For distributed Training submissions, the *submission validation checker* should emit a warning (not fail the validation) if the physical nodes that run the benchmark code are widely enough different in their capability. **_(not clear we should do this, so maybe remove?)_** -## 3.4. Training Access Via POSIX API Options +## 3.4. Training Access Via POSIX API Options 3.4.1. **trainingMlpstoragePathArgs** -- The arguments to `mlpstorage` that set the directory pathname where the dataset is stored and the directory where the output logfiles are stored must both be set and must be set to different values. 3.4.2. **trainingMlpstorageFilesystemCheck** -- The `mlpstorage` command should do a "df" command on the directory pathname where the dataset is stored and another one on the directory pathname where the output logfiles are stored and record those values in the logfile. The *submission validator* should find those entries in the run's logfile and verify that they are different filesystems. We don't want the submitter to, by acccident, place the logfiles onto the storage system under test since that would skew the results. -## 3.5. Training Access Via Object API Options +## 3.5. Training Access Via Object API Options -## 3.6. Training OPEN versus CLOSED Options +## 3.6. Training OPEN versus CLOSED Options 3.6.1. **trainingClosedSubmissionChecksum** -- For CLOSED submissions of this benchmark, the MLPerf Storage codebase must not be changed. The *submission validation checker* enforces this with a layered check: @@ -446,13 +443,13 @@ root_folder (or any name you prefer) | *Reader parameters* | | | | reader.data_loader | Supported options: Tensorflow or PyTorch. | 3D U-Net: PyTorch
ResNet-50: Tensorflow
Cosmoflow: Tensorflow | -# 4. Validating the Checkpointing Workloads +# 4. Validating the Checkpointing Workloads -## 4.1. Checkpointing Sizing Options +## 4.1. Checkpointing Sizing Options -## 4.2. Checkpointing Generation Options +## 4.2. Checkpointing Generation Options -## 4.3. Checkpointing Run Options +## 4.3. Checkpointing Run Options 4.3.1. **checkpointDataSizeRatio** -- The checkpoint data written per client node must be more than 3x the client node's memory capacity, otherwise the filesystem cache needs to be cleared between the write and read phases. @@ -487,7 +484,7 @@ root_folder (or any name you prefer) ## 4.5. Checkpointing Access Via Object API Options -## 4.6. Checkpointing OPEN versus CLOSED Options +## 4.6. Checkpointing OPEN versus CLOSED Options 4.6.1. **checkpointClosedMpiProcesses** -- For CLOSED submissions, the number of MPI processes must be set to 8, 64, 512, and 1024 for the respective models. (see table 2) @@ -517,7 +514,7 @@ root_folder (or any name you prefer) **\*\* NOTE: In CLOSED submissions, ``--num-checkpoints-write`` and ``--num-checkpoints-read`` may be set to ``0`` only as part of the two-invocation cache-flush workflow described in §4.7.1: one invocation runs the write phase with ``--num-checkpoints-read=0`` and the next runs the read phase with ``--num-checkpoints-write=0``. The default for both flags is 10 and the total work performed across both invocations must still be 10 writes followed by 10 reads.** -## 4.7. Storage System Must Be Simultaneously R/W or _Remappable_ +## 4.7. Storage System Must Be Simultaneously R/W or _Remappable_ 4.7.1. **checkpointCacheFlushValidation** -- A cache flush between the write and read phases is only required when the client node has enough memory to cache all of the checkpoints written by that client during the run. The benchmark writes 10 sequential checkpoints specifically to overfill typical filesystem caches; on most submission configurations the early checkpoints have already been evicted by the time the read phase begins, so no flush is required. As a rule of thumb (see `checkpointing/README.md`), a flush is required when the total checkpoint size written per client is less than 3× the client node's memory capacity. When a flush is required, the submitter must execute the run in two invocations: the write phase with ``--num-checkpoints-read=0``, followed by the cache flush during a pause of no more than 30 seconds, then the read phase with ``--num-checkpoints-write=0``. The validator must confirm this split occurred and that the inter-phase gap did not exceed 30 seconds. @@ -534,23 +531,23 @@ System: simultaneous_read__support: True # Are simultaneous reads by multiple hosts supported in the submitted configuration ``` -# 5. Validating the VDB Workloads +# 5. Validating the VDB Workloads -## 5.1. VDB Sizing Options +## 5.1. VDB Sizing Options 5.1.1. **vdbDatasetScale** -- The benchmark must be run against one of the defined dataset scales (collection vector counts) listed in the VDB scale table. The *submission validator* must read `num_vectors` and `dimension` from the run's `config.json`/`summary.json` and verify they match a defined scale; any other scale must generate a message and fail validation. 5.1.2. **vdbDimensionConsistency** -- The vector `dimension` recorded at `datagen` (load) time must equal the `dimension` used at `run` (query) time. The *submission validator* must compare the dimension in the load summary against the dimension in each run's `summary.json` and fail validation if they differ. -## 5.2. VDB Generation Options +## 5.2. VDB Generation Options 5.2.1. **vdbCollectionPopulated** -- The number of vectors actually inserted (`inserted_vectors`) during load must equal the declared `num_vectors` for the chosen scale. The *submission validator* must read the load summary and fail validation on a shortfall. 5.2.2. **vdbIndexBuildCompleted** -- The collection must be fully indexed and (when configured) compacted before the query phase. The *submission validator* must confirm an index-build / compaction record is present in the load output and that the index type recorded at load time matches the index type used at run time. -## 5.3. VDB Run Options +## 5.3. VDB Run Options -5.3.1. **vdbRunCount** -- Within each `vector_database//run/` directory (where `` is one of the UPPERCASE tokens `DISKANN`, `HNSW`, or `AISAQ`), there must be exactly five `` timestamp directories, each containing a `summary.json`. The count rule applies to query runs only — `datagen` is governed by §5.2. (see §2.1.27 directory diagram.) +5.3.1. **vdbRunCount** -- Within each `vector_database///run/` directory (where `` is `milvus` today and `` is one of the UPPERCASE tokens `DISKANN`, `HNSW`, or `AISAQ`), there must be exactly five `` timestamp directories, each containing a `summary.json`. The count rule applies to query runs only — `datagen` is governed by §5.2. (see §2.1.27 directory diagram.) 5.3.2. **vdbRecallReported** -- Each run's `summary.json` (or its rank-local `recall_stats.json`) must report a recall value computed outside the timed query loop. The *submission validator* must verify a recall field is present and that recall meets or exceeds the minimum recall target defined for the chosen scale/metric. @@ -558,105 +555,107 @@ System: 5.3.4. **vdbMetricsReported** -- Each run's `summary.json` must report `throughput_qps` and the latency percentile set (`mean_latency_ms`, `p95_latency_ms`, `p99_latency_ms`, `p999_latency_ms`). The *submission validator* must verify these fields exist and are populated. -## 5.4. VDB Access Via POSIX API Options +## 5.4. VDB Access Via POSIX API Options 5.4.1. **vdbPathArgs** -- The arguments to `mlpstorage` that set the storage path for the vector database data and the directory where output logfiles/results are stored must both be set and must be set to different values. 5.4.2. **vdbFilesystemCheck** -- The `mlpstorage` command should do a "df" command on the directory pathname where the vector database stores its data and another on the directory pathname where the output logfiles are stored, and record those values in the logfile. The *submission validator* must find those entries in the run's logfile and verify that they are different filesystems, so that logfiles are not accidentally placed on the storage system under test. -## 5.5. VDB Access Via Object API Options +## 5.5. VDB Access Via Object API Options 5.5.1. **vdbObjectStorageBackend** -- For object-API submissions, the vector database must be backed by S3-compatible object storage and the submission must record the storage backend in the system description. The *submission validator* must confirm the recorded backend is consistent with the declared API. -## 5.6. VDB OPEN versus CLOSED Options +## 5.6. VDB OPEN versus CLOSED Options -> **Index type token convention.** The index type is recorded, validated, and -> stored on disk using the uppercase token (`DISKANN`, `HNSW`, `AISAQ`) defined -> by `VDB_INDEX_TYPES_CLOSED` in `mlpstorage_py/config.py`. The same token is -> used by the CLI (`--index-type`), in `summary.json.index_type`, and as the -> index directory name in the §2.1 directory diagram. +> **Index and engine token convention.** `--vdb-engine` identifies the vector +> database implementation and `--vdb-index` identifies the benchmark/index +> family used in metadata, reporting, and the outer result path (`vector_database///`). For +> `datasize` and `datagen`, `--index-type` is the concrete Milvus build +> argument; it defaults to `--vdb-index`, and the two values must match when +> both are supplied. Index tokens are stored in uppercase (`DISKANN`, `HNSW`, +> `AISAQ`) as defined by `VDB_INDEX_TYPES_CLOSED`. 5.6.1. **vdbClosedSubmissionChecksum** -- For CLOSED VDB submissions, the *submission validator* enforces the same layered code-image check defined in §3.6.1: self-consistency against `.code-hash.json` always, plus upstream-identity against `REFERENCE_CHECKSUMS` (or `--reference-checksum`) for CLOSED. See §2.1.6 for the `.code-hash.json` schema and exclusion set. -5.6.2. **vdbClosedDatabaseBackend** -- For CLOSED submissions, the vector database backend must be Milvus. The *submission validator* must read the `database.database` field from the run's `config.json`/`summary.json` and fail validation if any backend other than `milvus` is recorded. - -5.6.3. **vdbClosedIndexTypes** -- For CLOSED submissions, the index type must be one of exactly three supported types: `DISKANN`, `HNSW`, or `AISAQ` (matching `VDB_INDEX_TYPES_CLOSED`). The *submission validator* must read the `index_type` field and the index directory name under "vector_database" and fail validation if any other index type (e.g. `IVF_FLAT`, `IVF_SQ8`, or `FLAT`) is recorded. Within these three index types, the submitter is free to choose the metric type and any index-specific build and search parameters (see 5.6.4). - -5.6.4. **vdbClosedSubmissionParameters** -- For CLOSED submissions of this benchmark, the database backend is fixed to Milvus (see 5.6.2) and the index type is restricted to `DISKANN`, `HNSW`, or `AISAQ` (see 5.6.3), but the submitter may freely choose the metric type and all index-specific build/search parameters for those three index types, plus the load and run parameters listed in the table below. Any other parameter being modified, any unsupported index type, or any attempt to substitute a different database backend must generate a message and fail the validation. - -**Table: VectorDB Tunable Parameters for CLOSED (Milvus backend; DISKANN / HNSW / AISAQ only)** - -| Parameter | CLI flag | Description | Default | -|----------------------------|----------------------|------------------------------------------------------------------|--------------| -| *Database parameters* | | | | -| database.database | -- | Backend database engine — **fixed to `milvus` for CLOSED** | milvus | -| | | | | -| *Index selection* | | *(restricted to the three CLOSED index types)* | | -| index.index_type | `--index-type` | Index family — **one of: `DISKANN`, `HNSW`, `AISAQ`** | DISKANN | -| index.metric_type | `--metric-type` | Distance metric (e.g. COSINE, L2, IP) | COSINE | -| | | | | -| *DISKANN index parameters* | | | | -| index.max_degree | `--max-degree` | Max graph degree (DiskANN build) | 64 | -| index.search_list_size | `--search-list-size` | Search list size (DiskANN build) | 200 | -| search.search_ef | `--search-ef` | DiskANN search-time list size (recall/throughput trade-off) | -- | -| | | | | -| *HNSW index parameters* | | | | -| index.M | `--max-degree` | Max neighbors per node (HNSW build; shares the degree flag) | 64 | -| index.ef_construction | `--ef-construction` | Construction-time candidate list size (HNSW build) | 200 | -| search.search_ef | `--search-ef` | HNSW search-time `ef` (recall/throughput trade-off) | -- | -| | | | | -| *AISAQ index parameters* | | | | -| index.max_degree | `--max-degree` | Max graph degree (AISAQ build) | 64 | -| index.search_list_size | `--search-list-size` | Search list size (AISAQ build) | 200 | -| index.inline_pq | `--inline-pq` | AISAQ inline product-quantization parameter (perf vs scale) | 16 | -| search.search_ef | `--search-ef` | AISAQ search-time list size (recall/throughput trade-off) | -- | -| | | | | -| *Search / run parameters* | | | | -| run.mode | `--mode` | Benchmark mode: `timed` or `query_count` | timed | -| run.num_query_processes | `--num-query-processes` | Local Python query workers inside each rank | -- | -| run.batch_size | `--batch-size` | Query batch size | -- | -| run.report_count | `--report-count` | Reporting interval (queries between reports) | -- | -| | | | | -| *Dataset / load parameters*| | | | -| dataset.collection_name | `--collection` | Name of the collection populated and queried | -- | -| dataset.num_shards | `--num-shards` | Number of collection shards | -- | -| dataset.chunk_size | `--chunk-size` | Vectors per load chunk | -- | -| dataset.batch_size | `--batch-size` | Vectors per insert batch at load time | 1000 | -| dataset.vector_dtype | `--vector-dtype` | Vector data type (e.g. FLOAT_VECTOR) | FLOAT_VECTOR | -| | | | | -| *Storage parameters* | | | | -| storage.storage_root | -- | The storage root directory for VDB data | -- | -| storage.storage_type | -- | The storage type (e.g. local_fs, s3) | local_fs | - -5.6.5. **vdbOpenSubmissionParameters** -- For OPEN submissions of this benchmark, the submitter may additionally run against vector database backends other than Milvus — including **Elasticsearch** and **pgvector** — in addition to everything already permitted in CLOSED. The *submission validator* must verify that the recorded `database.database` is one of the supported backends. OPEN submissions may use any index types, metrics, and parameters native to the chosen backend (including the full `VDB_INDEX_TYPES` set such as `IVF_FLAT`, `IVF_SQ8`, and `FLAT` on Milvus), but must still meet the recall target (5.3.2) and report the required metrics (5.3.4). Any parameter not listed here or in the CLOSED table, when modified, must generate a message and fail the validation. - -**Table: VectorDB Additional Tunable Parameters for OPEN** - -| Parameter | Description | Default | -|----------------------------|---------------------------------------------------------------------------------------------------------------------|---------| -| *Database parameters* | | | -| database.database | Backend database engine — OPEN permits alternative backends including `milvus`, `elasticsearch`, and `pgvector` | milvus | -| database.host | Database endpoint host for the selected backend | -- | -| database.port | Database endpoint port for the selected backend | -- | -| | | | -| *Extended Milvus indexes* | *(index types available on Milvus in OPEN beyond the three CLOSED types)* | | -| index.index_type | Adds `IVF_FLAT`, `IVF_SQ8`, `FLAT` to the CLOSED `DISKANN` / `HNSW` / `AISAQ` set | -- | -| | | | -| *Backend-specific options* | *(any index types, metrics, and parameters native to a non-Milvus backend)* | | -| index.index_type | Any index family supported by the chosen backend (e.g. HNSW on Elasticsearch; HNSW / IVFFlat on pgvector) | -- | -| index.metric_type | Any distance metric supported by the chosen backend | -- | -| index.* (backend-native) | Any backend-native build/search parameters (e.g. pgvector `lists` / `probes`; Elasticsearch `m` / `ef_construction` / `num_candidates`) | -- | - -# 6. Validating the KVCache Options - -## 6.1. KVCache Sizing Options - -## 6.2. KVCache Generation Options - -## 6.3. KVCache Run Options - -## 6.4. KVCache Access Via POSIX API Options - -## 6.5. KVCache Access Via Object API Options - -## 6.6. KVCache OPEN versus CLOSED Options +5.6.2. **vdbClosedDatabaseBackend** -- For CLOSED submissions, the vector database backend must be Milvus. The *submission validator* must read the recorded engine/database field from the run metadata and fail validation if any backend other than `milvus` is recorded. The integrated CLI selects this field with `--vdb-engine`, and current result paths place it between `vector_database` and the index token. + +5.6.3. **vdbClosedIndexTypes** -- For CLOSED submissions, the index type must be one of exactly three supported types: `DISKANN`, `HNSW`, or `AISAQ` (matching `VDB_INDEX_TYPES_CLOSED`). The *submission validator* must read the recorded engine/index fields and the engine/index directory names under `vector_database` and fail validation if any other index type (for example `IVF_FLAT`, `IVF_SQ8`, or `FLAT`) is recorded. Each index family is a distinct result category. Within these three index types, the parameters permitted by 5.6.4 must be recorded with their effective values. + +5.6.4. **vdbClosedSubmissionParameters** -- For CLOSED submissions, the database backend is fixed to Milvus (5.6.2) and the index family is restricted to `DISKANN`, `HNSW`, or `AISAQ` (5.6.3). Unless a row is explicitly marked fixed or restricted, this rule currently permits the parameter to vary and requires the effective value to be recorded. Any unsupported index type, substituted database backend, or modification of a parameter outside the permitted rule surface must generate a message and fail validation. + +The final column below is an **implementation fallback**, not a mandatory CLOSED profile value. It describes the current effective value when neither a command-line argument nor a selected configuration supplies another value. Named YAML profiles may override fallbacks. A future versioned CLOSED profile must state fixed values separately and must be enforced by the CLI and submission validator. + +**Table: VectorDB parameter surface for CLOSED (Milvus; DISKANN / HNSW / AISAQ only)** + +| Parameter | Current CLI representation | CLOSED status under this rule | Implementation fallback | +|---|---|---|---:| +| *Database and index identity* | | | | +| database.database | `--vdb-engine` | **Fixed to `milvus`** | `milvus` | +| index.index_type | `--vdb-index` | **Restricted to `DISKANN`, `HNSW`, `AISAQ`** | `DISKANN` | +| index.implementation_type | `--index-type` on `datasize`/`datagen` | Derived from, or required to match, `--vdb-index` | `DISKANN` | +| index.metric_type | `--metric-type` | Permitted by this rule; the direct flag is currently OPEN/WHATIF-only | `COSINE` | +| *DISKANN build/search* | | | | +| index.max_degree | `--max-degree` | Permitted; direct build flag is currently OPEN/WHATIF-only | 16 | +| index.search_list_size | `--search-list-size` | Permitted; direct build flag is currently OPEN/WHATIF-only | 200 | +| search.search_ef | `--search-ef` | Permitted | 200 | +| *HNSW build/search* | | | | +| index.M | `--M` | Permitted; direct build flag is currently OPEN/WHATIF-only | 16 | +| index.ef_construction | `--ef-construction` | Permitted; direct build flag is currently OPEN/WHATIF-only | 200 | +| search.search_ef | `--search-ef` | Permitted | 200 | +| *AISAQ build/search* | | | | +| index.max_degree | `--max-degree` | Permitted; direct build flag is currently OPEN/WHATIF-only | 16 | +| index.search_list_size | `--search-list-size` | Permitted; direct build flag is currently OPEN/WHATIF-only | 200 | +| index.inline_pq | `--inline-pq` | Permitted; direct build flag is currently OPEN/WHATIF-only | 16 | +| search.search_ef | `--search-ef` | Permitted | 200 | +| *Search and run* | | | | +| run.benchmark_mode | `--benchmark-mode` | `timed` or `query_count` for a simple CLOSED result | `timed` | +| run.num_query_processes | `--num-query-processes` | Permitted and recorded | 1 | +| run.batch_size | `--batch-size` on `run` | Permitted and recorded | 1 | +| run.report_count | `--report-count` | Permitted and recorded | 100 | +| *Dataset and load* | | | | +| dataset.collection_name | `--collection` | Required and recorded | none | +| dataset.num_shards | `--num-shards` | Permitted and recorded | 1 | +| dataset.chunk_size | `--chunk-size` | Permitted and recorded | 10,000 through `mlpstorage` | +| dataset.batch_size | `--batch-size` on `datagen` | Permitted and recorded | 1,000 through `mlpstorage` | +| dataset.vector_dtype | `--vector-dtype` | Permitted and recorded | `FLOAT_VECTOR` | +| *Storage* | | | | +| storage.storage_root | deployment and system-description fields | Required by the applicable POSIX/object rules | none | +| storage.storage_type | positional `file` or `object` | Required on `datagen` and `run` | none | + +The direct `load_vdb.py` script has different load batch/chunk fallbacks +(10,000 and 1,000,000). Those direct-script values do not replace the effective +values passed by the integrated wrapper. All submitted results must record the +effective values that actually reached the workload. + +5.6.5. **vdbOpenSubmissionParameters** -- OPEN submissions may use the full index set registered by `VDB_INDEX_TYPES`, including `IVF_FLAT`, `IVF_SQ8`, and `FLAT`, and may tune the build/search controls exposed by the OPEN CLI. A database engine is valid only when it is implemented and registered in `VDB_ENGINES`; the current integrated registry contains only `milvus`. OPEN does not by itself make an unimplemented backend valid. The *submission validator* must verify the recorded engine and index against the implemented registry and must require disclosure of all effective parameters. OPEN results must still meet the applicable recall and reporting rules. + +**Table: Additional VectorDB parameter surface for OPEN** + +| Parameter | Current CLI representation | OPEN treatment | Current fallback | +|---|---|---|---:| +| database.database | `--vdb-engine` | Any implemented/registered engine; currently `milvus` | `milvus` | +| database.host | `--host` | Permitted and disclosed | `127.0.0.1` | +| database.port | `--port` | Permitted and disclosed | 19530 | +| index.index_type | `--vdb-index` / matching `--index-type` | Full registered index set | `DISKANN` | +| index.metric_type | `--metric-type` | Permitted and disclosed | `COSINE` | +| index.max_degree | `--max-degree` | DiskANN/AISAQ build tuning | 16 | +| index.search_list_size | `--search-list-size` | DiskANN/AISAQ build tuning | 200 | +| index.M | `--M` | HNSW build tuning | 16 | +| index.ef_construction | `--ef-construction` | HNSW build tuning | 200 | +| index.inline_pq | `--inline-pq` | AISAQ build tuning | 16 | +| search.search_ef | `--search-ef` | Search tuning | 200 | + +# 6. Validating the KVCache Options + +## 6.1. KVCache Sizing Options + +## 6.2. KVCache Generation Options + +## 6.3. KVCache Run Options + +## 6.4. KVCache Access Via POSIX API Options + +## 6.5. KVCache Access Via Object API Options + +## 6.6. KVCache OPEN versus CLOSED Options + diff --git a/vdb_benchmark/README.md b/vdb_benchmark/README.md index 9768ac5a..9ea7892c 100644 --- a/vdb_benchmark/README.md +++ b/vdb_benchmark/README.md @@ -1,18 +1,17 @@ # Vector Database Benchmark Tool -Benchmarks and compares vector database performance for MLPerf Storage. Currently -supports Milvus with DiskANN, HNSW, AISAQ, FLAT, and IVF-style indexes. +Benchmarks and compares vector database performance for MLPerf Storage. Currently supports Milvus with DiskANN, HNSW, AISAQ, FLAT, and IVF-style indexes. The benchmark can be run in two ways: 1. Directly with the scripts in `vdb_benchmark/vdbbench/` -2. Through the MLPerf Storage CLI with `./mlpstorage vectordb` +2. Through the MLPerf Storage CLI with `./mlpstorage vectordb` The `mlpstorage` path is recommended for standard benchmark workflows. > The modular backend-agnostic runner is currently a standalone preview. > It is invoked with `python -m vdbbench.benchmark`. -> The existing `./mlpstorage vectordb` command continues to use the Milvus-oriented scripts until the modular runner is integrated. +> The existing `./mlpstorage vectordb` command continues to use the Milvus-oriented scripts until the modular runner is integrated. --- @@ -81,8 +80,8 @@ The `mlpstorage` path is recommended for standard benchmark workflows. | `pyyaml` | ≥ 6.0 | YAML config support | | `tabulate` | ≥ 0.9.0 | Collection info table display | -The `datasize` command does not require Milvus or `pymilvus`. Load and run -commands require a running Milvus server. +The `datasize` command does not require Milvus or `pymilvus`. +Load and run commands require a running Milvus server. ### Clone the Repository @@ -95,9 +94,7 @@ cd storage ## 2. Deploy Milvus -A running Milvus instance is required for all load (`datagen`) and benchmark -(`run`) commands. This section applies to both the mlpstorage CLI and direct -script paths. +A running Milvus instance is required for all load (`datagen`) and benchmark (`run`) commands. This section applies to both the mlpstorage CLI and direct script paths. Standalone Milvus stacks are available in the `vdb_benchmark/stacks` directory: @@ -114,8 +111,7 @@ vdb_benchmark/stacks/ └── docker-compose-s3.yml ``` -For each specific instance, copy the `.env.example` file to `.env` and update -the values as needed. +For each specific instance, copy the `.env.example` file to `.env` and update the values as needed. ### Option A: Local Storage with MinIO @@ -124,9 +120,7 @@ cp vdb_benchmark/stacks/milvus/standalone/minio/.env.example \ vdb_benchmark/stacks/milvus/standalone/minio/.env ``` -The compose file uses `/mnt/vdb` as the root directory for Docker volumes. Set -`DOCKER_VOLUME_DIRECTORY` in the `.env` file or edit the compose file to point to -your target storage location. +The compose file uses `/mnt/vdb` as the root directory for Docker volumes. Set `DOCKER_VOLUME_DIRECTORY` in the `.env` file or edit the compose file to point to your target storage location. The stack creates three containers: @@ -173,8 +167,7 @@ docker-compose -f vdb_benchmark/stacks/milvus/standalone/s3/docker-compose-s3.ym docker ps -a ``` -All three containers (`milvus-etcd`, `milvus-minio`, `milvus-standalone`) should -show healthy/running. +All three containers (`milvus-etcd`, `milvus-minio`, `milvus-standalone`) should show healthy/running. The default Milvus endpoint is: @@ -184,10 +177,9 @@ The default Milvus endpoint is: --- -## 3. Quick Start +## 3. Quick Start — First Benchmark in 10 Minutes -This section gets you from zero to a working benchmark result on a standalone-system. -Assumes Milvus is set up as per section #2 instructions. +This section gets you from zero to a working benchmark result on a standalone-system. Assumes Milvus is set up as per section #2 instructions. ### Step 1 — Install @@ -200,7 +192,7 @@ uv pip install -e ./vdb_benchmark Verify: ```bash -./mlpstorage vectordb --help +./mlpstorage open vectordb --help ``` ### Step 2 — Start Milvus @@ -218,15 +210,22 @@ Wait for healthy status: docker ps -a ``` -All three containers (`milvus-etcd`, `milvus-minio`, `milvus-standalone`) should -show healthy/running. The default endpoint is `127.0.0.1:19530`. +All three containers (`milvus-etcd`, `milvus-minio`, `milvus-standalone`) should show healthy/running. -### Step 3 — Load 50K vectors (smoke test) +The default endpoint is `127.0.0.1:19530`. + +### Step 3 — Initialize the results directory + +```bash +./mlpstorage init MLCommons /tmp/vdb_results +``` + +### Step 4 — Load 50K vectors (smoke test) ```bash -./mlpstorage vectordb datagen \ - --file \ - --open \ +./mlpstorage open vectordb datagen file \ + --vdb-engine milvus \ + --vdb-index DISKANN \ --host 127.0.0.1 \ --port 19530 \ --config default \ @@ -235,27 +234,29 @@ show healthy/running. The default endpoint is `127.0.0.1:19530`. --dimension 1536 \ --num-shards 1 \ --force \ + --systemname vdb_smoke_system \ --results-dir /tmp/vdb_results ``` -### Step 4 — Run a 30-second benchmark +### Step 5 — Run a 30-second benchmark ```bash -./mlpstorage vectordb run \ - --file \ - --open \ +./mlpstorage open vectordb run file \ + --vdb-engine milvus \ + --vdb-index DISKANN \ --host 127.0.0.1 \ --port 19530 \ --config default \ --collection mlps_smoke \ - --mode timed \ + --benchmark-mode timed \ --runtime 30 \ --num-query-processes 2 \ --batch-size 10 \ + --systemname vdb_smoke_system \ --results-dir /tmp/vdb_results ``` -### Step 5 — Check results +### Step 6 — Check results ```bash python - <<'PY' @@ -263,10 +264,11 @@ import json from pathlib import Path stats_files = sorted( - Path("/tmp/vdb_results").glob("**/vectordb/simple/statistics.json") + Path("/tmp/vdb_results").glob( + "**/vector_database/milvus/DISKANN/run/**/statistics.json" + ) ) assert stats_files, "No statistics.json found" - stats = json.loads(stats_files[-1].read_text()) print(f"Throughput: {stats['throughput_qps']:.1f} QPS") print(f"P95 latency: {stats['p95_latency_ms']:.2f} ms") @@ -274,15 +276,15 @@ print(f"Total queries: {stats['total_queries']}") PY ``` -If you see QPS and latency numbers, your setup is working. Continue to -[Section 4](#4-recommended-path-mlpstorage-cli) for full documentation. +If you see QPS and latency numbers, your setup is working. + +Continue to [Section 4](#4-recommended-path-mlpstorage-cli) for full documentation. --- ## 4. Recommended Path: mlpstorage CLI -This section covers the complete workflow using the `mlpstorage` CLI. This is the -recommended approach for standard benchmark workflows. +This section covers the complete workflow using the `mlpstorage` CLI. This is the recommended approach for standard benchmark workflows. ### 4.1 Installation @@ -301,10 +303,18 @@ uv pip install -e ./vdb_benchmark This makes the following commands available: ```bash -./mlpstorage vectordb --help -./mlpstorage vectordb datasize --help -./mlpstorage vectordb datagen --help -./mlpstorage vectordb run --help +./mlpstorage open vectordb --help +./mlpstorage open vectordb datasize --help +./mlpstorage open vectordb datagen --help +./mlpstorage open vectordb run --help +./mlpstorage closed vectordb run --help +./mlpstorage whatif vectordb run --help +``` + +Before running commands that write benchmark output, initialize the results directory once: + +```bash +./mlpstorage init MLCommons /tmp/vdb_results ``` The distributed VectorDB launcher additionally provides: @@ -327,16 +337,14 @@ uv run vdb-aggregate --help ### 4.2 Estimate Storage (datasize) -This step is optional. It is pure math and does not require a running Milvus -instance. +This step is optional. It is pure math and does not require a running Milvus instance. ```bash -./mlpstorage vectordb datasize \ - --file \ - --open \ +./mlpstorage open vectordb datasize \ + --vdb-engine milvus \ + --vdb-index DISKANN \ --dimension 1536 \ --num-vectors 10000000 \ - --index-type DISKANN \ --num-shards 10 ``` @@ -357,37 +365,39 @@ Estimated total: 798.72 GB #### Load using the default config (1M vectors) ```bash -./mlpstorage vectordb datagen \ - --file \ - --open \ +./mlpstorage open vectordb datagen file \ + --vdb-engine milvus \ + --vdb-index DISKANN \ --host 127.0.0.1 \ --port 19530 \ --config default \ --collection mlps_1m_1536dim_uniform_diskann \ --force \ + --systemname vdb_system \ --results-dir /tmp/vdb_results ``` #### Load using the 10M config ```bash -./mlpstorage vectordb datagen \ - --file \ - --open \ +./mlpstorage open vectordb datagen file \ + --vdb-engine milvus \ + --vdb-index DISKANN \ --host 127.0.0.1 \ --port 19530 \ --config 10m \ --collection mlps_10m_1536dim_uniform_diskann \ --force \ + --systemname vdb_system \ --results-dir /tmp/vdb_results ``` #### Override vector count for quick testing ```bash -./mlpstorage vectordb datagen \ - --file \ - --open \ +./mlpstorage open vectordb datagen file \ + --vdb-engine milvus \ + --vdb-index DISKANN \ --host 127.0.0.1 \ --port 19530 \ --config default \ @@ -396,27 +406,23 @@ Estimated total: 798.72 GB --dimension 1536 \ --num-shards 1 \ --force \ + --systemname vdb_system \ --results-dir /tmp/vdb_results ``` #### Notes -- The `--config` argument refers to YAML files in `configs/vectordbbench/` - without the `.yaml` extension. +- The `--config` argument refers to YAML files in `configs/vectordbbench/` without the `.yaml` extension. - The `--force` flag drops and recreates the collection if it already exists. -- See [Dimension Consistency](#dimension-consistency) for important rules about - keeping dimensions aligned between load and run. +- See [Dimension Consistency](#dimension-consistency) for important rules about keeping dimensions aligned between load and run. --- ### 4.4 Compact -The load script performs compaction automatically when enabled in the config or -when `--compact` is passed. +The load script performs compaction automatically when enabled in the config or when `--compact` is passed. Compaction runs as part of the `datagen` workflow. No separate command is needed unless the load command exits early. -Compaction runs as part of the `datagen` workflow. No separate command is needed -unless the load command exits early. See -[Section 5.3](#53-compact) for manual compaction if needed. +See [Section 5.3](#53-compact) for manual compaction if needed. --- @@ -424,44 +430,46 @@ unless the load command exits early. See #### Simple benchmark modes -| `mlpstorage` mode | Script | Purpose | -|-------------------|--------|---------| +| `--benchmark-mode` value | Script | Purpose | +|--------------------------|--------|---------| | `timed` | `vdbbench` | Run for a fixed duration | | `query_count` | `vdbbench` | Run exactly N total queries | ##### Timed mode ```bash -./mlpstorage vectordb run \ - --file \ - --open \ +./mlpstorage open vectordb run file \ + --vdb-engine milvus \ + --vdb-index DISKANN \ --host 127.0.0.1 \ --port 19530 \ --config default \ --collection mlps_1m_1536dim_uniform_diskann \ - --mode timed \ + --benchmark-mode timed \ --runtime 120 \ --num-query-processes 4 \ --batch-size 10 \ --report-count 100 \ + --systemname vdb_system \ --results-dir /tmp/vdb_results ``` ##### Query-count mode ```bash -./mlpstorage vectordb run \ - --file \ - --open \ +./mlpstorage open vectordb run file \ + --vdb-engine milvus \ + --vdb-index DISKANN \ --host 127.0.0.1 \ --port 19530 \ --config default \ --collection mlps_1m_1536dim_uniform_diskann \ - --mode query_count \ + --benchmark-mode query_count \ --queries 10000 \ --num-query-processes 4 \ --batch-size 10 \ --report-count 100 \ + --systemname vdb_system \ --results-dir /tmp/vdb_results ``` @@ -475,19 +483,20 @@ Use enhanced mode for: * richer disk and memory reporting * comparing index/search configurations -Enhanced mode is selected with `--mode sweep`: +Enhanced mode is selected with `--benchmark-mode sweep`: ```bash -./mlpstorage vectordb run \ - --file \ - --open \ +./mlpstorage open vectordb run file \ + --vdb-engine milvus \ + --vdb-index DISKANN \ --host 127.0.0.1 \ --port 19530 \ --config default \ --collection mlps_1m_1536dim_uniform_diskann \ - --mode sweep \ + --benchmark-mode sweep \ --queries 10000 \ --num-query-processes 4 \ + --systemname vdb_system \ --results-dir /tmp/vdb_results ``` @@ -503,8 +512,7 @@ Enhanced mode is selected with `--mode sweep`: ## 5. Alternative Path: Direct Scripts -This section covers the complete workflow using the Python scripts directly, -without the `mlpstorage` CLI wrapper. +This section covers the complete workflow using the Python scripts directly, without the `mlpstorage` CLI wrapper. ### 5.1 Installation @@ -530,16 +538,13 @@ uv run vdbbench --help uv run enhanced-bench --help ``` -**Note:** You still need a running Milvus instance. Follow the Docker setup -in [Section 2](#2-deploy-milvus). +**Note:** You still need a running Milvus instance. Follow the Docker setup in [Section 2](#2-deploy-milvus). --- ### 5.2 Load Vectors -> **Working directory:** All `python vdbbench/...` commands in this section -> assume you are in `storage/vdb_benchmark/`. Console scripts (`uv run load-vdb`, -> etc.) work from any directory. +> **Working directory:** All `python vdbbench/...` commands in this section assume you are in `storage/vdb_benchmark/`. Console scripts (`uv run load-vdb`, etc.) work from any directory. #### Load using a YAML config @@ -581,9 +586,9 @@ Direct script configs live in `vdbbench/configs/` (relative to `storage/vdb_benc ```text vdbbench/configs/ -├── 10m_diskann.yaml (10M vectors, 10 shards, 1536 dim) +├── 10m_diskann.yaml (10M vectors, 10 shards, 1536 dim) ├── 10m_hnsw.yaml -├── 1m_diskann.yaml (1M vectors, 1 shard, 1536 dim) +├── 1m_diskann.yaml (1M vectors, 1 shard, 1536 dim) ├── 1m_diskann_512dim.yaml ├── 1m_hnsw.yaml └── 1m_aisaq_512dim.yaml @@ -593,8 +598,7 @@ vdbbench/configs/ ### 5.3 Compact -The load script performs compaction automatically when enabled in the config or -when `--compact` is passed. +The load script performs compaction automatically when enabled in the config or when `--compact` is passed. If the load command exits early, run compaction manually: @@ -655,8 +659,7 @@ uv run enhanced-bench \ --out-dir /tmp/vdbbench_results ``` -See [Section 9](#9-enhanced-benchmark-full-reference) for full parameter -reference and execution paths. +See [Section 9](#9-enhanced-benchmark-full-reference) for full parameter reference and execution paths. --- @@ -665,25 +668,32 @@ reference and execution paths. ### Available Commands ```bash -./mlpstorage vectordb --help -./mlpstorage vectordb datasize --help -./mlpstorage vectordb datagen --help -./mlpstorage vectordb run --help +./mlpstorage open vectordb --help +./mlpstorage open vectordb datasize --help +./mlpstorage open vectordb datagen --help +./mlpstorage open vectordb run --help +./mlpstorage closed vectordb run --help +./mlpstorage whatif vectordb run --help ``` ### Important Terminology VectorDB uses two similar-looking host flags with different meanings. -| Flag | Meaning | -|------|---------| +| Flag / positional | Meaning | +|-------------------|---------| +| `open`, `closed`, `whatif` | Top-level benchmark mode before `vectordb` | +| `file` / `object` | Positional storage mode selector for `datagen` and `run` | +| `--vdb-engine` | Vector database engine identity; currently `milvus` | +| `--vdb-index` | Result identity index family, for example `DISKANN`, `HNSW`, or `AISAQ` | +| `--index-type` | Milvus implementation index type for `datasize` and `datagen`; defaults to `--vdb-index` when omitted | | `--host` / `-s` | Milvus database endpoint host | | `--port` / `-p` | Milvus database endpoint port | | `--hosts` | Benchmark client hosts used by MPI | | `--npernode` | MPI ranks to start on each benchmark client host | | `--num-query-processes` | Local Python query workers inside each MPI rank | -| `--file` | POSIX/file storage mode selector for `mlpstorage` | -| `--open` | Acknowledge OPEN category execution | +| `--benchmark-mode` | VectorDB run mode: `timed`, `query_count`, or `sweep` | +| `--systemname` | System-under-test directory name under `results/` | | `--results-dir` | Root directory for benchmark output | **Do not confuse `--host` and `--hosts`.** @@ -696,16 +706,13 @@ VectorDB uses two similar-looking host flags with different meanings. Effective distributed query workers: ```text -effective_workers = - len(--hosts) * --npernode * --num-query-processes +effective_workers = len(--hosts) * --npernode * --num-query-processes ``` Example: ```text ---hosts node01 node02 ---npernode 2 ---num-query-processes 4 +--hosts node01 node02 --npernode 2 --num-query-processes 4 ``` starts: @@ -749,12 +756,9 @@ Custom configs can be added to the same directory. The vector dimension must be consistent between data loading and benchmarking. -If you override `--dimension` during `datagen`, the config YAML used for `run` -must specify the same dimension. Otherwise, Milvus will reject queries with a -vector dimension mismatch. +If you override `--dimension` during `datagen`, the config YAML used for `run` must specify the same dimension. Otherwise, Milvus will reject queries with a vector dimension mismatch. -The safest approach is to use the same `--config` for both `datagen` and `run`, -or create a dedicated config YAML for non-default dimensions. +The safest approach is to use the same `--config` for both `datagen` and `run`, or create a dedicated config YAML for non-default dimensions. --- @@ -764,15 +768,14 @@ or create a dedicated config YAML for non-default dimensions. For multi-node runs: -1. Run `./mlpstorage vectordb ...` from one launcher host. +1. Run `./mlpstorage open vectordb ...` from one launcher host. 2. The launcher host participates in the benchmark. 3. Passwordless SSH must work from the launcher to all hosts listed in `--hosts`. 4. The repository path must be identical on every benchmark client host. 5. The same `uv` environment must be installed on every benchmark client host. 6. `mpiexec` must be installed and available on every benchmark client host. 7. The `--results-dir` path must be visible at the same path from every host. -8. The Milvus endpoint given by `--host` and `--port` must be reachable from - every benchmark client host. +8. The Milvus endpoint given by `--host` and `--port` must be reachable from every benchmark client host. #### Install on every benchmark client host @@ -806,8 +809,13 @@ mpiexec -n 2 -hosts node01,node02 \ ### 7.2 Distributed Load -Distributed load uses MPI to start one or more VectorDB loader ranks across -benchmark client hosts. +Distributed load uses MPI to start one or more VectorDB loader ranks across benchmark client hosts. + +Before using `/shared/vdb_results`, initialize it once: + +```bash +./mlpstorage init MLCommons /shared/vdb_results +``` #### Rank behavior @@ -833,14 +841,14 @@ rank 0: #### Command ```bash -./mlpstorage vectordb datagen \ - --file \ - --open \ +./mlpstorage open vectordb datagen file \ --distributed \ --mpi-impl mpich \ --mpi-bin mpiexec \ --hosts node01 node02 \ --npernode 1 \ + --vdb-engine milvus \ + --vdb-index DISKANN \ --host 10.0.0.10 \ --port 19530 \ --config default \ @@ -853,20 +861,21 @@ rank 0: --batch-size 1000 \ --chunk-size 10000 \ --force \ + --systemname vdb_multinode_system \ --results-dir /shared/vdb_results ``` #### Example with two MPI ranks per host ```bash -./mlpstorage vectordb datagen \ - --file \ - --open \ +./mlpstorage open vectordb datagen file \ --distributed \ --mpi-impl mpich \ --mpi-bin mpiexec \ --hosts node01 node02 \ --npernode 2 \ + --vdb-engine milvus \ + --vdb-index DISKANN \ --host 10.0.0.10 \ --port 19530 \ --config default \ @@ -877,16 +886,16 @@ rank 0: --batch-size 1000 \ --chunk-size 10000 \ --force \ + --systemname vdb_multinode_system \ --results-dir /shared/vdb_results ``` -With `--hosts node01 node02` and `--npernode 2`, the distributed load starts -four MPI ranks. +With `--hosts node01 node02` and `--npernode 2`, the distributed load starts four MPI ranks. #### Output structure ```text -/shared/vdb_results//vectordb/load/ +/shared/vdb_results/open/MLCommons/results/vdb_multinode_system/vector_database/milvus/DISKANN/datagen// ├── load_statistics.json ├── vdb_multi_node_summary.json ├── rank_0/ @@ -926,31 +935,32 @@ vectors_per_second = inserted_vectors / total_time_seconds ### 7.3 Distributed Simple Benchmark Distributed simple benchmark mode starts one `vdbbench` instance per MPI rank. -Each rank writes rank-local CSV, recall, and statistics files. The launcher then -aggregates the rank outputs. + +Each rank writes rank-local CSV, recall, and statistics files. The launcher then aggregates the rank outputs. #### Timed mode In timed mode, every MPI rank runs for the requested runtime. ```bash -./mlpstorage vectordb run \ - --file \ - --open \ +./mlpstorage open vectordb run file \ --distributed \ --mpi-impl mpich \ --mpi-bin mpiexec \ --hosts node01 node02 \ --npernode 2 \ + --vdb-engine milvus \ + --vdb-index DISKANN \ --host 10.0.0.10 \ --port 19530 \ --config default \ --collection mlps_1m_1536dim_uniform_diskann \ - --mode timed \ + --benchmark-mode timed \ --runtime 120 \ --num-query-processes 2 \ --batch-size 10 \ --report-count 100 \ + --systemname vdb_multinode_system \ --results-dir /shared/vdb_results ``` @@ -962,44 +972,42 @@ This starts: #### Query-count mode -In query-count mode, `--queries` is interpreted as the global query count. The -MPI wrapper splits the query count across ranks. +In query-count mode, `--queries` is interpreted as the global query count. The MPI wrapper splits the query count across ranks. For example: ```text ---queries 100000 ---hosts node01 node02 ---npernode 2 +--queries 100000 --hosts node01 node02 --npernode 2 ``` starts four MPI ranks, and each rank receives approximately 25,000 queries. ```bash -./mlpstorage vectordb run \ - --file \ - --open \ +./mlpstorage open vectordb run file \ --distributed \ --mpi-impl mpich \ --mpi-bin mpiexec \ --hosts node01 node02 \ --npernode 2 \ + --vdb-engine milvus \ + --vdb-index DISKANN \ --host 10.0.0.10 \ --port 19530 \ --config default \ --collection mlps_1m_1536dim_uniform_diskann \ - --mode query_count \ + --benchmark-mode query_count \ --queries 100000 \ --num-query-processes 2 \ --batch-size 10 \ --report-count 100 \ + --systemname vdb_multinode_system \ --results-dir /shared/vdb_results ``` #### Output structure ```text -/shared/vdb_results//vectordb/simple/ +/shared/vdb_results/open/MLCommons/results/vdb_multinode_system/vector_database/milvus/DISKANN/run// ├── statistics.json ├── vdb_multi_node_summary.json ├── rank_0/ @@ -1066,49 +1074,49 @@ recall_by_query ### 7.4 Distributed Enhanced / Sweep Benchmark -Distributed enhanced benchmark mode starts one `enhanced-bench` instance per MPI -rank. Each rank writes enhanced-bench output under a rank-local output directory. -The launcher then groups and aggregates rank outputs by parameter set. +Distributed enhanced benchmark mode starts one `enhanced-bench` instance per MPI rank. Each rank writes enhanced-bench output under a rank-local output directory. The launcher then groups and aggregates rank outputs by parameter set. #### Command ```bash -./mlpstorage vectordb run \ - --file \ - --open \ +./mlpstorage open vectordb run file \ --distributed \ --mpi-impl mpich \ --mpi-bin mpiexec \ --hosts node01 node02 \ --npernode 1 \ + --vdb-engine milvus \ + --vdb-index DISKANN \ --host 10.0.0.10 \ --port 19530 \ --config default \ --collection mlps_1m_1536dim_uniform_diskann \ - --mode sweep \ + --benchmark-mode sweep \ --queries 10000 \ --num-query-processes 4 \ + --systemname vdb_multinode_system \ --results-dir /shared/vdb_results ``` #### Example with four total MPI ranks ```bash -./mlpstorage vectordb run \ - --file \ - --open \ +./mlpstorage open vectordb run file \ --distributed \ --mpi-impl mpich \ --mpi-bin mpiexec \ --hosts node01 node02 \ --npernode 2 \ + --vdb-engine milvus \ + --vdb-index DISKANN \ --host 10.0.0.10 \ --port 19530 \ --config default \ --collection mlps_1m_1536dim_uniform_diskann \ - --mode sweep \ + --benchmark-mode sweep \ --queries 20000 \ --num-query-processes 2 \ + --systemname vdb_multinode_system \ --results-dir /shared/vdb_results ``` @@ -1121,7 +1129,7 @@ With `--hosts node01 node02`, `--npernode 2`, and `--num-query-processes 2`: #### Output structure ```text -/shared/vdb_results//vectordb/enhanced/ +/shared/vdb_results/open/MLCommons/results/vdb_multinode_system/vector_database/milvus/DISKANN/run// ├── enhanced_statistics.json ├── vdb_multi_node_summary.json ├── rank_0/ @@ -1174,32 +1182,26 @@ p99_latency_ms = max(rank p99_latency_ms) recall_mean = query-count-weighted mean, or exact when per-query recall is present ``` -For simple-bench, p95/p99 latency percentiles are exact because raw per-batch CSV -rows are available. For enhanced-bench, p95/p99 are conservative max-rank values -unless raw latency samples are also emitted by the enhanced output. +For simple-bench, p95/p99 latency percentiles are exact because raw per-batch CSV rows are available. For enhanced-bench, p95/p99 are conservative max-rank values unless raw latency samples are also emitted by the enhanced output. --- ### 7.5 Metrics Aggregation -Distributed VectorDB runs use rank-local output directories and a final -aggregation step. - -The aggregation script is: +Distributed VectorDB runs use rank-local output directories and a final aggregation step. The aggregation script is: ```bash uv run vdb-aggregate ``` -It is normally invoked automatically by `./mlpstorage vectordb`. It can also be -run manually. +It is normally invoked automatically by `./mlpstorage open vectordb`. It can also be run manually. #### Manual load aggregation ```bash uv run vdb-aggregate \ --phase load \ - --base-output-dir /shared/vdb_results//vectordb/load \ + --base-output-dir /shared/vdb_results/open/MLCommons/results/vdb_multinode_system/vector_database/milvus/DISKANN/datagen/ \ --expected-ranks 2 ``` @@ -1208,7 +1210,7 @@ uv run vdb-aggregate \ ```bash uv run vdb-aggregate \ --phase simple \ - --base-output-dir /shared/vdb_results//vectordb/simple \ + --base-output-dir /shared/vdb_results/open/MLCommons/results/vdb_multinode_system/vector_database/milvus/DISKANN/run/ \ --expected-ranks 2 ``` @@ -1217,7 +1219,7 @@ uv run vdb-aggregate \ ```bash uv run vdb-aggregate \ --phase enhanced \ - --base-output-dir /shared/vdb_results//vectordb/enhanced \ + --base-output-dir /shared/vdb_results/open/MLCommons/results/vdb_multinode_system/vector_database/milvus/DISKANN/run/ \ --expected-ranks 2 ``` @@ -1225,12 +1227,9 @@ uv run vdb-aggregate \ ### 7.6 Disk I/O Deduplication -Disk I/O counters are node-local. If multiple MPI ranks run on the same host, -summing every rank's `/proc/diskstats` delta would double-count that host's disk -I/O. +Disk I/O counters are node-local. If multiple MPI ranks run on the same host, summing every rank's `/proc/diskstats` delta would double-count that host's disk I/O. -Distributed aggregation counts disk I/O only once per benchmark client host, -using the rank where: +Distributed aggregation counts disk I/O only once per benchmark client host, using the rank where: ```text local_rank == 0 @@ -1242,28 +1241,24 @@ The aggregated `disk_io` field records this policy. ### 7.7 Ground Truth and Recall -Recall is computed outside the timed query loop so it does not inflate latency -measurements. +Recall is computed outside the timed query loop so it does not inflate latency measurements. -The benchmark uses a FLAT ground-truth collection for exact nearest-neighbor -results. +The benchmark uses a FLAT ground-truth collection for exact nearest-neighbor results. Recommended ground-truth collection name: ```text -_flat_gt +_flat_gt ``` -Distributed wrappers should avoid multiple ranks racing to create/drop the same -FLAT ground-truth collection. The orchestration flow is: +Distributed wrappers should avoid multiple ranks racing to create/drop the same FLAT ground-truth collection. -```text -rank 0: - create or validate FLAT ground-truth collection +The orchestration flow is: -non-rank-0: - validate existing FLAT ground-truth collection - run with --no-create-flat +```text +rank 0: create or validate FLAT ground-truth collection +non-rank-0: validate existing FLAT ground-truth collection +run with --no-create-flat ``` Rank-local recall files include: @@ -1287,8 +1282,7 @@ per_query_recall recall_by_query ``` -The `per_query_recall` and `recall_by_query` fields are used for exact -multi-rank recall aggregation. +The `per_query_recall` and `recall_by_query` fields are used for exact multi-rank recall aggregation. --- @@ -1311,45 +1305,47 @@ Open MPI can be selected with: Example: ```bash -./mlpstorage vectordb run \ - --file \ - --open \ +./mlpstorage open vectordb run file \ --distributed \ --mpi-impl openmpi \ --mpi-bin mpirun \ --hosts node01 node02 \ --npernode 1 \ + --vdb-engine milvus \ + --vdb-index DISKANN \ --host 10.0.0.10 \ --port 19530 \ --config default \ --collection mlps_1m_1536dim_uniform_diskann \ - --mode timed \ + --benchmark-mode timed \ --runtime 120 \ --num-query-processes 2 \ --batch-size 10 \ + --systemname vdb_multinode_system \ --results-dir /shared/vdb_results ``` Additional MPI arguments can be passed with `--mpi-params`: ```bash -./mlpstorage vectordb run \ - --file \ - --open \ +./mlpstorage open vectordb run file \ --distributed \ --mpi-impl mpich \ --mpi-bin mpiexec \ --hosts node01 node02 \ --npernode 1 \ - --mpi-params -env UCX_TLS tcp \ + --mpi-params="-env UCX_TLS tcp" \ + --vdb-engine milvus \ + --vdb-index DISKANN \ --host 10.0.0.10 \ --port 19530 \ --config default \ --collection mlps_1m_1536dim_uniform_diskann \ - --mode query_count \ + --benchmark-mode query_count \ --queries 10000 \ --num-query-processes 2 \ --batch-size 10 \ + --systemname vdb_multinode_system \ --results-dir /shared/vdb_results ``` @@ -1360,50 +1356,55 @@ Additional MPI arguments can be passed with `--mpi-params`: ### Single-Node Example ```bash +# Initialize output root once. +./mlpstorage init MLCommons ~/vdb_results + # 1. Estimate storage. -./mlpstorage vectordb datasize \ - --file \ - --open \ +./mlpstorage open vectordb datasize \ + --vdb-engine milvus \ + --vdb-index DISKANN \ --dimension 1536 \ - --num-vectors 1000000 \ - --index-type DISKANN + --num-vectors 1000000 # 2. Load vectors. -./mlpstorage vectordb datagen \ - --file \ - --open \ +./mlpstorage open vectordb datagen file \ + --vdb-engine milvus \ + --vdb-index DISKANN \ --host 127.0.0.1 \ --port 19530 \ --config default \ --collection mlps_single_1m \ --force \ + --systemname vdb_single_system \ --results-dir ~/vdb_results # 3. Run simple benchmark. -./mlpstorage vectordb run \ - --file \ - --open \ +./mlpstorage open vectordb run file \ + --vdb-engine milvus \ + --vdb-index DISKANN \ --host 127.0.0.1 \ --port 19530 \ --config default \ --collection mlps_single_1m \ - --mode timed \ + --benchmark-mode timed \ --num-query-processes 2 \ --runtime 60 \ --batch-size 10 \ + --systemname vdb_single_system \ --results-dir ~/vdb_results # 4. Run enhanced benchmark. -./mlpstorage vectordb run \ - --file \ - --open \ +./mlpstorage open vectordb run file \ + --vdb-engine milvus \ + --vdb-index DISKANN \ --host 127.0.0.1 \ --port 19530 \ --config default \ --collection mlps_single_1m \ - --mode sweep \ + --benchmark-mode sweep \ --queries 10000 \ --num-query-processes 2 \ + --systemname vdb_single_system \ --results-dir ~/vdb_results # 5. View history. @@ -1416,15 +1417,18 @@ Additional MPI arguments can be passed with `--mpi-params`: # 1. Verify MPI. mpiexec -n 2 -hosts node01,node02 hostname -# 2. Load vectors across two benchmark client hosts. -./mlpstorage vectordb datagen \ - --file \ - --open \ +# 2. Initialize output root once. +./mlpstorage init MLCommons /shared/vdb_results + +# 3. Load vectors across two benchmark client hosts. +./mlpstorage open vectordb datagen file \ --distributed \ --mpi-impl mpich \ --mpi-bin mpiexec \ --hosts node01 node02 \ --npernode 1 \ + --vdb-engine milvus \ + --vdb-index DISKANN \ --host 10.0.0.10 \ --port 19530 \ --config default \ @@ -1433,43 +1437,46 @@ mpiexec -n 2 -hosts node01,node02 hostname --dimension 1536 \ --num-shards 4 \ --force \ + --systemname vdb_dist_system \ --results-dir /shared/vdb_results -# 3. Run distributed simple benchmark. -./mlpstorage vectordb run \ - --file \ - --open \ +# 4. Run distributed simple benchmark. +./mlpstorage open vectordb run file \ --distributed \ --mpi-impl mpich \ --mpi-bin mpiexec \ --hosts node01 node02 \ --npernode 1 \ + --vdb-engine milvus \ + --vdb-index DISKANN \ --host 10.0.0.10 \ --port 19530 \ --config default \ --collection mlps_dist_1m \ - --mode timed \ + --benchmark-mode timed \ --runtime 120 \ --num-query-processes 2 \ --batch-size 10 \ + --systemname vdb_dist_system \ --results-dir /shared/vdb_results -# 4. Run distributed enhanced benchmark. -./mlpstorage vectordb run \ - --file \ - --open \ +# 5. Run distributed enhanced benchmark. +./mlpstorage open vectordb run file \ --distributed \ --mpi-impl mpich \ --mpi-bin mpiexec \ --hosts node01 node02 \ --npernode 1 \ + --vdb-engine milvus \ + --vdb-index DISKANN \ --host 10.0.0.10 \ --port 19530 \ --config default \ --collection mlps_dist_1m \ - --mode sweep \ + --benchmark-mode sweep \ --queries 10000 \ --num-query-processes 2 \ + --systemname vdb_dist_system \ --results-dir /shared/vdb_results ``` @@ -1477,12 +1484,9 @@ mpiexec -n 2 -hosts node01,node02 hostname ## 9. Enhanced Benchmark Full Reference -> **Working directory:** All commands below assume you are in -> `storage/vdb_benchmark/`. +> **Working directory:** All commands below assume you are in `storage/vdb_benchmark/`. -`enhanced_bench.py` merges the operational features of `simple_bench.py` with -advanced features for parameter sweeps, warm/cold cache regimes, budget mode, -YAML config, and memory estimation. +`enhanced_bench.py` merges the operational features of `simple_bench.py` with advanced features for parameter sweeps, warm/cold cache regimes, budget mode, YAML config, and memory estimation. ### Two Execution Paths @@ -1495,8 +1499,7 @@ The script automatically selects the path based on the flags provided. ### Path A — Runtime / Query-Count Mode -This path mimics `simple_bench.py`. It runs workers for a fixed duration or query -count, writes per-process CSV files, and aggregates latency and recall stats. +This path mimics `simple_bench.py`. It runs workers for a fixed duration or query count, writes per-process CSV files, and aggregates latency and recall stats. Create the FLAT ground-truth collection (first run only): @@ -1604,7 +1607,7 @@ python vdbbench/enhanced_bench.py \ | `--search-ef` | `200` | Search parameter override | | `--num-query-vectors` | `1000` | Pre-generated query vectors for recall | | `--recall-k` | `--search-limit` | k for recall@k | -| `--gt-collection` | `_flat_gt` | FLAT GT collection name | +| `--gt-collection` | `_flat_gt` | FLAT GT collection name | | `--auto-create-flat` | `False` | Auto-create FLAT GT collection from source | | `--no-create-flat` | `False` | Validate and reuse existing FLAT GT collection | | `--vector-dim` | `1536` | Vector dimension | @@ -1631,9 +1634,9 @@ statistics.json #### Sweep path ```text -combined_bench_.json -combined_bench_.csv -combined_bench_.sweep.csv +combined_bench_.json +combined_bench_.csv +combined_bench_.sweep.csv ``` --- @@ -1642,11 +1645,9 @@ combined_bench_.sweep.csv ### Recall Measurement -Recall is computed outside the timed benchmark loop so it does not inflate -latency measurements. +Recall is computed outside the timed benchmark loop so it does not inflate latency measurements. -The benchmark uses a FLAT ground-truth collection for exact nearest-neighbor -results. +The benchmark uses a FLAT ground-truth collection for exact nearest-neighbor results. Simple benchmark output includes: @@ -1670,13 +1671,11 @@ per_query_recall recall_by_query ``` -The `per_query_recall` and `recall_by_query` fields are used for exact -multi-rank aggregation. +The `per_query_recall` and `recall_by_query` fields are used for exact multi-rank aggregation. ### Disk I/O Metrics -Disk I/O is measured by reading `/proc/diskstats` before and after each -benchmark run. +Disk I/O is measured by reading `/proc/diskstats` before and after each benchmark run. Fields include: @@ -1689,8 +1688,7 @@ read_iops write_iops ``` -In distributed mode, disk I/O is aggregated once per benchmark client host to -avoid double-counting multiple MPI ranks on the same host. +In distributed mode, disk I/O is aggregated once per benchmark client host to avoid double-counting multiple MPI ranks on the same host. --- @@ -1725,26 +1723,33 @@ MpiContext(rank=1, world_size=2, local_rank=0, hostname='node02') ### 3. `mlpstorage` dry run -Use `--what-if` to inspect the generated command without running it. +Initialize the shared results directory once: + +```bash +./mlpstorage init MLCommons /shared/vdb_results +``` + +Use top-level `whatif` plus `--dry-run` to inspect the generated command without running it: ```bash -./mlpstorage vectordb run \ - --file \ - --open \ - --what-if \ +./mlpstorage whatif vectordb run file \ + --dry-run \ --distributed \ --mpi-impl mpich \ --mpi-bin mpiexec \ --hosts node01 node02 \ --npernode 1 \ + --vdb-engine milvus \ + --vdb-index DISKANN \ --host 10.0.0.10 \ --port 19530 \ --config default \ --collection mlps_smoke \ - --mode query_count \ + --benchmark-mode query_count \ --queries 100 \ --num-query-processes 1 \ --batch-size 10 \ + --systemname vdb_smoke_system \ --results-dir /shared/vdb_results ``` @@ -1753,14 +1758,16 @@ Use `--what-if` to inspect the generated command without running it. This uses MPI on localhost and is useful before testing multiple nodes. ```bash -./mlpstorage vectordb datagen \ - --file \ - --open \ +./mlpstorage init MLCommons /tmp/vdb_results + +./mlpstorage open vectordb datagen file \ --distributed \ --mpi-impl mpich \ --mpi-bin mpiexec \ --hosts localhost \ --npernode 2 \ + --vdb-engine milvus \ + --vdb-index DISKANN \ --host 127.0.0.1 \ --port 19530 \ --config default \ @@ -1771,28 +1778,31 @@ This uses MPI on localhost and is useful before testing multiple nodes. --batch-size 500 \ --chunk-size 1000 \ --force \ + --systemname vdb_smoke_system \ --results-dir /tmp/vdb_results ``` Then run a query-count benchmark: ```bash -./mlpstorage vectordb run \ - --file \ - --open \ +./mlpstorage open vectordb run file \ --distributed \ --mpi-impl mpich \ --mpi-bin mpiexec \ --hosts localhost \ --npernode 2 \ + --vdb-engine milvus \ + --vdb-index DISKANN \ --host 127.0.0.1 \ --port 19530 \ --config default \ --collection mlps_smoke_10k \ - --mode query_count \ + --benchmark-mode query_count \ --queries 200 \ + --vector-dim 128 \ --num-query-processes 1 \ --batch-size 10 \ + --systemname vdb_smoke_system \ --results-dir /tmp/vdb_results ``` @@ -1804,29 +1814,37 @@ import json from pathlib import Path stats_files = sorted( - Path("/tmp/vdb_results").glob("**/vectordb/simple/statistics.json") + Path("/tmp/vdb_results").glob( + "**/vector_database/milvus/DISKANN/run/**/statistics.json" + ) ) assert stats_files, "No distributed statistics.json found" - stats = json.loads(stats_files[-1].read_text()) assert stats["total_queries"] == 200 assert stats["mpi"]["partial_failure"] is False - print(json.dumps(stats, indent=2)[:2000]) PY ``` ### 5. Multi-node load test +Initialize the shared results directory once: + +```bash +./mlpstorage init MLCommons /shared/vdb_results +``` + +Load data: + ```bash -./mlpstorage vectordb datagen \ - --file \ - --open \ +./mlpstorage open vectordb datagen file \ --distributed \ --mpi-impl mpich \ --mpi-bin mpiexec \ --hosts node01 node02 \ --npernode 1 \ + --vdb-engine milvus \ + --vdb-index DISKANN \ --host 10.0.0.10 \ --port 19530 \ --config default \ @@ -1837,6 +1855,7 @@ PY --batch-size 1000 \ --chunk-size 10000 \ --force \ + --systemname vdb_multinode_system \ --results-dir /shared/vdb_results ``` @@ -1848,14 +1867,14 @@ import json from pathlib import Path load_files = sorted( - Path("/shared/vdb_results").glob("**/vectordb/load/load_statistics.json") + Path("/shared/vdb_results").glob( + "**/vector_database/milvus/DISKANN/datagen/**/load_statistics.json" + ) ) assert load_files, "No load_statistics.json found" - stats = json.loads(load_files[-1].read_text()) assert stats["inserted_vectors"] == 1000000 assert stats["mpi"]["partial_failure"] is False - print(json.dumps(stats, indent=2)[:2000]) PY ``` @@ -1863,53 +1882,55 @@ PY ### 6. Multi-node simple benchmark test ```bash -./mlpstorage vectordb run \ - --file \ - --open \ +./mlpstorage open vectordb run file \ --distributed \ --mpi-impl mpich \ --mpi-bin mpiexec \ --hosts node01 node02 \ --npernode 1 \ + --vdb-engine milvus \ + --vdb-index DISKANN \ --host 10.0.0.10 \ --port 19530 \ --config default \ --collection mlps_multinode_1m \ - --mode timed \ + --benchmark-mode timed \ --runtime 120 \ --num-query-processes 2 \ --batch-size 10 \ + --systemname vdb_multinode_system \ --results-dir /shared/vdb_results ``` ### 7. Multi-node enhanced benchmark test ```bash -./mlpstorage vectordb run \ - --file \ - --open \ +./mlpstorage open vectordb run file \ --distributed \ --mpi-impl mpich \ --mpi-bin mpiexec \ --hosts node01 node02 \ --npernode 1 \ + --vdb-engine milvus \ + --vdb-index DISKANN \ --host 10.0.0.10 \ --port 19530 \ --config default \ --collection mlps_multinode_1m \ - --mode sweep \ + --benchmark-mode sweep \ --queries 10000 \ --num-query-processes 2 \ + --systemname vdb_multinode_system \ --results-dir /shared/vdb_results ``` Expected files: ```text -/shared/vdb_results//vectordb/load/load_statistics.json -/shared/vdb_results//vectordb/simple/statistics.json -/shared/vdb_results//vectordb/enhanced/enhanced_statistics.json -/shared/vdb_results//vectordb/*/vdb_multi_node_summary.json +/shared/vdb_results/open/MLCommons/results/vdb_multinode_system/vector_database/milvus/DISKANN/datagen//load_statistics.json +/shared/vdb_results/open/MLCommons/results/vdb_multinode_system/vector_database/milvus/DISKANN/run//statistics.json +/shared/vdb_results/open/MLCommons/results/vdb_multinode_system/vector_database/milvus/DISKANN/run//enhanced_statistics.json +/shared/vdb_results/open/MLCommons/results/vdb_multinode_system/vector_database/milvus/DISKANN/run//vdb_multi_node_summary.json ``` --- @@ -1918,10 +1939,7 @@ Expected files: ### `vector dimension mismatch` -The dimension used for load and run does not match. - -Use the same config for both `datagen` and `run`, or update the config to match -the dimension passed to `datagen`. +The dimension used for load and run does not match. Use the same config for both `datagen` and `run`, or update the config to match the dimension passed to `datagen`. ### MPI launches only on one host @@ -1931,8 +1949,7 @@ Check: mpiexec -n 2 -hosts node01,node02 hostname ``` -If both lines show the same host, inspect the MPICH/Hydra host configuration and -SSH setup. +If both lines show the same host, inspect the MPICH/Hydra host configuration and SSH setup. ### Rank output is missing @@ -1954,8 +1971,9 @@ partial_failure ### Distributed aggregation cannot find files -Ensure `--results-dir` is visible at the same path from all benchmark client -hosts. For example, use a shared filesystem path such as: +Ensure `--results-dir` is visible at the same path from all benchmark client hosts. + +For example, use a shared filesystem path such as: ```text /shared/vdb_results @@ -1963,8 +1981,7 @@ hosts. For example, use a shared filesystem path such as: ### Recall is zero -Check that the FLAT ground-truth collection exists and contains the same vectors -as the ANN collection. +Check that the FLAT ground-truth collection exists and contains the same vectors as the ANN collection. Also check that rank-local `recall_stats.json` files contain non-empty: @@ -2006,11 +2023,11 @@ uv run vdb-aggregate --help ## 13. Contributing -Contributions are welcome. Pull requests that add or modify distributed -VectorDB behavior should include: +Contributions are welcome. Pull requests that add or modify distributed VectorDB behavior should include: * implementation changes * unit tests * single-host MPI smoke test results * multi-node test results when applicable * README updates +