[Question/Enhancement] Unconditional force-merge compaction in Milvus client _optimize() may be too costly and non-reproducible at 100M scale (LAION-100M)

Summary

In the Milvus client, _optimize() unconditionally triggers a force-merge compaction
before search for all non-GPU index types:

https://github.com/zilliztech/VectorDBBench/blob/main/vectordb_bench/backend/clients/milvus/milvus.py#L312-L316

MILVUS_FORCE_MERGE_TARGET_SIZE_MB = ((1 << 63) - 1) // (1024**2)   # line 21
...
self._wait_for_segments_sorted()
self._wait_for_index()
compaction_id = self.client.compact(
    self.collection_name, target_size=MILVUS_FORCE_MERGE_TARGET_SIZE_MB
)

Concerns at LAION-100M scale
I'd like to ask whether this is intended to run at 100M-vector scale (e.g. the
Performance768D100M / LAION-100M case), and whether it should be made configurable.

LAION-100M is 100M × 768-dim float32 ≈ 307 GB of raw vector data.

Compaction cost: force merge rewrites the segments; at 300 GB+ this is a very long,
IO/CPU-heavy operation on DataNodes. _optimize() is supposed to be a "prepare for
search" step, but here it becomes a multi-hour data rewrite.
Index rebuild memory: after merging into much larger segments, the vector index is
rebuilt per segment. Index construction memory scales with segment size, so very large
merged segments raise real OOM risk on the index/data node — especially for
memory-hungry builds (HNSW, DiskANN).

Questions
1、Was unconditional force merge validated on the 100M-scale cases, and if so, what was
the observed compaction duration and peak DataNode / IndexNode memory?
2、Is the intent to normalize segment layout for fair comparison? If yes, wouldn't an
explicit, fixed target_size be more reproducible than the memory-dependent auto mode?

for milvus, it's just tune to a reason size, doesn't mean you only merge to one segment.

we might end up with 10 30GB segments. and with more machines in your benchmark, it might be more.

if you have streaming use case you can skip the optimize stage.

@xiaofan-luan

@xiaofan-luan
Thanks for the reply. My understanding is that VDBBench does not enforce a fair or standardized requirement on the number of segments—each product can adjust this parameter based on its own configuration and hardware setup. In practice, however, we've observed that the number of segments in Milvus has a significant impact on performance.

@frankleaf

@frankleaf Hi, just saw the issue. I think the concern is valid, and perhaps more so now that #840 is merged, because now the default behavior is that the force merge behavior is looped until the all potentially force-mergeable segments are force-merged to the minimum segment size permittable. This pr is merged due to observed problems about unstable compaction behaviors in current Milvus, which makes the benchmark results fluctuate. But, per your suggestion, I think a more reasonable approach might be this:

we make force merge a tunnable CLI toggle like already done with backends such as ES, and be explicit about the load performance impact of force merge behaviors. Users can either opt in for force merge if they desire the best search benchmark performance possible, or opt out for a sooner ready-to-search state but accept segments are not merged to their fullest potential.

Will be added soon to the main branch.

@frankleaf Hi, just saw the issue. I think the concern is valid, and perhaps more so now that #840 is merged, because now the default behavior is that the force merge behavior is looped until the all potentially force-mergeable segments are force-merged to the minimum segment size permittable. This pr is merged due to observed problems about unstable compaction behaviors in current Milvus, which makes the benchmark results fluctuate. But, per your suggestion, I think a more reasonable approach might be this:

we make force merge a tunnable CLI toggle like already done with backends such as ES, and be explicit about the load performance impact of force merge behaviors. Users can either opt in for force merge if they desire the best search benchmark performance possible, or opt out for a sooner ready-to-search state but accept segments are not merged to their fullest potential.

Will be added soon to the main branch.

Has this change been implemented yet? will you provide the CLI toggle for this configuration（skip force merge）?

@frankleaf Hi, just saw the issue. I think the concern is valid, and perhaps more so now that #840 is merged, because now the default behavior is that the force merge behavior is looped until the all potentially force-mergeable segments are force-merged to the minimum segment size permittable. This pr is merged due to observed problems about unstable compaction behaviors in current Milvus, which makes the benchmark results fluctuate. But, per your suggestion, I think a more reasonable approach might be this:
we make force merge a tunnable CLI toggle like already done with backends such as ES, and be explicit about the load performance impact of force merge behaviors. Users can either opt in for force merge if they desire the best search benchmark performance possible, or opt out for a sooner ready-to-search state but accept segments are not merged to their fullest potential.
Will be added soon to the main branch.

Has this change been implemented yet? will you provide the CLI toggle for this configuration（skip force merge）?

Yes, coming right up :)

@frankleaf Hi, just saw the issue. I think the concern is valid, and perhaps more so now that #840 is merged, because now the default behavior is that the force merge behavior is looped until the all potentially force-mergeable segments are force-merged to the minimum segment size permittable. This pr is merged due to observed problems about unstable compaction behaviors in current Milvus, which makes the benchmark results fluctuate. But, per your suggestion, I think a more reasonable approach might be this:
we make force merge a tunnable CLI toggle like already done with backends such as ES, and be explicit about the load performance impact of force merge behaviors. Users can either opt in for force merge if they desire the best search benchmark performance possible, or opt out for a sooner ready-to-search state but accept segments are not merged to their fullest potential.
Will be added soon to the main branch.

Has this change been implemented yet? will you provide the CLI toggle for this configuration（skip force merge）?

Yes, coming right up :)

Please ping me when ready. thank you!

Please take a look at #871

@jamesgao-jpg

@jamesgao-jpg Great! Supporting these two parameters will make it much more flexible

@jamesgao-jpg

@jamesgao-jpg May I ask when this PR is expected to be merged into the main branch? Thank you!

just merged :)

feat: add CLI control for Milvus force-merge size and toggle (#871)

just merged :)

That's great! I'll give it a try

Navigation Menu

Repository navigation

[Question/Enhancement] Unconditional force-merge compaction in Milvus client `_optimize()` may be too costly and non-reproducible at 100M scale (LAION-100M) #825

Summary

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Question/Enhancement] Unconditional force-merge compaction in Milvus client _optimize() may be too costly and non-reproducible at 100M scale (LAION-100M) #825

Description

Summary

Activity

xiaofan-luan commented on Jul 29, 2026

frankleaf commented on Jul 30, 2026

jamesgao-jpg commented on Aug 13, 2026

frankleaf commented on Sep 15, 2026

jamesgao-jpg commented on Sep 15, 2026

frankleaf commented on Sep 15, 2026

jamesgao-jpg commented on Sep 15, 2026

frankleaf commented on Sep 15, 2026

frankleaf commented on Sep 28, 2026

jamesgao-jpg commented on Oct 8, 2026

frankleaf commented on Oct 8, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

[Question/Enhancement] Unconditional force-merge compaction in Milvus client `_optimize()` may be too costly and non-reproducible at 100M scale (LAION-100M) #825