[Question/Enhancement] Unconditional force-merge compaction in Milvus client `_optimize()` may be too costly and non-reproducible at 100M scale (LAION-100M)

[frankleaf]

Summary

In the Milvus client, _optimize() unconditionally triggers a force-merge compaction
before search for all non-GPU index types:

https://github.com/zilliztech/VectorDBBench/blob/main/vectordb_bench/backend/clients/milvus/milvus.py#L312-L316

MILVUS_FORCE_MERGE_TARGET_SIZE_MB = ((1 << 63) - 1) // (1024**2)   # line 21
...
self._wait_for_segments_sorted()
self._wait_for_index()
compaction_id = self.client.compact(
    self.collection_name, target_size=MILVUS_FORCE_MERGE_TARGET_SIZE_MB
)

Concerns at LAION-100M scale
I'd like to ask whether this is intended to run at 100M-vector scale (e.g. the
Performance768D100M / LAION-100M case), and whether it should be made configurable.

LAION-100M is 100M × 768-dim float32 ≈ 307 GB of raw vector data.

• Compaction cost: force merge rewrites the segments; at 300 GB+ this is a very long,
  IO/CPU-heavy operation on DataNodes. _optimize() is supposed to be a "prepare for
  search" step, but here it becomes a multi-hour data rewrite.
• Index rebuild memory: after merging into much larger segments, the vector index is
  rebuilt per segment. Index construction memory scales with segment size, so very large
  merged segments raise real OOM risk on the index/data node — especially for
  memory-hungry builds (HNSW, DiskANN).

Questions
1、Was unconditional force merge validated on the 100M-scale cases, and if so, what was
the observed compaction duration and peak DataNode / IndexNode memory?
2、Is the intent to normalize segment layout for fair comparison? If yes, wouldn't an
explicit, fixed target_size be more reproducible than the memory-dependent auto mode?

[xiaofan-luan]

for milvus, it's just tune to a reason size, doesn't mean you only merge to one segment.

we might end up with 10 30GB segments. and with more machines in your benchmark, it might be more.

if you have streaming use case you can skip the optimize stage.

[frankleaf]

@xiaofan-luan
Thanks for the reply. My understanding is that VDBBench does not enforce a fair or standardized requirement on the number of segments—each product can adjust this parameter based on its own configuration and hardware setup. In practice, however, we've observed that the number of segments in Milvus has a significant impact on performance.

[jamesgao-jpg]

@frankleaf Hi, just saw the issue. I think the concern is valid, and perhaps more so now that #840 is merged, because now the default behavior is that the force merge behavior is looped until the all potentially force-mergeable segments are force-merged to the minimum segment size permittable. This pr is merged due to observed problems about unstable compaction behaviors in current Milvus, which makes the benchmark results fluctuate. But, per your suggestion, I think a more reasonable approach might be this:

we make force merge a tunnable CLI toggle like already done with backends such as ES, and be explicit about the load performance impact of force merge behaviors. Users can either opt in for force merge if they desire the best search benchmark performance possible, or opt out for a sooner ready-to-search state but accept segments are not merged to their fullest potential.

Will be added soon to the main branch.