Recommended way to set up baseline + incremental sync between two Milvus standalone instances?

[gphmath]

Goal

I want to set up primary-standby replication / read-write separation between two Milvus standalone instances.

Deployment target:

• Two separate cloud hosts
• Each host runs one Milvus standalone instance
• Source Milvus: mainly for writes
• Target Milvus: read-only / query
• Current data scale: about 72 million vectors

Expected behavior:

• Baseline sync: initialize existing data from source to target
• Incremental sync: automatically replicate new writes from source to target
• Target should be queryable as a standby / read-only Milvus instance

Version constraint

Milvus 2.6.14 is not a hard requirement.

However, I prefer a recent 2.6.x version or another currently recommended version, because my previous performance, recall, and capacity tests were mainly done around Milvus 2.6.x.

Older versions are acceptable only if they are officially recommended for standalone primary-standby replication and have clear documentation or a verified working example.

If the only working standalone replication solution requires an older Milvus version, I would like to know:

• which version is recommended
• whether it is still maintained or recommended
• whether there are known differences in search performance, recall, index behavior, or operational behavior compared with 2.6.x
• whether it is suitable for a large collection, around 72 million vectors

Acceptable options

I am not strictly tied to a specific MQ type or CDC implementation.

Acceptable options include:

• Woodpecker / Pulsar / Kafka
• built-in CDC / standby replication
• external milvus-cdc server
• milvus-backup restore / restore secondary
• any other recommended approach

The main requirement is:

Is there any officially recommended or community-verified way to achieve baseline + incremental sync between two Milvus standalone instances?

Current candidate solution

The most promising option I heard so far is:

Milvus 2.6.14 standalone

• Woodpecker
• built-in CDC / standby replication
• milvus-backup restore secondary

Some Milvus staff informally suggested that standalone + Woodpecker + built-in CDC + restore secondary might be possible, but I could not find an official end-to-end guide for this setup.

So I would like to confirm whether this combination is supported, or whether another standalone solution is recommended instead.

Questions

1. Is primary-standby replication between two Milvus standalone instances supported in any recent Milvus version?

2. If yes, what is the recommended version and architecture?

   • Recommended Milvus version
   • Whether it is close to 2.6.x or requires an older version
   • MQ/WAL type
   • CDC method
   • baseline restore method
   • Any known impact on search performance, recall, index behavior, or compatibility compared with Milvus 2.6.x

3. Is there an official or community-verified standalone solution for:

   • baseline sync of existing data
   • incremental sync of new writes
   • read-only query on the standby instance

4. For Milvus 2.6.x, is the following combination supported?

   Milvus standalone

   • Woodpecker
   • built-in CDC / standby replication
   • milvus-backup restore secondary

5. If built-in CDC is supported for standalone:

   • Is there a CDC node / replication worker in standalone mode?
   • How should it be enabled?
   • Is update_replicate_configuration expected to work with standalone endpoints?

6. If external milvus-cdc server is the recommended standalone solution:

   • Which Milvus versions are known to work?
   • Which MQ should be used: Pulsar, Kafka, or Woodpecker?
   • Is there a working example for two standalone instances?
   • How should baseline data be restored before starting incremental CDC?

7. For initializing existing data, what is the recommended baseline method?

   • normal milvus-backup restore
   • milvus-backup restore secondary
   • another method

8. If using CDC after baseline restore, how should the following be aligned?

   • collection_id
   • pchannel / vchannel
   • checkpoint

9. If standalone primary-standby replication is not recommended or not supported, is cluster/operator mode currently the only recommended solution for baseline + incremental sync?

Previous tests

I have already tested normal backup/restore between two Milvus standalone instances, and it can restore data across instances.

However, normal restore creates different collection_id on the target, so I am concerned it cannot be used directly as the CDC baseline.

I also tested:

Milvus 2.6.14 standalone

• Pulsar
• external milvus-cdc

Observed result:

• Source and target Milvus could start with Pulsar.
• The external milvus-cdc task could run.
• Source collection schema could be replicated to target.
• But index was not replicated.
• After source insert + flush, target row count stayed 0.
• Logs included:

MqTtMsgStream can not consume the message from streaming service

So I am now looking for a verified standalone solution instead of blindly trying different combinations.

Expected answer

Please clarify which of the following is true:

1. There is an officially supported standalone solution for baseline + incremental sync.
2. There is a community-verified standalone solution, but it is not fully documented.
3. Standalone primary-standby replication is not currently supported/recommended.
4. Cluster/operator mode is required for reliable primary-standby replication with baseline + incremental sync.

If a standalone solution exists, it would be very helpful to provide:

• recommended Milvus version
• recommended MQ/WAL type
• whether to use built-in CDC or external milvus-cdc
• correct baseline restore method
• minimal working configuration or documentation link

[xiaofan-luan]

cdc don't replicate index, and it shouldn't, it only replicate dml and ddl operations.
each cluster build their own index.

1. milvus cdc project is deprecated. you should use the cdc in milvus repo
2. wal doesn't matter, milvus 2.6 latest is recommended.
3. existing data should be backup and restored to new cluster.
4. cdc take the checkpoint of the backup cluster and replicate the streaming data.

we don't have documents for cdc yet, @bigsheeper is working on it.
Our zilliz cloud support product level replication support between two mlilvus clusters, in case if you are interested.

[yhmo]

Q1: yes, there is a new built-in CDC implemented for the recent milvus versions 2.6.x, but the doc is not updated. The major function is ok, seems there are still some issues that need to be solved. A design doc of CDC is here: https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260203-primary_standby_failover_user_guide.md
Q2: the new CDC is for 2.6.x(later version is better)
Q3: I believe the new CDC will cover all
Q4: In the new CDC, each milvus server has at least a CDC node to transfer DDL/DCL/DML operations from the primary instance to the standby instance. It supports all WAL types, including woodpacker. I believe the milvus-backup is still necessary to sync the existing data to a standby instance.
Q5: the new CDC is supposed to support both cluster and standalone
Q6: new CDC is supposed to support all kinds WAL/MQ
Q7: I think milvus-backup is required, not sure. wait @bigsheeper to confirm
Q8: wait @bigsheeper to confirm
Q9: the new CDC is supposed to support both cluster and standalone

Wait @bigsheeper for more input.

[gphmath]

@xiaofan-luan @yhmo Thanks for the clarification from both of you.

I also did a small empty-start verification based on the built-in CDC direction.

My previous test failed because I only started milvus run standalone on both source and target. After clarification in the related bug issue, I added an extra source-side built-in CDC component using the same milvusdb/milvus:v2.6.14 image and command:

milvus run cdc

The CDC container used the same source-side configuration and connected to the source etcd.

After that, I re-ran update_replicate_configuration on both source and target.

Small empty-start test result:

1. Source created a new collection after the CDC component was running.
2. Target received the collection.
3. Source inserted 10 rows and flushed.
4. Target queried 10 rows.
5. Source inserted 2 additional rows.
6. Target queried 12 rows.

So, for a small empty-start test, standalone + Woodpecker + built-in CDC works when the source-side CDC component is started separately.

However, this only verifies the empty-start incremental path.

My original production scenario still requires a baseline step for existing data, because the source has about 72 million vectors.

I still need to verify:

• whether the baseline should use normal backup/restore or restore secondary
• how CDC uses the backup checkpoint
• whether collection_id / pchannel / checkpoint are aligned after restore
• whether the standby target can continue receiving incremental data after baseline restore

I will continue with a small backup/restore + CDC baseline test next.

[bigsheeper]

@gphmath Thanks for the detailed verification.

Your empty-start result is consistent with the expected behavior: standalone + Woodpecker + built-in CDC can work when the source-side CDC component is started separately.

For the production case with existing data, I would recommend setting up two fresh empty clusters, enabling/configuring CDC from the beginning, and then re-inserting the full dataset into the source cluster.

The backup/restore + CDC baseline path is technically more complex. It requires careful alignment of collection metadata, pchannel/vchannel mapping, and checkpoints. It is not a simple plug-and-play path, and misalignment can lead to missing or duplicated incremental data.

So for now, if re-inserting the data is feasible, that is the recommended path:

1. Start fresh empty source and target clusters.
2. Start the source-side built-in CDC component.
3. Apply update_replicate_configuration.
4. Re-insert the full dataset into the source cluster.
5. Verify that the target catches up.
6. Keep CDC running for subsequent incremental writes.