From 9c566dea59750e24e9dc259b784be5512341dde5 Mon Sep 17 00:00:00 2001 From: tinswzy Date: Mon, 8 Jun 2026 17:31:06 +0800 Subject: [PATCH 01/19] improve switch mq user docs Signed-off-by: tinswzy --- .../switch_milvus_cluster_mq_type-helm.md | 260 ++++++++++++++++++ .../switch_milvus_cluster_mq_type-operator.md | 170 ++++++------ ...itch_milvus_standalone_mq_type-operator.md | 165 ----------- .../switch_milvus_standalone_mq_type.md | 203 ++++++++++++++ site/en/menuStructure/en.json | 19 +- 5 files changed, 571 insertions(+), 246 deletions(-) create mode 100644 site/en/adminGuide/switch_milvus_cluster_mq_type-helm.md delete mode 100644 site/en/adminGuide/switch_milvus_standalone_mq_type-operator.md create mode 100644 site/en/adminGuide/switch_milvus_standalone_mq_type.md diff --git a/site/en/adminGuide/switch_milvus_cluster_mq_type-helm.md b/site/en/adminGuide/switch_milvus_cluster_mq_type-helm.md new file mode 100644 index 000000000..13f7e3155 --- /dev/null +++ b/site/en/adminGuide/switch_milvus_cluster_mq_type-helm.md @@ -0,0 +1,260 @@ +--- +id: switch_milvus_cluster_mq_type-helm.md +summary: Learn how to switch the message queue type for a Milvus cluster. +title: Switch MQ Type for Milvus Cluster +--- + +# Switch MQ Type for Milvus Cluster + +This topic describes how to switch the message queue (MQ) type for an existing Milvus cluster deployment. Milvus supports online MQ switching between Pulsar, Kafka, and Woodpecker without downtime. + +
+ +This feature is pending release and is subject to change. Please reach out to Milvus support if you want to try it out or have any questions. + +
+ +## Prerequisites + +- A running Milvus cluster instance installed via [Helm](install_cluster-helm.md). +- The Milvus instance has been upgraded to the latest version that supports this Switch MQ feature (v2.6.14 or later). + +## Switch from builtin/external Pulsar/Kafka to Woodpecker (MinIO) + +Follow these steps to switch the MQ type from Pulsar or Kafka to Woodpecker with MinIO storage. + +### Step 1: Verify the Milvus instance is running + +Before switching, ensure that your Milvus cluster instance is running properly. You can verify this by creating a test collection, inserting data, and running a query. + +### Step 2: Execute the MQ switch + +
+ +If you are switching to Woodpecker for the first time, you can skip this note. Otherwise, when switching repeatedly, make sure to clean up the related data before switching. Residual data may cause unexpected behavior. For example, switching from Woodpecker (MinIO/local) to another MQ and then back to Woodpecker requires cleaning up Woodpecker meta and data first. + +
+ +Run the following command to trigger the switch to Woodpecker: + +```shell +curl -X POST http://:/management/wal/alter \ + -H "Content-Type: application/json" \ + -d '{"target_wal_name": "woodpecker"}' +``` + +
+ +Replace `` and `` with the actual address of your MixCoord service. You can typically use `kubectl port-forward` to expose the MixCoord management interface: + +```shell +kubectl port-forward --address 0.0.0.0 service/my-release-milvus-mixcoord 29091:9091 +``` + +In this case, `mixcoord_addr` is the IP of the machine running the command, and `mixcoord_port` is the forwarded port `29091`. + +
+ +### Step 3: Verify the switch is complete + +The switch process completes automatically. Monitor the Milvus logs for the following key messages to confirm the switch has finished: + +```shell +kubectl logs my-release-milvus-mixcoord-68d9889549-vdj8n | grep "successfully updated mq.type configuration in etcd" +``` + +If the switch is successful, the output should look similar to the following: + +``` +[INFO] [coordinator/wal_callbacks.go:90] ["successfully updated mq.type configuration in etcd"] +[targetWALName=WoodPecker] [config=null] [broadcastID=464632819924869973] [configKey=mq.type] [mqTypeValue=woodpecker] +``` + +### Step 4: [Optional] Stop Pulsar pods and clean up Pulsar data + +#### For builtin Pulsar instances + +After confirming the switch to Woodpecker is successful, you can remove the Pulsar pods and their associated PVC data. + +Run the following command to disable Pulsar and enable Woodpecker: + +```shell +helm upgrade my-release zilliztech/milvus \ + --set image.all.tag=v2.6.13 \ + --set pulsarv3.enabled=false \ + --set woodpecker.enabled=true \ + --set streaming.enabled=true \ + --set indexNode.enabled=false +``` + +Run the following commands to delete the Pulsar PVC storage: + +```shell +# Delete Pulsar bookie journal PVCs +kubectl delete pvc my-release-pulsarv3-bookie-journal-my-release-pulsarv3-bookie-0 +kubectl delete pvc my-release-pulsarv3-bookie-journal-my-release-pulsarv3-bookie-1 +kubectl delete pvc my-release-pulsarv3-bookie-journal-my-release-pulsarv3-bookie-2 + +# Delete Pulsar bookie ledger PVCs +kubectl delete pvc my-release-pulsarv3-bookie-ledgers-my-release-pulsarv3-bookie-0 +kubectl delete pvc my-release-pulsarv3-bookie-ledgers-my-release-pulsarv3-bookie-1 +kubectl delete pvc my-release-pulsarv3-bookie-ledgers-my-release-pulsarv3-bookie-2 + +# Delete Pulsar ZooKeeper PVCs +kubectl delete pvc my-release-pulsarv3-zookeeper-data-my-release-pulsarv3-zookeeper-0 +kubectl delete pvc my-release-pulsarv3-zookeeper-data-my-release-pulsarv3-zookeeper-1 +kubectl delete pvc my-release-pulsarv3-zookeeper-data-my-release-pulsarv3-zookeeper-2 +``` + +
+ +If you plan to switch back to Pulsar in the future, you should clean up these data files first to avoid conflicts. Currently, due to Helm chart limitations, it is not possible to switch back to a builtin Pulsar instance. + +
+ +#### For external Pulsar instances + +You can clean up the Milvus-related topics in the external Pulsar instance. The Milvus topic naming format is: + +``` +-dml__ +``` + +For example: `by-dev-rootcoord-dml_10_464633776992639586v0` + +
+ +If you plan to switch back to external Pulsar in the future, you should clean up these topics first to avoid conflicts. + +
+ +## Switch from Woodpecker (MinIO) to external Pulsar/Kafka + +Follow these steps to switch the MQ type from Woodpecker back to external Pulsar or external Kafka. + +### Step 1: Verify the Milvus instance is running + +Before switching, ensure that your Milvus cluster instance is running properly. + +### Step 2: Configure the target MQ connection configuration + +Before triggering the switch, you need to ensure the target MQ service (Pulsar or Kafka) is available and its access configuration is rendered into the Milvus configuration. + +
+ +The exact steps in this section depend on whether you are using an internal (bundled) or external MQ service. + +
+ +If you are using the bundled Pulsar or Kafka deployed by Helm, update your Helm release to enable the target MQ service and disable Woodpecker. +The `streaming.enabled=true` flag is required to enable the Streaming Node, which is a prerequisite for the Switch MQ feature. For example, to switch to Pulsar: + +First, create or edit a `values.yaml` file: + +```yaml +extraConfigFiles: + user.yaml: |+ + pulsar: + address: + port: +``` + +```shell +helm upgrade -i my-release milvus/milvus \ + --set pulsarv3.enabled=true \ + --set woodpecker.enabled=false \ + --set streaming.enabled=true \ + -f values.yaml +``` + +After the upgrade, wait for all pods to be ready. Then, verify that the target MQ access configuration has been rendered into the Milvus configuration. + +### Step 3: Execute the MQ switch + +
+ +Ensure that the target Pulsar/Kafka does not contain Milvus topics from a previous configuration. If this is your first time switching to Pulsar, you can skip this note. Otherwise, clean up the residual Milvus topics with the same names in Pulsar/Kafka first. + +
+ +Run the following command to trigger the switch to Pulsar (replace `pulsar` with `kafka` if switching to Kafka): + +```shell +curl -X POST http://:/management/wal/alter \ + -H "Content-Type: application/json" \ + -d '{"target_wal_name": "pulsar"}' +``` + +
+ +Replace `` and `` with the actual address of your MixCoord service. You can typically use `kubectl port-forward` to expose the MixCoord management interface: + +```shell +kubectl port-forward --address 0.0.0.0 service/my-release-milvus-mixcoord 29091:9091 +``` + +In this case, `mixcoord_addr` is the IP of the machine running the command, and `mixcoord_port` is the forwarded port `29091`. + +
+ +### Step 4: Verify the switch is complete + +The switch process completes automatically. Monitor the Milvus logs for the following key messages to confirm the switch has finished: + +```shell +kubectl logs my-release-milvus-mixcoord-68d9889549-vdj8n | grep "successfully updated mq.type configuration in etcd" +``` + +If the switch is successful, the output should look similar to the following: + +``` +[INFO] [coordinator/wal_callbacks.go:90] ["successfully updated mq.type configuration in etcd"] +[targetWALName=pulsar] [config=null] [broadcastID=464632819924869973] [configKey=mq.type] [mqTypeValue=pulsar] +``` + +### Step 5: [Optional] Clean up Woodpecker data on MinIO/S3 and etcd + +Log in to MinIO/S3 and navigate to the corresponding bucket. Delete the Woodpecker data at the following path: + +- Format: `/wp//...` +- Example: typically `files/wp/...` + +Use an etcd tool to delete the Woodpecker metadata. The key-value prefix is as follows: + +- Prefix format: `/...` +- Example: typically `woodpecker/...` + +You can run the following command to view and then clean up the metadata: + +```shell +etcdctl get woodpecker --prefix +``` + +
+ +If you plan to switch back to Woodpecker in the future, you should clean up these data files first to avoid conflicts. + +
+ +## Supported MQ switch scenarios for Helm-installed Milvus Cluster + +| Source MQ | Target MQ | Status | Notes | +|-----------|-----------|--------|-------| +| Builtin Pulsar | Woodpecker (MinIO) | **Supported** | | +| Builtin Pulsar | External Kafka | **Supported** | | +| Builtin Pulsar | Builtin Kafka | **Not supported** | Helm does not support installing two builtin MQ systems simultaneously. | +| External Pulsar | Woodpecker (MinIO) | **Supported** | | +| External Pulsar | External Kafka | **Supported** | | +| Builtin Kafka | Woodpecker (MinIO) | **Supported** | | +| Builtin Kafka | External Pulsar | **Supported** | | +| Builtin Kafka | Builtin Pulsar | **Not supported** | Helm does not support installing two builtin MQ systems simultaneously. | +| External Kafka | Woodpecker (MinIO) | **Supported** | | +| External Kafka | External Pulsar | **Supported** | | +| Builtin/external Pulsar/Kafka | Woodpecker (local) | **Supported but not recommended** | All pods must access the same local disk (e.g., NFS mount), and additional configuration is required. | +| Woodpecker local | Woodpecker MinIO | **Not supported** | Switching between Woodpecker storage modes is not yet supported. May be supported in the future, but not recommended because Woodpecker local mode should not be used in cluster deployments. | + +
+ +Avoid switching MQ types back and forth repeatedly. If you do need to switch, make sure to clean up the related data before each switch. Residual data may cause unexpected behavior. + +
diff --git a/site/en/adminGuide/switch_milvus_cluster_mq_type-operator.md b/site/en/adminGuide/switch_milvus_cluster_mq_type-operator.md index 6b522aa19..3d2940326 100644 --- a/site/en/adminGuide/switch_milvus_cluster_mq_type-operator.md +++ b/site/en/adminGuide/switch_milvus_cluster_mq_type-operator.md @@ -16,10 +16,10 @@ This feature is pending release and is subject to change. Please reach out to Mi ## Prerequisites -- A running Milvus cluster instance installed via [Milvus Operator](install_cluster-milvusoperator.md) or [Helm](install_cluster-helm.md). -- The Milvus instance has been upgraded to the latest version that supports this Switch MQ feature. +- A running Milvus cluster instance installed via [Milvus Operator](install_cluster-milvusoperator.md). +- The Milvus instance has been upgraded to the latest version that supports this Switch MQ feature (v2.6.14 or later). -## Switch from Pulsar/Kafka to Woodpecker (MinIO) +## Switch from builtin/external Pulsar/Kafka to Woodpecker (MinIO) Follow these steps to switch the MQ type from Pulsar or Kafka to Woodpecker with MinIO storage. @@ -27,135 +27,149 @@ Follow these steps to switch the MQ type from Pulsar or Kafka to Woodpecker with Before switching, ensure that your Milvus cluster instance is running properly. You can verify this by creating a test collection, inserting data, and running a query. -### Step 2: (Optional) Verify Woodpecker configuration +### Step 2: Execute the MQ switch -The default Milvus configuration already sets Woodpecker storage type to MinIO, so no additional configuration is required in most cases. - -However, if you have previously customized the Woodpecker configuration, you must ensure that `woodpecker.storage.type` is set to `minio`. Update the Milvus configuration **without** changing the `mqType` value: - -```yaml -woodpecker: - storage: - type: minio -``` +
-- For **Helm**, refer to [Configure Milvus with Helm Charts](configure-helm.md) for instructions on updating configuration. -- For **Milvus Operator**, refer to [Configure Milvus with Milvus Operator](configure_operator.md) for instructions on updating configuration. +If you are switching to Woodpecker for the first time, you can skip this note. Otherwise, when switching repeatedly, make sure to clean up the related data before switching. Residual data may cause unexpected behavior. For example, switching from Woodpecker (MinIO/local) to another MQ and then back to Woodpecker requires cleaning up Woodpecker meta and data first. -### Step 3: Execute the MQ switch +
Run the following command to trigger the switch to Woodpecker: ```shell -curl -X POST http://:9091/management/wal/alter \ +curl -X POST http://:/management/wal/alter \ -H "Content-Type: application/json" \ -d '{"target_wal_name": "woodpecker"}' ```
-Replace `` with the actual address of your MixCoord service. +Replace `` and `` with the actual address of your MixCoord service. You can typically use `kubectl port-forward` to expose the MixCoord management interface: -
+```shell +kubectl port-forward --address 0.0.0.0 service/my-release-milvus-mixcoord 29091:9091 +``` -### Step 4: Verify the switch is complete +In this case, `mixcoord_addr` is the IP of the machine running the command, and `mixcoord_port` is the forwarded port `29091`. -The switch process completes automatically. Monitor the Milvus logs for the following key messages to confirm the switch has finished: +However, Milvus Operator currently does not expose the MixCoord service. You need to use `kubectl exec` to enter the MixCoord pod and run the `curl` command from inside the pod. -``` -WAL switch success: switch to finish, re-opening required -AlterWAL broadcast message acknowledged by all vchannels -successfully updated mq.type configuration in etcd -``` - -
+
-In the log messages above, `` is the source MQ type (e.g., `pulsar` or `kafka`), and `` is the target MQ type (`woodpecker`). +### Step 3: Verify the switch is complete -- The first message indicates that the WAL switch from the source to the target has completed. -- The second message indicates that all physical channels have been switched. -- The third message indicates that the `mq.type` configuration has been updated in etcd. +The switch process completes automatically. Monitor the Milvus logs for the following key messages to confirm the switch has finished: - +```shell +kubectl logs my-release-milvus-mixcoord-68d9889549-vdj8n | grep "successfully updated mq.type configuration in etcd" +``` -## Switch from Woodpecker (MinIO) to Pulsar or Kafka +If the switch is successful, the output should look similar to the following: -Follow these steps to switch the MQ type from Woodpecker back to Pulsar or Kafka. +``` +[INFO] [coordinator/wal_callbacks.go:90] ["successfully updated mq.type configuration in etcd"] +[targetWALName=WoodPecker] [config=null] [broadcastID=464632819924869973] [configKey=mq.type] [mqTypeValue=woodpecker] +``` -### Step 1: Verify the Milvus instance is running +### Step 4: Update the MQ type in the Operator ConfigMap -Before switching, ensure that your Milvus cluster instance is running properly. +Update the MQ type in the Operator ConfigMap to Woodpecker. First, create a file named `change_configmap.yaml`: -### Step 2: Configure the target MQ +```yaml +apiVersion: milvus.io/v1beta1 +kind: Milvus +metadata: + name: my-release + labels: + app: milvus +spec: + dependencies: + msgStreamType: woodpecker +``` -Before triggering the switch, you need to ensure the target MQ service (Pulsar or Kafka) is available and its access configuration is rendered into the Milvus configuration. +Then apply the configuration update: -
+```shell +kubectl patch -f change_configmap.yaml --patch-file change_configmap.yaml --type merge +``` -The exact steps in this section depend on whether you are using an internal (bundled) or external MQ service. +### Step 5: [Optional] Stop Pulsar pods and clean up Pulsar data -
+#### For builtin Pulsar instances -#### Option A: Internal Pulsar/Kafka (bundled with Helm) +After confirming the switch to Woodpecker is successful, you can remove the Pulsar pods and their associated PVC data. -If you are using the bundled Pulsar or Kafka deployed by Helm, update your Helm release to enable the target MQ service and disable Woodpecker. The `streaming.enabled=true` flag is required to enable the Streaming Node, which is a prerequisite for the Switch MQ feature. For example, to switch to Pulsar: +Run the following command to uninstall Pulsar: ```shell -helm upgrade -i my-release milvus/milvus \ - --set pulsarv3.enabled=true \ - --set woodpecker.enabled=false \ - --set streaming.enabled=true \ - -f values.yaml +helm uninstall my-release-pulsar ``` -After the upgrade, verify that the target MQ access configuration has been rendered into the Milvus configuration. For example, for Pulsar: +Run the following commands to delete the Pulsar PVC storage: -```yaml -pulsar: - address: - port: 6650 +```shell +# Delete Pulsar bookie journal PVCs +kubectl delete pvc my-release-pulsarv3-bookie-journal-my-release-pulsarv3-bookie-0 +kubectl delete pvc my-release-pulsarv3-bookie-journal-my-release-pulsarv3-bookie-1 +kubectl delete pvc my-release-pulsarv3-bookie-journal-my-release-pulsarv3-bookie-2 + +# Delete Pulsar bookie ledger PVCs +kubectl delete pvc my-release-pulsarv3-bookie-ledgers-my-release-pulsarv3-bookie-0 +kubectl delete pvc my-release-pulsarv3-bookie-ledgers-my-release-pulsarv3-bookie-1 +kubectl delete pvc my-release-pulsarv3-bookie-ledgers-my-release-pulsarv3-bookie-2 + +# Delete Pulsar ZooKeeper PVCs +kubectl delete pvc my-release-pulsarv3-zookeeper-data-my-release-pulsarv3-zookeeper-0 +kubectl delete pvc my-release-pulsarv3-zookeeper-data-my-release-pulsarv3-zookeeper-1 +kubectl delete pvc my-release-pulsarv3-zookeeper-data-my-release-pulsarv3-zookeeper-2 ``` -#### Option B: Internal Pulsar/Kafka (managed by Milvus Operator) - -If you are using Milvus Operator, update the Milvus custom resource to include the target MQ access configuration. Refer to [Configure Milvus with Milvus Operator](configure_operator.md) for details on updating Milvus configuration. +
-#### Option C: External Pulsar/Kafka +If you plan to switch back to Pulsar in the future, you should clean up these data files first to avoid conflicts. Currently, due to Helm chart limitations, it is not possible to switch back to a builtin Pulsar instance. -If you are using an external Pulsar or Kafka service, you do not need to change the `mqType`. Simply add the external MQ access configuration to your `values.yaml` and restart the Milvus instance to render the configuration. +
-### Step 3: Execute the MQ switch +#### For external Pulsar instances -Run the following command to trigger the switch to Pulsar (replace `pulsar` with `kafka` if switching to Kafka): +You can clean up the Milvus-related topics in the external Pulsar instance. The Milvus topic naming format is: -```shell -curl -X POST http://:9091/management/wal/alter \ - -H "Content-Type: application/json" \ - -d '{"target_wal_name": "pulsar"}' ``` +-dml__ +``` + +For example: `by-dev-rootcoord-dml_10_464633776992639586v0`
-Replace `` with the actual address of your MixCoord service. +If you plan to switch back to external Pulsar in the future, you should clean up these topics first to avoid conflicts.
-### Step 4: Verify the switch is complete +## Switch from Woodpecker (MinIO) to external Pulsar/Kafka -The switch process completes automatically. Monitor the Milvus logs for the following key messages to confirm the switch has finished: +This is currently not supported. When Milvus Operator sets the MQ type to Woodpecker, it does not allow pre-configuring the Pulsar/Kafka connection address in the configuration file, which is required for the switch. -``` -WAL switch success: switch to finish, re-opening required -AlterWAL broadcast message acknowledged by all vchannels -successfully updated mq.type configuration in etcd -``` +## Supported MQ switch scenarios for Operator-installed Milvus Cluster -
+| Source MQ | Target MQ | Status | Notes | +|-----------|-----------|--------|--------------------------------------------------------------------------------------------------------------------------| +| Builtin Pulsar | Woodpecker (MinIO) | **Supported** | | +| Builtin Pulsar | External Kafka | **Not supported** | Milvus Operator does not support pre-configuring the Kafka access address. | +| Builtin Pulsar | Builtin Kafka | **Not supported** | Helm does not support installing two builtin MQ systems simultaneously. | +| External Pulsar | Woodpecker (MinIO) | **Supported** | | +| External Pulsar | External Kafka | **Not supported** | Milvus Operator does not support pre-configuring the Kafka access address. | +| Builtin Kafka | Woodpecker (MinIO) | **Supported** | | +| Builtin Kafka | External Pulsar | **Not supported** | Milvus Operator does not support pre-configuring the Pulsar access address. | +| Builtin Kafka | Builtin Pulsar | **Not supported** | Helm does not support installing two builtin MQ systems simultaneously. | +| External Kafka | Woodpecker (MinIO) | **Supported** | | +| External Kafka | External Pulsar | **Not supported** | Milvus Operator does not support pre-configuring the Pulsar access address. | +| Builtin/external Pulsar/Kafka | Woodpecker (local) | **Supported but not recommended** | All pods must access the same local disk (e.g., NFS mount), and additional configuration is required. | +| Woodpecker local | Woodpecker MinIO | **Not supported** | Switching between Woodpecker storage modes is not yet supported. May be supported in the future, but not recommended because Woodpecker local mode should not be used in cluster deployments. | -In the log messages above, `` is the source MQ type (`woodpecker`), and `` is the target MQ type (e.g., `pulsar` or `kafka`). +
-- The first message indicates that the WAL switch from the source to the target has completed. -- The second message indicates that all physical channels have been switched. -- The third message indicates that the `mq.type` configuration has been updated in etcd. +Avoid switching MQ types back and forth repeatedly. If you do need to switch, make sure to clean up the related data before each switch. Residual data may cause unexpected behavior.
diff --git a/site/en/adminGuide/switch_milvus_standalone_mq_type-operator.md b/site/en/adminGuide/switch_milvus_standalone_mq_type-operator.md deleted file mode 100644 index 8b6dcbeef..000000000 --- a/site/en/adminGuide/switch_milvus_standalone_mq_type-operator.md +++ /dev/null @@ -1,165 +0,0 @@ ---- -id: switch_milvus_standalone_mq_type-operator.md -summary: Learn how to switch the message queue type for Milvus standalone. -title: Switch MQ Type for Milvus Standalone ---- - -# Switch MQ Type for Milvus Standalone - -This topic describes how to switch the message queue (MQ) type for an existing Milvus standalone deployment. Milvus supports online MQ switching without downtime. - -
- -This feature is pending release and is subject to change. Please reach out to Milvus support if you want to try it out or have any questions. - -
- -## Prerequisites - -- A running Milvus Standalone instance installed via [Docker](install_standalone-docker.md) or [Docker Compose](install_standalone-docker-compose.md). -- The Milvus instance has been upgraded to the latest version that supports this Switch MQ feature. - -## General workflow - -The general workflow for switching the MQ type is as follows: - -1. Ensure the Milvus instance is running properly. -2. Confirm the source MQ type and the target MQ type. -3. Configure the target MQ's access settings into the Milvus configuration without changing the `mqType` value. -4. Trigger the switch by calling the WAL alter API. -5. Monitor the logs to verify the switch has completed successfully. - -
- -Before switching, ensure that the target MQ does not contain topics with the same names as those used by the current Milvus instance. This is especially important if the target MQ service has been previously used by another Milvus instance, as conflicting topic names can lead to unexpected behavior. - -
- -## Switch from RocksMQ to Woodpecker (Local Storage) - -This procedure applies to **Milvus Standalone Docker** deployments that use RocksMQ by default. - -### Step 1: Verify the Milvus instance is running - -Ensure your Milvus Standalone Docker instance is running properly. You can verify this by creating a test collection, inserting data, and running a query. - -### Step 2: Configure Woodpecker with local storage - -Update the Milvus configuration to add Woodpecker settings **without** changing the `mqType` value. Create or update the `user.yaml` file with the following content: - -```yaml -woodpecker: - storage: - type: local -``` - -Then restart the Milvus instance to apply the configuration: - -```shell -bash standalone_embed.sh restart -``` - -### Step 3: Execute the MQ switch - -Run the following command to trigger the switch to Woodpecker: - -```shell -curl -X POST http://:9091/management/wal/alter \ - -H "Content-Type: application/json" \ - -d '{"target_wal_name": "woodpecker"}' -``` - -
- -Replace `` with the actual address of your MixCoord service (by default, `localhost` for standalone deployments). - -
- -### Step 4: Verify the switch is complete - -The switch process completes automatically. Monitor the Milvus logs for the following key messages to confirm the switch has finished: - -``` -WAL switch success: switch to finish, re-opening required -AlterWAL broadcast message acknowledged by all vchannels -successfully updated mq.type configuration in etcd -``` - -
- -In the log messages above, `` is the source MQ type (`rocksmq`), and `` is the target MQ type (`woodpecker`). - -- The first message indicates that the WAL switch from the source to the target has completed. -- The second message indicates that all physical channels have been switched. -- The third message indicates that the `mq.type` configuration has been updated in etcd. - -
- -## Switch from RocksMQ to Woodpecker (MinIO Storage) - -This procedure applies to **Milvus Standalone Docker Compose** deployments. - -
- -Starting from Milvus v2.6, the default `docker-compose.yaml` already declares `mqType` as Woodpecker. Unless you have modified the default configuration or upgraded from v2.5, this procedure may not be necessary. - -
- -### Step 1: Verify the Milvus instance is running - -Ensure your Milvus Standalone Docker Compose instance is running properly. - -### Step 2: (Optional) Verify Woodpecker configuration - -The default Milvus configuration already sets Woodpecker storage type to MinIO, so no additional configuration is required in most cases. - -However, if you have previously customized the Woodpecker configuration, you must ensure that `woodpecker.storage.type` is set to `minio`. Create or update the `user.yaml` file with the following content: - -```yaml -woodpecker: - storage: - type: minio -``` - -Then restart the Milvus instance to apply the configuration: - -```shell -docker compose down -docker compose up -d -``` - -### Step 3: Execute the MQ switch - -Run the following command to trigger the switch to Woodpecker: - -```shell -curl -X POST http://:9091/management/wal/alter \ - -H "Content-Type: application/json" \ - -d '{"target_wal_name": "woodpecker"}' -``` - -
- -Replace `` with the actual address of your MixCoord service (by default, `localhost` for standalone deployments). - -
- -### Step 4: Verify the switch is complete - -The switch process completes automatically. Monitor the Milvus logs for the following key messages to confirm the switch has finished: - -``` -WAL switch success: switch to finish, re-opening required -AlterWAL broadcast message acknowledged by all vchannels -successfully updated mq.type configuration in etcd -``` - -
- -In the log messages above, `` is the source MQ type (`rocksmq`), and `` is the target MQ type (`woodpecker`). - -- The first message indicates that the WAL switch from the source to the target has completed. -- The second message indicates that all physical channels have been switched. -- The third message indicates that the `mq.type` configuration has been updated in etcd. - -
diff --git a/site/en/adminGuide/switch_milvus_standalone_mq_type.md b/site/en/adminGuide/switch_milvus_standalone_mq_type.md new file mode 100644 index 000000000..2e0086c42 --- /dev/null +++ b/site/en/adminGuide/switch_milvus_standalone_mq_type.md @@ -0,0 +1,203 @@ +--- +id: switch_milvus_standalone_mq_type.md +summary: Learn how to switch the message queue type for Milvus standalone. +title: Switch MQ Type for Milvus Standalone +--- + +# Switch MQ Type for Milvus Standalone + +This topic describes how to switch the message queue (MQ) type for an existing Milvus standalone deployment. Milvus supports online MQ switching without downtime. + +
+ +This feature is pending release and is subject to change. Please reach out to Milvus support if you want to try it out or have any questions. + +
+ +## Prerequisites + +- A running Milvus Standalone instance installed via [Docker Compose](install_standalone-docker-compose.md). +- The Milvus instance has been upgraded to the latest version that supports this Switch MQ feature (v2.6.14 or later). + +## General workflow + +The general workflow for switching the MQ type is as follows: + +1. Ensure the Milvus instance is running properly. +2. Confirm the source MQ type and the target MQ type. +3. Configure the target MQ's access settings into the Milvus configuration without changing the `mqType` value. +4. Trigger the switch by calling the WAL alter API. +5. Monitor the logs to verify the switch has completed successfully. + +
+ +Before switching, ensure that the target MQ does not contain topics with the same names as those used by the current Milvus instance. +This is especially important if the target MQ service has been previously used by another Milvus instance, as conflicting topic names can lead to unexpected behavior. + +
+ +## Docker Compose Standalone: Switch from RocksMQ to Woodpecker (Local/MinIO) + +This procedure applies to **Milvus Standalone Docker Compose** deployments that use RocksMQ by default. + +### Step 1: Verify the Milvus instance is running + +Ensure your Milvus Standalone docker compose instance is running properly. You can verify this by creating a test collection, inserting data, and running a query. + +### Step 2: Configure Woodpecker with local storage + +Update the Milvus configuration to add Woodpecker settings **without** changing the `mqType` value. Run `docker exec -it milvus-standalone bash` to enter the container, then edit the `/milvus/configs/user.yaml` file with the following key configuration: + +```yaml +woodpecker: + storage: + type: minio # minio or local +``` + +Then restart the Milvus instance to apply the configuration: + +```shell +docker-compose restart +``` + +### Step 3: Execute the MQ switch + +
+ +If you are switching to Woodpecker for the first time, you can skip this note. Otherwise, when switching repeatedly, make sure to clean up the related data before switching. Residual data may cause unexpected behavior. For example, switching from Woodpecker (MinIO/local) to RocksMQ and then back to Woodpecker requires cleaning up Woodpecker meta and data first. + +
+ +Run the following command to trigger the switch to Woodpecker: + +```shell +curl -X POST http://:/management/wal/alter \ + -H "Content-Type: application/json" \ + -d '{"target_wal_name": "woodpecker"}' +``` + +
+ +This command sends a switch MQ type request to the internal MixCoord component in the standalone instance. Typically, the MixCoord port is `9091`. + +
+ +### Step 4: Verify the switch is complete + +The switch process completes automatically. Monitor the Milvus logs for the following key messages to confirm the switch has finished. Run the following command to check the switch progress: + +```shell +docker logs milvus-standalone | grep "successfully updated mq.type configuration in etcd" +``` + +If the switch is successful, the output should look similar to the following: + +``` +[INFO] [coordinator/wal_callbacks.go:90] ["successfully updated mq.type configuration in etcd"] [targetWALName=WoodPecker] +[config=null] [broadcastID=464677797180736841] [configKey=mq.type] [mqTypeValue=woodpecker] +``` + +### Step 5: [Optional] Clean up RocksMQ data + +The RocksMQ data is located in the `volumes/milvus/rdb_data` and `volumes/milvus/rdb_data_meta_kv` directories defined in `docker-compose.yaml`. + +
+ +If you plan to switch back to RocksMQ in the future, you should clean up these data files first to avoid conflicts. + +
+ +## Docker Compose Standalone: Switch from Woodpecker (Local/MinIO) to RocksMQ + +This procedure applies to **Milvus Standalone Docker Compose** deployments that are currently using Woodpecker and want to switch to RocksMQ. + +### Step 1: Verify the Milvus instance is running + +Ensure your Milvus Standalone docker compose instance is running properly. You can verify this by creating a test collection, inserting data, and running a query. + +### Step 2: Execute the MQ switch + +
+ +Ensure that the instance does not have residual RocksMQ data from a previous run. If this is your first time switching to RocksMQ, you can skip this note. Otherwise, clean up the related RocksMQ meta data and data first. + +
+ +Run the following command to trigger the switch to RocksMQ: + +```shell +curl -X POST http://:/management/wal/alter \ + -H "Content-Type: application/json" \ + -d '{"target_wal_name": "rocksmq"}' +``` + +
+ +This command sends a switch MQ type request to the internal MixCoord component in the standalone instance. Typically, the MixCoord port is `9091`. + +
+ +### Step 3: Verify the switch is complete + +The switch process completes automatically. Monitor the Milvus logs for the following key messages to confirm the switch has finished. Run the following command to check the switch progress: + +```shell +docker logs milvus-standalone | grep "successfully updated mq.type configuration in etcd" +``` + +If the switch is successful, the output should look similar to the following: + +``` +[INFO] [coordinator/wal_callbacks.go:90] ["successfully updated mq.type configuration in etcd"] [targetWALName=rocksmq] +[config=null] [broadcastID=464677797180736841] [configKey=mq.type] [mqTypeValue=rocksmq] +``` + +### Step 4: [Optional] Clean up Woodpecker data + +#### Clean up Woodpecker metadata + +Use an etcd tool to delete the Woodpecker metadata. The key-value prefix is as follows: + +- Prefix format: `/...` +- Example: typically `woodpecker/...` + +You can run the following command to view and then clean up the metadata: + +```shell +etcdctl get woodpecker --prefix +``` + +#### Clean up Woodpecker storage data + +If Woodpecker was using **MinIO mode**, delete the log data on object storage. Log in to MinIO and navigate to the corresponding bucket. The data path is as follows: + +- Format: `/wp//...` +- Example: typically `files/wp/...` + +If Woodpecker was using **local mode**, the data is stored on the local disk at `volumes/milvus/data/wp/...`. + +
+ +If you plan to switch back to Woodpecker in the future, you should clean up these data files first to avoid conflicts. + +
+ +## Supported MQ switch scenarios for Milvus Standalone + +**Docker deployment (single container):** MQ switching is not supported because etcd source is not enabled and the `mq.type` config cannot be updated. + +**Docker Compose deployment:** + +| Source MQ | Target MQ | Status | Notes | +|-----------|-----------|--------|-------| +| RocksMQ | Woodpecker (MinIO/local) | **Supported** | | +| Woodpecker (MinIO/local) | RocksMQ | **Supported** | | +| Woodpecker MinIO | Woodpecker local | **Not supported** | Switching between Woodpecker storage modes requires additional metadata handling, which is not yet supported. May be supported in the future. | +| Woodpecker local | Woodpecker MinIO | **Not supported** | Same reason as above. | +| RocksMQ / Woodpecker (MinIO/local) | External Pulsar / Kafka | **Supported but not recommended** | Standalone instances should be kept as simple as possible for ease of operation and maintenance. | + +
+ +Avoid switching MQ types back and forth repeatedly. If you do need to switch, make sure to clean up the related data before each switch. Residual data may cause unexpected behavior. + +
diff --git a/site/en/menuStructure/en.json b/site/en/menuStructure/en.json index 3c72e787c..282ff00bb 100644 --- a/site/en/menuStructure/en.json +++ b/site/en/menuStructure/en.json @@ -1966,13 +1966,26 @@ "children": [ { "label": "Milvus Cluster switch MQ Type", - "id": "switch_milvus_cluster_mq_type-operator.md", + "id": "switch_milvus_cluster_mq_type", "order": 0, - "children": [] + "children": [ + { + "label": "Helm Chart", + "id": "switch_milvus_cluster_mq_type-helm.md", + "order": 0, + "children": [] + }, + { + "label": "Milvus Operator", + "id": "switch_milvus_cluster_mq_type-operator.md", + "order": 1, + "children": [] + } + ] }, { "label": "Milvus Standalone switch MQ type", - "id": "switch_milvus_standalone_mq_type-operator.md", + "id": "switch_milvus_standalone_mq_type.md", "order": 1, "children": [] } From 36a62ca0016d64b2943325e046ce01472d6e80e3 Mon Sep 17 00:00:00 2001 From: tinswzy Date: Mon, 8 Jun 2026 20:42:21 +0800 Subject: [PATCH 02/19] docs: state Woodpecker + local FS as the Docker standalone default Clarify that the single-container Docker deployment runs Woodpecker (local-filesystem WAL) by default, and add an Optional dependencies section linking the message-queue, object-storage, and metadata options. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../run-milvus-docker/install_standalone-docker.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/site/en/getstarted/run-milvus-docker/install_standalone-docker.md b/site/en/getstarted/run-milvus-docker/install_standalone-docker.md index 81944c33c..66669866c 100644 --- a/site/en/getstarted/run-milvus-docker/install_standalone-docker.md +++ b/site/en/getstarted/run-milvus-docker/install_standalone-docker.md @@ -33,7 +33,7 @@ $ bash standalone_embed.sh start **What's new in v{{var.milvus_release_version}}:** - **Streaming Node**: Enhanced data processing capabilities -- **Woodpecker MQ**: Improved message queue with reduced maintenance overhead, see [Use Woodpecker](use-woodpecker.md) for detail +- **Woodpecker MQ (default)**: This Docker deployment runs Woodpecker as the message queue with the **local filesystem** as its WAL backend, so no external message-queue service is required. See [Use Woodpecker](use-woodpecker.md). - **Optimized Architecture**: Consolidated components for better performance Always download the latest script to ensure you get the most recent configurations and architecture improvements. @@ -106,6 +106,14 @@ $ bash standalone_embed.sh stop $ bash standalone_embed.sh delete ``` +## Optional dependencies + +By default this deployment runs **Woodpecker** (local-filesystem WAL) as the message queue and an **embedded etcd** for metadata — nothing else to install. To use a different message queue or connect external object storage / metadata, see: + +- Message queue: [Woodpecker](use-woodpecker.md) (default) · [Pulsar](deploy_pulsar.md) · [Kafka](deploy_pulsar.md#Configure-Kafka-with-Helm) · [RocksMQ](deploy_pulsar.md#Configure-RocksMQ-with-Helm) +- Object storage: [MinIO](deploy_s3.md) (default) · [AWS S3](deploy_s3.md) · [Azure Blob](abs.md) · [GCP Cloud Storage](gcs.md) · [Aliyun OSS](deploy_s3.md) · [Tencent COS](deploy_s3.md) · [Huawei OBS](deploy_s3.md) · [S3-compatible](deploy_s3.md) +- Metadata: [etcd](deploy_etcd.md) + ## What's next Having installed Milvus in Docker, you can: From 337461363039363a1c4658f9f37ba202cefdb11d Mon Sep 17 00:00:00 2001 From: tinswzy Date: Mon, 8 Jun 2026 20:49:00 +0800 Subject: [PATCH 03/19] =?UTF-8?q?docs:=20document=20Compose=20default=20de?= =?UTF-8?q?ployment,=20version=20MQ=20defaults,=20and=202.5=E2=86=922.6=20?= =?UTF-8?q?upgrade=20caveat?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Describe the default three-container deployment with Woodpecker (embedded, local-filesystem WAL), note the per-version default message queue (2.5.x RocksMQ vs 2.6.x+ Woodpecker), add a 2.5.x->2.6.x upgrade section that pins RocksMQ via mq.type, and add an Optional dependencies section. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../install_standalone-docker-compose.md | 31 ++++++++++++++++--- 1 file changed, 27 insertions(+), 4 deletions(-) diff --git a/site/en/getstarted/run-milvus-docker/install_standalone-docker-compose.md b/site/en/getstarted/run-milvus-docker/install_standalone-docker-compose.md index 700f97c3f..39b113eb5 100644 --- a/site/en/getstarted/run-milvus-docker/install_standalone-docker-compose.md +++ b/site/en/getstarted/run-milvus-docker/install_standalone-docker-compose.md @@ -33,10 +33,11 @@ Creating milvus-standalone ... done
-**What's new in v{{var.milvus_release_version}}:** -- **Enhanced Architecture**: Features the new Streaming Node and optimized components -- **Updated Dependencies**: Includes the latest MinIO and etcd versions -- **Improved Configuration**: Optimized settings for better performance +**Default deployment (v{{var.milvus_release_version}}):** `docker compose up -d` starts three containers — `milvus-etcd` (metadata), `milvus-minio` (object storage), and `milvus-standalone`. The message queue is **Woodpecker (embedded, local-filesystem WAL backend)**, so no separate message-queue container is required. + +**Message-queue default by version:** +- **2.5.x** — default message queue is **RocksMQ**. +- **2.6.x and later** — default message queue is **Woodpecker (embedded)**. Always download the latest Docker Compose configuration to ensure compatibility with v{{var.milvus_release_version}} features. @@ -106,6 +107,28 @@ $ sudo docker compose down $ sudo rm -rf volumes ``` +## Upgrading from Milvus 2.5.x to 2.6.x + +{{fragments/mq_upgrade_limitation.md}} + +Because 2.6.x changes the default message queue to Woodpecker, an instance running **RocksMQ** on 2.5.x must **explicitly pin RocksMQ before upgrading** — otherwise the upgrade would attempt to change the message queue, which is not supported. After downloading the 2.6.x Docker Compose file, set the message-queue type back to `rocksmq` in your `user.yaml` override, then upgrade: + +```yaml +# user.yaml — keep RocksMQ across the 2.5.x → 2.6.x upgrade +mq: + type: rocksmq +``` + +To switch the message queue *after* upgrading, see [Switch MQ Type](switch_milvus_standalone_mq_type.md). + +## Optional dependencies + +This deployment runs **Woodpecker** (local-filesystem WAL) for messaging, **etcd** for metadata, and **MinIO** for object storage. To use a different message queue or connect external object storage / metadata, see: + +- Message queue: [Woodpecker](use-woodpecker.md) (default) · [Pulsar](deploy_pulsar.md) · [Kafka](deploy_pulsar.md#Configure-Kafka-with-Helm) · [RocksMQ](deploy_pulsar.md#Configure-RocksMQ-with-Helm) +- Object storage: [MinIO](deploy_s3.md) (default) · [AWS S3](deploy_s3.md) · [Azure Blob](abs.md) · [GCP Cloud Storage](gcs.md) · [Aliyun OSS](deploy_s3.md) · [Tencent COS](deploy_s3.md) · [Huawei OBS](deploy_s3.md) · [S3-compatible](deploy_s3.md) +- Metadata: [etcd](deploy_etcd.md) + ## What's next Having installed Milvus in Docker, you can: From 4607d469aa5c126bb78da1b77d9c63552242fb23 Mon Sep 17 00:00:00 2001 From: tinswzy Date: Tue, 9 Jun 2026 17:04:07 +0800 Subject: [PATCH 04/19] docs: remove stale Pulsar references from the Milvus Operator install page The default Operator sample now deploys Woodpecker, so drop the leftover Pulsar mentions in the overview, dependency-creation note, and uninstall note; remove the PulsarReady condition from the status example; and add an Optional dependencies section pointing to the Operator dependency configuration pages. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../install_cluster-milvusoperator.md | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/site/en/getstarted/run-milvus-k8s/install_cluster-milvusoperator.md b/site/en/getstarted/run-milvus-k8s/install_cluster-milvusoperator.md index 152bdac32..228b5b7d3 100644 --- a/site/en/getstarted/run-milvus-k8s/install_cluster-milvusoperator.md +++ b/site/en/getstarted/run-milvus-k8s/install_cluster-milvusoperator.md @@ -12,7 +12,7 @@ This page illustrates how to start a Milvus instance in Kubernetes using [Milvus ## Overview -Milvus Operator is a solution that helps you deploy and manage a full Milvus service stack to target Kubernetes (K8s) clusters. The stack includes all Milvus components and relevant dependencies like etcd, Pulsar, and MinIO. +Milvus Operator is a solution that helps you deploy and manage a full Milvus service stack to target Kubernetes (K8s) clusters. The stack includes all Milvus components and relevant dependencies like etcd and MinIO, with Woodpecker as the built-in message queue. ## Prerequisites @@ -170,11 +170,6 @@ status: reason: StorageReady status: "True" type: StorageReady - - lastTransitionTime: "xxxx-xx-xxTxx:xx:xxZ" - message: Pulsar is ready - reason: PulsarReady - status: "True" - type: PulsarReady - lastTransitionTime: "xxxx-xx-xxTxx:xx:xxZ" message: Etcd endpoints is healthy reason: EtcdReady @@ -189,7 +184,7 @@ status: status: Healthy ``` -Milvus Operator creates Milvus dependencies, such as etcd, Pulsar, and MinIO, and then Milvus components, such as proxy, coordinators, and nodes. +Milvus Operator creates Milvus dependencies, such as etcd and MinIO, and then Milvus components, such as proxy, coordinators, and nodes. Once your Milvus cluster is ready, the status of all pods in the Milvus cluster should be similar to the following. @@ -293,7 +288,7 @@ $ kubectl delete milvus my-release
-- When you delete the Milvus cluster using the default configuration, dependencies like etcd, Pulsar, and MinIO are not deleted. Therefore, next time when you install the same Milvus cluster instance, these dependencies will be used again. +- When you delete the Milvus cluster using the default configuration, dependencies like etcd and MinIO are not deleted. Therefore, next time when you install the same Milvus cluster instance, these dependencies will be used again. - To delete the dependencies and persistent volume claims (PVCs) along with the Milvus cluster, see [configuration file](https://github.com/zilliztech/milvus-operator/blob/main/config/samples/milvus_deletion.yaml).
@@ -317,6 +312,14 @@ $ helm -n milvus-operator uninstall milvus-operator $ kubectl delete -f https://raw.githubusercontent.com/zilliztech/milvus-operator/v{{var.milvus_operator_version}}/deploy/manifests/deployment.yaml ``` +## Optional dependencies + +This deployment uses **Woodpecker** as the message queue, **etcd** for metadata, and **MinIO** for object storage, all provisioned by Milvus Operator. To use a different backend with the Operator, see: + +- Message queue: [Woodpecker](use-woodpecker.md) (default) · [Pulsar](message_storage_operator.md#Configure-Pulsar) · [Kafka](message_storage_operator.md#Configure-Kafka) · [RocksMQ](message_storage_operator.md#Configure-RocksMQ) +- Object storage: [Configure object storage with Milvus Operator](object_storage_operator.md) +- Metadata: [Configure etcd with Milvus Operator](meta_storage_operator.md) + ## What's next Having installed Milvus in Docker, you can: From 6b4254cd312725e4be7825e8b19a634a96592262 Mon Sep 17 00:00:00 2001 From: tinswzy Date: Tue, 9 Jun 2026 17:45:06 +0800 Subject: [PATCH 05/19] docs: add dedicated Pulsar page and stand up the Data Infrastructure & Integration chapter Create mq_pulsar.md capturing how to install Milvus with Pulsar via Helm (including the Kubernetes v1.25+ PodDisruptionBudget workaround), and add a new top-level Data Infrastructure & Integration chapter with a Message Queue (mqType) section hosting the Pulsar page in its final location. Co-Authored-By: Claude Opus 4.8 (1M context) --- site/en/adminGuide/mq_pulsar.md | 48 +++++++++++++++++++++++++++++++++ site/en/menuStructure/en.json | 26 ++++++++++++++++-- 2 files changed, 72 insertions(+), 2 deletions(-) create mode 100644 site/en/adminGuide/mq_pulsar.md diff --git a/site/en/adminGuide/mq_pulsar.md b/site/en/adminGuide/mq_pulsar.md new file mode 100644 index 000000000..e122389f9 --- /dev/null +++ b/site/en/adminGuide/mq_pulsar.md @@ -0,0 +1,48 @@ +--- +id: mq_pulsar.md +title: Pulsar +--- + +# Use Pulsar as the Milvus Message Queue + +Apache Pulsar is one of the message-queue (WAL) backends Milvus supports. Starting from Milvus 2.6.x, **Woodpecker** is the default and recommended message queue; Pulsar remains fully supported for users who prefer it. + +## Supported versions + +- Milvus Distributed supports Pulsar as the message queue. +- Since Milvus 2.5, the Milvus Helm chart and Milvus Operator deploy Pulsar v3 by default, while Pulsar v2 remains compatible. See [Upgrade to Pulsar v3](upgrade-pulsar-v3.md) and [Continue Using Pulsar v2](use-pulsar-v2.md). + +## Install Milvus with Pulsar (Helm) + +To deploy a Milvus cluster with Pulsar instead of Woodpecker, install the Helm chart without disabling Pulsar: + +```bash +helm install my-release zilliztech/milvus \ + --set image.all.tag=v{{var.milvus_release_version}} \ + --set streaming.enabled=true \ + --set indexNode.enabled=false +``` + +### Kubernetes v1.25+ PodDisruptionBudget workaround + +On Kubernetes v1.25 and later, the bundled Pulsar sub-chart may run into PodDisruptionBudget API issues. If you encounter them, disable the Pulsar PodDisruptionBudget policies: + +```bash +helm install my-release zilliztech/milvus \ + --set pulsar.bookkeeper.pdb.usePolicy=false \ + --set pulsar.broker.pdb.usePolicy=false \ + --set pulsar.proxy.pdb.usePolicy=false \ + --set pulsar.zookeeper.pdb.usePolicy=false +``` + +## Configure Pulsar + +For detailed Pulsar parameters across deployment methods, see: + +- [Configure Pulsar with Docker Compose or Helm](deploy_pulsar.md) +- [Configure Pulsar with Milvus Operator](message_storage_operator.md#Configure-Pulsar) + +## What's next + +- [Woodpecker (default message queue)](use-woodpecker.md) +- [Switch MQ Type for Milvus Cluster](switch_milvus_cluster_mq_type-helm.md) diff --git a/site/en/menuStructure/en.json b/site/en/menuStructure/en.json index 282ff00bb..f7417704d 100644 --- a/site/en/menuStructure/en.json +++ b/site/en/menuStructure/en.json @@ -1471,11 +1471,33 @@ } ] }, + { + "label": "Data Infrastructure & Integration", + "id": "data_infra_integration", + "isMenu": true, + "order": 9, + "children": [ + { + "label": "Message Queue (mqType)", + "id": "mqtype", + "isMenu": true, + "order": 4, + "children": [ + { + "label": "Pulsar", + "id": "mq_pulsar.md", + "order": 2, + "children": [] + } + ] + } + ] + }, { "label": "Administration Guide", "id": "admin_guide", "isMenu": true, - "order": 9, + "order": 10, "children": [ { "label": "Deployment", @@ -3062,7 +3084,7 @@ "label": "FAQs", "id": "faq", "isMenu": true, - "order": 9, + "order": 11, "children": [ { "label": "Performance FAQs", From 79f952df08c9bc2fc83177c10cb17f72fdd049ec Mon Sep 17 00:00:00 2001 From: tinswzy Date: Wed, 10 Jun 2026 19:36:24 +0800 Subject: [PATCH 06/19] docs: switch Helm cluster install to Woodpecker service mode and clean up stale Pulsar content Make the cluster install use Woodpecker service mode (streaming.woodpecker.embedded=false), update the pod example to show the dedicated my-release-milvus-woodpecker StatefulSet (verified on a real v3.0-beta + milvusdb/woodpecker:latest deployment), fix the streaming-node -> streamingnode workload name, and remove leftover Pulsar content (pods, PodDisruptionBudget workaround, alternative recipe). Add an Optional dependencies section and align the offline standalone template with Woodpecker. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../run-milvus-k8s/install_cluster-helm.md | 54 +++++++++---------- 1 file changed, 25 insertions(+), 29 deletions(-) diff --git a/site/en/getstarted/run-milvus-k8s/install_cluster-helm.md b/site/en/getstarted/run-milvus-k8s/install_cluster-helm.md index b80e3620b..ec9b58310 100644 --- a/site/en/getstarted/run-milvus-k8s/install_cluster-helm.md +++ b/site/en/getstarted/run-milvus-k8s/install_cluster-helm.md @@ -94,11 +94,13 @@ helm install my-release zilliztech/milvus \ --set pulsarv3.enabled=false \ --set woodpecker.enabled=true \ --set streaming.enabled=true \ + --set streaming.woodpecker.embedded=false \ --set indexNode.enabled=false ``` **What this command does:** - Uses **Woodpecker** as the message queue (recommended for reduced maintenance) +- Runs **Woodpecker as a dedicated service** (a separate StatefulSet), not embedded in the streaming node - Enables the new **Streaming Node** component for improved performance - Disables the legacy **Index Node** (functionality is now handled by Data Node) - Disables Pulsar to use Woodpecker instead @@ -115,16 +117,7 @@ For complete architecture details, refer to the [Architecture Overview](architec
-**Alternative Message Queue Options:** - -If you prefer to use **Pulsar** (traditional choice) instead of Woodpecker: - -```bash -helm install my-release zilliztech/milvus \ - --set image.all.tag=v{{var.milvus_release_version}} \ - --set streaming.enabled=true \ - --set indexNode.enabled=false -``` +**Alternative message queues:** To deploy with Pulsar, Kafka, or RocksMQ instead of Woodpecker, see [Optional dependencies](#optional-dependencies). **Next steps:** The command above deploys Milvus with recommended configurations. For production use: @@ -136,14 +129,6 @@ The command above deploys Milvus with recommended configurations. For production **Important notes:** - **Release naming**: Use only letters, numbers, and dashes (no dots allowed) -- **Kubernetes v1.25+**: If you encounter PodDisruptionBudget issues, use this workaround: - ```bash - helm install my-release zilliztech/milvus \ - --set pulsar.bookkeeper.pdb.usePolicy=false \ - --set pulsar.broker.pdb.usePolicy=false \ - --set pulsar.proxy.pdb.usePolicy=false \ - --set pulsar.zookeeper.pdb.usePolicy=false - ``` For more information, see [Milvus Helm Chart](https://artifacthub.io/packages/helm/milvus/milvus) and [Helm documentation](https://helm.sh/docs/). @@ -168,23 +153,26 @@ my-release-milvus-datanode-68cb87dcbd-4khpm 1/1 Running 0 3m23s my-release-milvus-mixcoord-7fb9488465-dmbbj 1/1 Running 0 3m23s my-release-milvus-proxy-6bd7f5587-ds2xv 1/1 Running 0 3m24s my-release-milvus-querynode-5cd8fff495-k6gtg 1/1 Running 0 3m24s -my-release-milvus-streaming-node-xxxxxxxxx 1/1 Running 0 3m24s +my-release-milvus-streamingnode-7b8cfc769c-mqmsm 1/1 Running 0 3m24s +my-release-milvus-woodpecker-0 1/1 Running 0 3m24s +my-release-milvus-woodpecker-1 1/1 Running 0 3m24s +my-release-milvus-woodpecker-2 1/1 Running 0 3m24s +my-release-milvus-woodpecker-3 1/1 Running 0 3m24s my-release-minio-0 1/1 Running 0 3m23s my-release-minio-1 1/1 Running 0 3m23s my-release-minio-2 1/1 Running 0 3m23s my-release-minio-3 1/1 Running 0 3m23s -my-release-pulsar-autorecovery-86f5dbdf77-lchpc 1/1 Running 0 3m24s -my-release-pulsar-bookkeeper-0 1/1 Running 0 3m23s -my-release-pulsar-bookkeeper-1 1/1 Running 0 98s -my-release-pulsar-broker-556ff89d4c-2m29m 1/1 Running 0 3m23s -my-release-pulsar-proxy-6fbd75db75-nhg4v 1/1 Running 0 3m23s -my-release-pulsar-zookeeper-0 1/1 Running 0 3m23s -my-release-pulsar-zookeeper-metadata-98zbr 0/1 Completed 0 3m24s ``` **Key components to verify:** -- **Milvus components**: `mixcoord`, `datanode`, `querynode`, `proxy`, `streaming-node` -- **Dependencies**: `etcd` (metadata), `minio` (object storage), `pulsar` (message queue) +- **Milvus components**: `mixcoord`, `datanode`, `querynode`, `proxy`, `streamingnode` +- **Dependencies**: `etcd` (metadata), `minio` (object storage), `woodpecker` (message queue) + +
+ +With `streaming.woodpecker.embedded=false`, Woodpecker runs as a **dedicated StatefulSet** (`my-release-milvus-woodpecker`, 4 replicas by default) fronted by a headless service, using MinIO as its storage backend — so the cluster has a separate `woodpecker` pod set, distinct from the streaming node. + +
You can also access the **Milvus WebUI** at `http://127.0.0.1:9091/webui/` once port forwarding is set up (see next step). For details, refer to [Milvus WebUI](milvus-webui.md). @@ -284,7 +272,7 @@ The above command renders chart templates for a Milvus cluster and saves the out
-- To install a Milvus instance in the standalone mode where all Milvus components are contained within a single pod, you should run `helm template my-release --set cluster.enabled=false --set etcd.replicaCount=1 --set minio.mode=standalone --set pulsarv3.enabled=false zilliztech/milvus > milvus_manifest.yaml` instead to render chart templates for a Milvus instance in a standalone mode. +- To install a Milvus instance in the standalone mode where all Milvus components are contained within a single pod, you should run `helm template my-release --set cluster.enabled=false --set etcd.replicaCount=1 --set minio.mode=standalone --set pulsarv3.enabled=false --set standalone.messageQueue=woodpecker --set woodpecker.enabled=true --set streaming.enabled=true zilliztech/milvus > milvus_manifest.yaml` instead to render chart templates for a Milvus instance in a standalone mode. - To change Milvus configurations, download the [`value.yaml`](https://raw.githubusercontent.com/milvus-io/milvus-helm/master/charts/milvus/values.yaml) template, place your desired settings in it, and use `helm template -f values.yaml my-release zilliztech/milvus > milvus_manifest.yaml` to render the manifest accordingly.
@@ -342,6 +330,14 @@ Run the following command to uninstall Milvus. $ helm uninstall my-release ``` +## Optional dependencies + +This deployment runs **Woodpecker** as the message queue, **etcd** for metadata, and **MinIO** for object storage. To use a different message queue or connect external object storage / metadata, see: + +- Message queue: [Woodpecker](use-woodpecker.md) (default) · [Pulsar](mq_pulsar.md) · [Kafka](deploy_pulsar.md#Configure-Kafka-with-Helm) · [RocksMQ](deploy_pulsar.md#Configure-RocksMQ-with-Helm) +- Object storage: [MinIO](deploy_s3.md) (default) · [AWS S3](deploy_s3.md) · [Azure Blob](abs.md) · [GCP Cloud Storage](gcs.md) · [Aliyun OSS](deploy_s3.md) · [Tencent COS](deploy_s3.md) · [Huawei OBS](deploy_s3.md) · [S3-compatible](deploy_s3.md) +- Metadata: [etcd](deploy_etcd.md) + ## What's next Having installed Milvus in Docker, you can: From 5f7bd00906a922aa5d6e5bd3b59c15abc1284167 Mon Sep 17 00:00:00 2001 From: tinswzy Date: Wed, 10 Jun 2026 19:53:40 +0800 Subject: [PATCH 07/19] docs: clarify Woodpecker service mode is distributed/cluster only (standalone uses embedded) Co-Authored-By: Claude Opus 4.8 (1M context) --- site/en/getstarted/run-milvus-k8s/install_cluster-helm.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/en/getstarted/run-milvus-k8s/install_cluster-helm.md b/site/en/getstarted/run-milvus-k8s/install_cluster-helm.md index ec9b58310..cee9ffe5d 100644 --- a/site/en/getstarted/run-milvus-k8s/install_cluster-helm.md +++ b/site/en/getstarted/run-milvus-k8s/install_cluster-helm.md @@ -80,7 +80,7 @@ helm install my-release zilliztech/milvus \ --set streaming.enabled=true ``` -**Note**: Standalone mode uses Woodpecker as the default message queue and enables the Streaming Node component. For details, refer to the [Architecture Overview](architecture_overview.md) and [Use Woodpecker](use-woodpecker.md). +**Note**: Standalone mode uses Woodpecker as the default message queue and enables the Streaming Node component. Standalone deployments run Woodpecker **embedded** in the Milvus pod; the dedicated Woodpecker **service** (separate pods) is used for **distributed/cluster** deployments only. For details, refer to the [Architecture Overview](architecture_overview.md) and [Use Woodpecker](use-woodpecker.md).
From 8c6b2638619406910ab0b37bfd7c3c300da22aa4 Mon Sep 17 00:00:00 2001 From: tinswzy Date: Wed, 10 Jun 2026 20:09:58 +0800 Subject: [PATCH 08/19] docs: mark Woodpecker as the default bundled MQ and Pulsar as optional in prerequisite tables Co-Authored-By: Claude Opus 4.8 (1M context) --- site/en/getstarted/run-milvus-docker/prerequisite-docker.md | 3 ++- site/en/getstarted/run-milvus-k8s/prerequisite-helm.md | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/site/en/getstarted/run-milvus-docker/prerequisite-docker.md b/site/en/getstarted/run-milvus-docker/prerequisite-docker.md index 8fab33b69..10d32be13 100644 --- a/site/en/getstarted/run-milvus-docker/prerequisite-docker.md +++ b/site/en/getstarted/run-milvus-docker/prerequisite-docker.md @@ -32,7 +32,8 @@ The following dependencies will be obtained and configured automatically when Mi | -------- | ----------------------------- | ---- | | etcd | 3.5.0 | See [additional disk requirements](#Additional-disk-requirements). | | MinIO | RELEASE.2024-12-18T13-15-44Z | | -| Pulsar | 2.8.2 | | +| Woodpecker | Bundled with Milvus | Default message queue (embedded); no separate service to install. | +| Pulsar | 2.8.2 | Optional — only if you switch the message queue to Pulsar; not installed by default. | ### Additional disk requirements diff --git a/site/en/getstarted/run-milvus-k8s/prerequisite-helm.md b/site/en/getstarted/run-milvus-k8s/prerequisite-helm.md index 10a2e6165..b47a79c23 100644 --- a/site/en/getstarted/run-milvus-k8s/prerequisite-helm.md +++ b/site/en/getstarted/run-milvus-k8s/prerequisite-helm.md @@ -35,7 +35,8 @@ minikube is required when running Kubernetes cluster locally. minikube requires | -------- | ----------------------------- | ---- | | etcd | 3.5.0 | See [additional disk requirements](#Additional-disk-requirements). | | MinIO | RELEASE.2024-12-18T13-15-44Z | | -| Pulsar | 2.8.2 | | +| Woodpecker | Bundled with Milvus | Default message queue; runs as a dedicated service for distributed deployments. | +| Pulsar | 2.8.2 | Optional — only if you switch the message queue to Pulsar; not installed by default. | ### Additional disk requirements From 312dbe2d463c58109ded8ddcda56cca88f8f0275 Mon Sep 17 00:00:00 2001 From: tinswzy Date: Wed, 10 Jun 2026 20:33:23 +0800 Subject: [PATCH 09/19] docs: note Woodpecker as the default message queue in the deployment overview Co-Authored-By: Claude Opus 4.8 (1M context) --- site/en/getstarted/install-overview.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/site/en/getstarted/install-overview.md b/site/en/getstarted/install-overview.md index 3ab7e6bd9..3d575821d 100644 --- a/site/en/getstarted/install-overview.md +++ b/site/en/getstarted/install-overview.md @@ -16,11 +16,11 @@ To integrate Milvus Lite into your applications, run `pip install pymilvus` to i ## Milvus Standalone -Milvus Standalone is a single-machine server deployment. All components of Milvus Standalone are packed into a single [Docker image](https://milvus.io/docs/install_standalone-docker.md), making deployment convenient. If you have a production workload but prefer not to use Kubernetes, running Milvus Standalone on a single machine with sufficient memory is a good option. +Milvus Standalone is a single-machine server deployment. All components of Milvus Standalone are packed into a single [Docker image](https://milvus.io/docs/install_standalone-docker.md), making deployment convenient. If you have a production workload but prefer not to use Kubernetes, running Milvus Standalone on a single machine with sufficient memory is a good option. By default, Milvus Standalone runs **Woodpecker** (embedded) as its message queue, so there's no separate messaging service to operate. ## Milvus Distributed -Milvus Distributed can be deployed on [Kubernetes](https://milvus.io/docs/install_cluster-milvusoperator.md) clusters. This deployment features a cloud-native architecture, where ingestion load and search queries are separately handled by isolated nodes, allowing redundancy for critical components. It offers the highest scalability and availability, as well as the flexibility in customizing the allocated resources in each component. Milvus Distributed is the top choice for enterprise users running large-scale vector search systems in production. +Milvus Distributed can be deployed on [Kubernetes](https://milvus.io/docs/install_cluster-milvusoperator.md) clusters. This deployment features a cloud-native architecture, where ingestion load and search queries are separately handled by isolated nodes, allowing redundancy for critical components. It offers the highest scalability and availability, as well as the flexibility in customizing the allocated resources in each component. Milvus Distributed is the top choice for enterprise users running large-scale vector search systems in production. By default, Milvus Distributed runs **Woodpecker** as its message queue, deployed as a dedicated service alongside Milvus. ## Choose the Right Deployment for Your Use Case @@ -60,4 +60,5 @@ The choice of Milvus deployment mode depends on your project's stage and scale. | CRUD operations | ✔️ | ✔️ | ✔️ | | Advanced data management | N/A | Access Control
Partition
Partition Key | Access Control
Partition
Partition Key
Physical Resource Grouping | | Consistency Levels | Strong | Strong
Bounded Staleness
Session
Eventual | Strong
Bounded Staleness
Session
Eventual | +| Message queue | N/A | Woodpecker (embedded) | Woodpecker (service) | From 0ab6d2d88330782b04980aa8a6a948b52b5535da Mon Sep 17 00:00:00 2001 From: tinswzy Date: Wed, 10 Jun 2026 20:38:49 +0800 Subject: [PATCH 10/19] docs: fix Windows page title and note Woodpecker default on Windows/RPM-DEB standalone pages Co-Authored-By: Claude Opus 4.8 (1M context) --- .../run-milvus-docker/install_standalone-binary.md | 6 ++++++ .../run-milvus-docker/install_standalone-windows.md | 4 +++- 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/site/en/getstarted/run-milvus-docker/install_standalone-binary.md b/site/en/getstarted/run-milvus-docker/install_standalone-binary.md index 28849cf43..1d5dfea9a 100644 --- a/site/en/getstarted/run-milvus-docker/install_standalone-binary.md +++ b/site/en/getstarted/run-milvus-docker/install_standalone-binary.md @@ -74,6 +74,12 @@ If Milvus is running successfully, you should see the following output: You can find the Milvus binary at `/usr/bin/milvus`, the systemd service file at `/lib/systemd/system/milvus.service`, and the dependencies at `/usr/lib/milvus/`. +
+ +By default, Milvus Standalone runs **Woodpecker** (local filesystem) as its message queue with embedded etcd, so no external messaging or metadata service is required. See [Use Woodpecker](use-woodpecker.md). + +
+ ## (Optional) Update Milvus configurations You can modify the Milvus configurations in the `/etc/milvus/configs/milvus.yaml` file. For example, to change the `proxy.healthCheckTimeout` to `1000` ms, you can search for the target parameter and modify accordingly. For applicable configuration items, refer to [System Configuration](system_configuration.md). diff --git a/site/en/getstarted/run-milvus-docker/install_standalone-windows.md b/site/en/getstarted/run-milvus-docker/install_standalone-windows.md index d4405c6d0..682d0b488 100644 --- a/site/en/getstarted/run-milvus-docker/install_standalone-windows.md +++ b/site/en/getstarted/run-milvus-docker/install_standalone-windows.md @@ -3,11 +3,13 @@ id: install_standalone-windows.md label: Docker related_key: Docker summary: Learn how to install Milvus standalone with Docker Desktop for Windows. -title: Run Milvus in Docker (Linux) +title: Run Milvus in Docker (Windows) --- # Run Milvus in Docker (Windows) +> By default, this deployment runs **Woodpecker** (local filesystem) as its message queue, so no external message-queue service is required. See [Use Woodpecker](use-woodpecker.md). + This page demonstrates how to run Milvus on Windows using Docker Desktop for Windows.​ ## Prerequisites​ From 78e04d62d0e227dd12747dd9f6cfff056b5b3694 Mon Sep 17 00:00:00 2001 From: tinswzy Date: Wed, 10 Jun 2026 20:55:06 +0800 Subject: [PATCH 11/19] docs: migrate Use Woodpecker into the new chapter as woodpecker.md Move adminGuide/use-woodpecker.md to woodpecker.md under the new Data Infrastructure & Integration > mqType section, reframe Woodpecker as the default message queue, document the three storage.type modes (minio/local/service) with a support matrix, and add a verified 'Woodpecker service mode' deployment section (separate StatefulSet, 4 replicas, headless service, MinIO backend; distributed/cluster only). Repoint all use-woodpecker.md links to woodpecker.md and update the nav. Co-Authored-By: Claude Opus 4.8 (1M context) --- site/en/adminGuide/deploy_pulsar.md | 2 +- .../en/adminGuide/message_storage_operator.md | 2 +- site/en/adminGuide/mq_pulsar.md | 2 +- .../{use-woodpecker.md => woodpecker.md} | 62 ++++++++++++++----- .../install_standalone-binary.md | 2 +- .../install_standalone-docker-compose.md | 2 +- .../install_standalone-docker.md | 4 +- .../install_standalone-windows.md | 2 +- .../run-milvus-k8s/install_cluster-helm.md | 4 +- .../install_cluster-milvusoperator.md | 4 +- site/en/menuStructure/en.json | 12 ++-- 11 files changed, 64 insertions(+), 34 deletions(-) rename site/en/adminGuide/{use-woodpecker.md => woodpecker.md} (80%) diff --git a/site/en/adminGuide/deploy_pulsar.md b/site/en/adminGuide/deploy_pulsar.md index 4d7dd952e..c6d14d61c 100644 --- a/site/en/adminGuide/deploy_pulsar.md +++ b/site/en/adminGuide/deploy_pulsar.md @@ -75,7 +75,7 @@ helm install milvus/milvus -f values.yaml For Milvus clusters on K8s, you can configure Woodpecker in the same command that starts Milvus. Alternatively, you can configure Woodpecker using the values.yml file on the /charts/milvus path in the [milvus-helm](https://github.com/milvus-io/milvus-helm) repository before you start Milvus. -For details on how to configure Milvus using Helm, refer to [Configure Milvus with Helm Charts](configure-helm.md). For details on Woodpecker-related configuration items, refer to [woodpecker-related configurations](use-woodpecker.md). +For details on how to configure Milvus using Helm, refer to [Configure Milvus with Helm Charts](configure-helm.md). For details on Woodpecker-related configuration items, refer to [woodpecker-related configurations](woodpecker.md). | ### Using the YAML file diff --git a/site/en/adminGuide/message_storage_operator.md b/site/en/adminGuide/message_storage_operator.md index a2b6a43fd..701ad5ebd 100644 --- a/site/en/adminGuide/message_storage_operator.md +++ b/site/en/adminGuide/message_storage_operator.md @@ -85,7 +85,7 @@ spec: ## Configure Woodpecker -Woodpecker is a cloud-native Write-Ahead Log (WAL) designed for object storage. It offers high throughput, low operational overhead, and seamless scalability. For more details, see [Use Woodpecker](use-woodpecker.md). +Woodpecker is a cloud-native Write-Ahead Log (WAL) designed for object storage. It offers high throughput, low operational overhead, and seamless scalability. For more details, see [Use Woodpecker](woodpecker.md). ## Configure Pulsar diff --git a/site/en/adminGuide/mq_pulsar.md b/site/en/adminGuide/mq_pulsar.md index e122389f9..f7a4c4940 100644 --- a/site/en/adminGuide/mq_pulsar.md +++ b/site/en/adminGuide/mq_pulsar.md @@ -44,5 +44,5 @@ For detailed Pulsar parameters across deployment methods, see: ## What's next -- [Woodpecker (default message queue)](use-woodpecker.md) +- [Woodpecker (default message queue)](woodpecker.md) - [Switch MQ Type for Milvus Cluster](switch_milvus_cluster_mq_type-helm.md) diff --git a/site/en/adminGuide/use-woodpecker.md b/site/en/adminGuide/woodpecker.md similarity index 80% rename from site/en/adminGuide/use-woodpecker.md rename to site/en/adminGuide/woodpecker.md index 49bb74bda..5eacad8f3 100644 --- a/site/en/adminGuide/use-woodpecker.md +++ b/site/en/adminGuide/woodpecker.md @@ -1,20 +1,19 @@ --- -id: use-woodpecker.md -title: Use Woodpecker +id: woodpecker.md +title: Woodpecker related_key: Woodpecker -summary: Learn how to enable woodpecker as the WAL in milvus. -beta: Milvus 2.6.x +summary: Learn how Woodpecker works as the default message queue (WAL) in Milvus, and how to run it in embedded or service mode. --- -# Use Woodpecker +# Woodpecker -This guide explains how to enable and use Woodpecker as the Write-Ahead Log (WAL) in Milvus 2.6.x. Woodpecker is a cloud‑native WAL designed for object storage, offering high throughput, low operational overhead, and seamless scalability. For architecture and benchmark details, see [Woodpecker](woodpecker_architecture.md). +Woodpecker is the **default message queue (write-ahead log, WAL)** in Milvus starting from 2.6.x. It is a cloud‑native WAL designed for object storage, offering high throughput, low operational overhead, and seamless scalability. For architecture and benchmark details, see [Woodpecker](woodpecker_architecture.md). ## Overview -- Starting from Milvus 2.6, Woodpecker is an optional WAL that provides ordered writes and recovery as the logging service. -- As a message queue choice, it behaves similarly to Pulsar/Kafka and can be enabled via configuration. -- Two storage backends are supported: local file system (`local`) and object storage (`minio`/S3-compatible). +- Starting from Milvus 2.6, Woodpecker is the **default** WAL/message queue, providing ordered writes and recovery as the logging service. No external message-queue service (such as Pulsar or Kafka) is required. +- Woodpecker can run **embedded** in the Milvus/streaming node (default), or as a **dedicated service** with its own pods (distributed/cluster only). +- It supports three `storage.type` modes: object storage (`minio`, the default), local file system (`local`), and the dedicated `service`. See [Deployment modes](#Deployment-modes). ## Quick start @@ -84,17 +83,19 @@ Key notes: ## Deployment modes -Milvus supports both Standalone and Cluster modes. Woodpecker storage backend support matrix: +Woodpecker supports three `storage.type` modes: -| | `storage.type=local` | `storage.type=minio` | -| ----------------- | ------------------------ | ------------------------ | -| Milvus Standalone | Supported | Supported | -| Milvus Cluster | Limited (needs shared FS) | Supported | +| `storage.type` | How Woodpecker runs | WAL backend | Milvus Standalone | Milvus Distributed (cluster) | +| -------------- | ------------------- | ----------- | ----------------- | ---------------------------- | +| `minio` (default) | Embedded in the Milvus/streaming node | Object storage (MinIO/S3‑compatible) | Supported | Supported | +| `local` | Embedded in the Milvus/streaming node | Local file system | Supported | Limited (all nodes need a shared FS, e.g. NFS) | +| `service` | **Dedicated Woodpecker service** (its own pods) | Object storage (MinIO/S3‑compatible) | **Not supported** | Supported | Notes: - With `minio`, Woodpecker shares the same object storage with Milvus (MinIO/S3/GCS/OSS, etc.). - With `local`, a single‑node local disk is only suitable for Standalone. If all pods can access a shared file system (e.g., NFS), Cluster mode can also use `local`. +- **`service` mode runs Woodpecker as a separate, independently scalable service and is only available for distributed/cluster deployments.** Standalone deployments use the embedded modes (`minio` or `local`). ## Object storage compatibility for `storage.type=minio` @@ -156,7 +157,7 @@ Run the following command to uninstall the Milvus cluster. kubectl delete milvus my-release ``` -If you need to adjust Woodpecker parameters, follow the settings described in [message storage config](deploy_pulsar.md). +If you need to adjust Woodpecker parameters, follow the settings described in [Configuration](#Configuration). ### Enable Woodpecker for a Milvus Cluster on Kubernetes (Helm Chart, storage=minio) @@ -187,7 +188,7 @@ helm install my-release zilliztech/milvus \ --set streaming.enabled=true ``` -After deployment, follow the docs to port‑forward and connect. To adjust Woodpecker parameters, follow the settings described in [message storage config](deploy_pulsar.md). +After deployment, follow the docs to port‑forward and connect. To adjust Woodpecker parameters, follow the settings described in [Configuration](#Configuration). ### Enable Woodpecker for Milvus Standalone in Docker (storage=local) @@ -237,6 +238,35 @@ EOF' docker restart milvus-standalone ``` +### Enable Woodpecker service mode for a Milvus Cluster (Helm) + +For distributed/cluster deployments, you can run Woodpecker as a **dedicated service** (separate pods) instead of embedded in the streaming node by setting `streaming.woodpecker.embedded=false`: + +```bash +helm install my-release zilliztech/milvus \ + --set image.all.tag=v{{var.milvus_release_version}} \ + --set pulsarv3.enabled=false \ + --set woodpecker.enabled=true \ + --set streaming.enabled=true \ + --set streaming.woodpecker.embedded=false \ + --set indexNode.enabled=false +``` + +This deploys Woodpecker as a dedicated StatefulSet (`my-release-milvus-woodpecker`, 4 replicas by default) fronted by a headless service, gossip-clustered on ports `18080` (service), `17946` (gossip), and `9091` (metrics), with MinIO as its storage backend. The cluster then includes a separate `woodpecker` pod set: + +``` +my-release-milvus-woodpecker-0 +my-release-milvus-woodpecker-1 +my-release-milvus-woodpecker-2 +my-release-milvus-woodpecker-3 +``` + +
+ +Woodpecker `service` mode is for **distributed/cluster** deployments only — standalone deployments run Woodpecker embedded (`minio` or `local`). Milvus Operator does not yet support Woodpecker service mode. + +
+ ## Throughput tuning tips Based on the benchmarks and backend limits in [Woodpecker](woodpecker_architecture.md), optimize end‑to‑end write throughput from the following aspects: diff --git a/site/en/getstarted/run-milvus-docker/install_standalone-binary.md b/site/en/getstarted/run-milvus-docker/install_standalone-binary.md index 1d5dfea9a..d9119a289 100644 --- a/site/en/getstarted/run-milvus-docker/install_standalone-binary.md +++ b/site/en/getstarted/run-milvus-docker/install_standalone-binary.md @@ -76,7 +76,7 @@ You can find the Milvus binary at `/usr/bin/milvus`, the systemd service file at
-By default, Milvus Standalone runs **Woodpecker** (local filesystem) as its message queue with embedded etcd, so no external messaging or metadata service is required. See [Use Woodpecker](use-woodpecker.md). +By default, Milvus Standalone runs **Woodpecker** (local filesystem) as its message queue with embedded etcd, so no external messaging or metadata service is required. See [Use Woodpecker](woodpecker.md).
diff --git a/site/en/getstarted/run-milvus-docker/install_standalone-docker-compose.md b/site/en/getstarted/run-milvus-docker/install_standalone-docker-compose.md index 39b113eb5..f879d3e45 100644 --- a/site/en/getstarted/run-milvus-docker/install_standalone-docker-compose.md +++ b/site/en/getstarted/run-milvus-docker/install_standalone-docker-compose.md @@ -125,7 +125,7 @@ To switch the message queue *after* upgrading, see [Switch MQ Type](switch_milvu This deployment runs **Woodpecker** (local-filesystem WAL) for messaging, **etcd** for metadata, and **MinIO** for object storage. To use a different message queue or connect external object storage / metadata, see: -- Message queue: [Woodpecker](use-woodpecker.md) (default) · [Pulsar](deploy_pulsar.md) · [Kafka](deploy_pulsar.md#Configure-Kafka-with-Helm) · [RocksMQ](deploy_pulsar.md#Configure-RocksMQ-with-Helm) +- Message queue: [Woodpecker](woodpecker.md) (default) · [Pulsar](deploy_pulsar.md) · [Kafka](deploy_pulsar.md#Configure-Kafka-with-Helm) · [RocksMQ](deploy_pulsar.md#Configure-RocksMQ-with-Helm) - Object storage: [MinIO](deploy_s3.md) (default) · [AWS S3](deploy_s3.md) · [Azure Blob](abs.md) · [GCP Cloud Storage](gcs.md) · [Aliyun OSS](deploy_s3.md) · [Tencent COS](deploy_s3.md) · [Huawei OBS](deploy_s3.md) · [S3-compatible](deploy_s3.md) - Metadata: [etcd](deploy_etcd.md) diff --git a/site/en/getstarted/run-milvus-docker/install_standalone-docker.md b/site/en/getstarted/run-milvus-docker/install_standalone-docker.md index 66669866c..fa3034be2 100644 --- a/site/en/getstarted/run-milvus-docker/install_standalone-docker.md +++ b/site/en/getstarted/run-milvus-docker/install_standalone-docker.md @@ -33,7 +33,7 @@ $ bash standalone_embed.sh start **What's new in v{{var.milvus_release_version}}:** - **Streaming Node**: Enhanced data processing capabilities -- **Woodpecker MQ (default)**: This Docker deployment runs Woodpecker as the message queue with the **local filesystem** as its WAL backend, so no external message-queue service is required. See [Use Woodpecker](use-woodpecker.md). +- **Woodpecker MQ (default)**: This Docker deployment runs Woodpecker as the message queue with the **local filesystem** as its WAL backend, so no external message-queue service is required. See [Use Woodpecker](woodpecker.md). - **Optimized Architecture**: Consolidated components for better performance Always download the latest script to ensure you get the most recent configurations and architecture improvements. @@ -110,7 +110,7 @@ $ bash standalone_embed.sh delete By default this deployment runs **Woodpecker** (local-filesystem WAL) as the message queue and an **embedded etcd** for metadata — nothing else to install. To use a different message queue or connect external object storage / metadata, see: -- Message queue: [Woodpecker](use-woodpecker.md) (default) · [Pulsar](deploy_pulsar.md) · [Kafka](deploy_pulsar.md#Configure-Kafka-with-Helm) · [RocksMQ](deploy_pulsar.md#Configure-RocksMQ-with-Helm) +- Message queue: [Woodpecker](woodpecker.md) (default) · [Pulsar](deploy_pulsar.md) · [Kafka](deploy_pulsar.md#Configure-Kafka-with-Helm) · [RocksMQ](deploy_pulsar.md#Configure-RocksMQ-with-Helm) - Object storage: [MinIO](deploy_s3.md) (default) · [AWS S3](deploy_s3.md) · [Azure Blob](abs.md) · [GCP Cloud Storage](gcs.md) · [Aliyun OSS](deploy_s3.md) · [Tencent COS](deploy_s3.md) · [Huawei OBS](deploy_s3.md) · [S3-compatible](deploy_s3.md) - Metadata: [etcd](deploy_etcd.md) diff --git a/site/en/getstarted/run-milvus-docker/install_standalone-windows.md b/site/en/getstarted/run-milvus-docker/install_standalone-windows.md index 682d0b488..069ab80eb 100644 --- a/site/en/getstarted/run-milvus-docker/install_standalone-windows.md +++ b/site/en/getstarted/run-milvus-docker/install_standalone-windows.md @@ -8,7 +8,7 @@ title: Run Milvus in Docker (Windows) # Run Milvus in Docker (Windows) -> By default, this deployment runs **Woodpecker** (local filesystem) as its message queue, so no external message-queue service is required. See [Use Woodpecker](use-woodpecker.md). +> By default, this deployment runs **Woodpecker** (local filesystem) as its message queue, so no external message-queue service is required. See [Use Woodpecker](woodpecker.md). This page demonstrates how to run Milvus on Windows using Docker Desktop for Windows.​ diff --git a/site/en/getstarted/run-milvus-k8s/install_cluster-helm.md b/site/en/getstarted/run-milvus-k8s/install_cluster-helm.md index cee9ffe5d..3e6d5afe2 100644 --- a/site/en/getstarted/run-milvus-k8s/install_cluster-helm.md +++ b/site/en/getstarted/run-milvus-k8s/install_cluster-helm.md @@ -80,7 +80,7 @@ helm install my-release zilliztech/milvus \ --set streaming.enabled=true ``` -**Note**: Standalone mode uses Woodpecker as the default message queue and enables the Streaming Node component. Standalone deployments run Woodpecker **embedded** in the Milvus pod; the dedicated Woodpecker **service** (separate pods) is used for **distributed/cluster** deployments only. For details, refer to the [Architecture Overview](architecture_overview.md) and [Use Woodpecker](use-woodpecker.md). +**Note**: Standalone mode uses Woodpecker as the default message queue and enables the Streaming Node component. Standalone deployments run Woodpecker **embedded** in the Milvus pod; the dedicated Woodpecker **service** (separate pods) is used for **distributed/cluster** deployments only. For details, refer to the [Architecture Overview](architecture_overview.md) and [Use Woodpecker](woodpecker.md). @@ -334,7 +334,7 @@ $ helm uninstall my-release This deployment runs **Woodpecker** as the message queue, **etcd** for metadata, and **MinIO** for object storage. To use a different message queue or connect external object storage / metadata, see: -- Message queue: [Woodpecker](use-woodpecker.md) (default) · [Pulsar](mq_pulsar.md) · [Kafka](deploy_pulsar.md#Configure-Kafka-with-Helm) · [RocksMQ](deploy_pulsar.md#Configure-RocksMQ-with-Helm) +- Message queue: [Woodpecker](woodpecker.md) (default) · [Pulsar](mq_pulsar.md) · [Kafka](deploy_pulsar.md#Configure-Kafka-with-Helm) · [RocksMQ](deploy_pulsar.md#Configure-RocksMQ-with-Helm) - Object storage: [MinIO](deploy_s3.md) (default) · [AWS S3](deploy_s3.md) · [Azure Blob](abs.md) · [GCP Cloud Storage](gcs.md) · [Aliyun OSS](deploy_s3.md) · [Tencent COS](deploy_s3.md) · [Huawei OBS](deploy_s3.md) · [S3-compatible](deploy_s3.md) - Metadata: [etcd](deploy_etcd.md) diff --git a/site/en/getstarted/run-milvus-k8s/install_cluster-milvusoperator.md b/site/en/getstarted/run-milvus-k8s/install_cluster-milvusoperator.md index 228b5b7d3..06219a2f2 100644 --- a/site/en/getstarted/run-milvus-k8s/install_cluster-milvusoperator.md +++ b/site/en/getstarted/run-milvus-k8s/install_cluster-milvusoperator.md @@ -136,7 +136,7 @@ $ kubectl apply -f https://raw.githubusercontent.com/zilliztech/milvus-operator/ The command above deploys a Milvus cluster with **Woodpecker** as the message queue (recommended for v{{var.milvus_release_version}}) and all new architectural components including the Streaming Node. **Architecture highlights in this deployment:** -- **Message Queue**: [Uses Woodpecker](use-woodpecker.md) (reduces infrastructure maintenance) +- **Message Queue**: [Uses Woodpecker](woodpecker.md) (reduces infrastructure maintenance) - **Streaming Node**: Enabled for enhanced data processing - **Mix Coordinator**: Consolidated coordinator components for improved efficiency @@ -316,7 +316,7 @@ $ kubectl delete -f https://raw.githubusercontent.com/zilliztech/milvus-operator This deployment uses **Woodpecker** as the message queue, **etcd** for metadata, and **MinIO** for object storage, all provisioned by Milvus Operator. To use a different backend with the Operator, see: -- Message queue: [Woodpecker](use-woodpecker.md) (default) · [Pulsar](message_storage_operator.md#Configure-Pulsar) · [Kafka](message_storage_operator.md#Configure-Kafka) · [RocksMQ](message_storage_operator.md#Configure-RocksMQ) +- Message queue: [Woodpecker](woodpecker.md) (default) · [Pulsar](message_storage_operator.md#Configure-Pulsar) · [Kafka](message_storage_operator.md#Configure-Kafka) · [RocksMQ](message_storage_operator.md#Configure-RocksMQ) - Object storage: [Configure object storage with Milvus Operator](object_storage_operator.md) - Metadata: [Configure etcd with Milvus Operator](meta_storage_operator.md) diff --git a/site/en/menuStructure/en.json b/site/en/menuStructure/en.json index f7417704d..dbe4f8428 100644 --- a/site/en/menuStructure/en.json +++ b/site/en/menuStructure/en.json @@ -1483,6 +1483,12 @@ "isMenu": true, "order": 4, "children": [ + { + "label": "Woodpecker", + "id": "woodpecker.md", + "order": 1, + "children": [] + }, { "label": "Pulsar", "id": "mq_pulsar.md", @@ -1975,12 +1981,6 @@ "order": "12", "children": [] }, - { - "label": "Woodpecker", - "id": "use-woodpecker.md", - "order": 13, - "children": [] - }, { "label": "Switch MQ Type", "id": "switch_milvus_mq_type", From 318af606b73cbd10301cf4683092409b98b8c396 Mon Sep 17 00:00:00 2001 From: tinswzy Date: Thu, 11 Jun 2026 09:44:06 +0800 Subject: [PATCH 12/19] docs: build out the mqType section with a message queue overview and complete Pulsar, Kafka, and RocksMQ pages Add a Message Queue overview (support matrix across Woodpecker/Pulsar/ Kafka/RocksMQ x standalone/cluster), and expand the per-MQ pages into self-contained guides with version compatibility and install/configure/ uninstall steps: Pulsar and Kafka for Milvus cluster via Helm and Milvus Operator, RocksMQ for standalone via Docker. Each page links Woodpecker as the default and notes the 2.5.x->2.6.x upgrade behavior. Co-Authored-By: Claude Opus 4.8 (1M context) --- site/en/adminGuide/mq_kafka.md | 122 ++++++++++++++++++++++++ site/en/adminGuide/mq_pulsar.md | 129 +++++++++++++++++++++++--- site/en/adminGuide/mq_rocksmq.md | 71 ++++++++++++++ site/en/adminGuide/mqtype-overview.md | 32 +++++++ site/en/menuStructure/en.json | 18 ++++ 5 files changed, 359 insertions(+), 13 deletions(-) create mode 100644 site/en/adminGuide/mq_kafka.md create mode 100644 site/en/adminGuide/mq_rocksmq.md create mode 100644 site/en/adminGuide/mqtype-overview.md diff --git a/site/en/adminGuide/mq_kafka.md b/site/en/adminGuide/mq_kafka.md new file mode 100644 index 000000000..c11b4bc94 --- /dev/null +++ b/site/en/adminGuide/mq_kafka.md @@ -0,0 +1,122 @@ +--- +id: mq_kafka.md +title: Kafka +--- + +# Use Kafka as the Milvus Message Queue + +Apache Kafka is one of the message-queue (WAL) backends Milvus supports for **distributed (cluster)** deployments. Starting from Milvus 2.6.x, [Woodpecker](woodpecker.md) is the default message queue; Kafka remains fully supported for users who prefer it. Kafka is used with Milvus Distributed (cluster) only — standalone deployments use embedded Woodpecker or [RocksMQ](mq_rocksmq.md). + +## Version compatibility + +- Milvus supports **Kafka 2.x and 3.x** only. +- Kafka is configured for Milvus Distributed (cluster) via Helm or Milvus Operator. + +## Deploy a Milvus cluster with Kafka using Helm + +### Install and configure + +To use an external Kafka service, put the Kafka connection settings in a `values.yaml` override, then install Milvus with it: + +```yaml +extraConfigFiles: + user.yaml: |+ + kafka: + brokerList: + - : + saslUsername: + saslPassword: + saslMechanisms: PLAIN + securityProtocol: SASL_SSL +``` + +```bash +helm install my-release zilliztech/milvus -f values.yaml +``` + +For SASL/SSL authentication details, see [Connect to Kafka with SASL/SSL](connect_kafka_ssl.md). + +### Uninstall + +```bash +helm uninstall my-release +``` + +## Deploy a Milvus cluster with Kafka using Milvus Operator + +With Milvus Operator, set `spec.dependencies.msgStreamType: "kafka"` and configure Kafka under `spec.dependencies.kafka` (cluster only). `kafka` supports `external` and `inCluster`. + +### External Kafka + +```yaml +apiVersion: milvus.io/v1alpha1 +kind: Milvus +metadata: + name: my-release + labels: + app: milvus +spec: + config: + kafka: + # securityProtocol supports: PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL + securityProtocol: PLAINTEXT + # saslMechanisms supports: PLAIN, SCRAM-SHA-256, SCRAM-SHA-512 + saslMechanisms: PLAIN + saslUsername: "" + saslPassword: "" + dependencies: + msgStreamType: "kafka" + kafka: + external: true + brokerList: + - "kafkaBrokerAddr1:9092" + - "kafkaBrokerAddr2:9092" +``` + +
+ +SASL configurations are supported in Milvus Operator v0.8.5 or later. + +
+ +### Internal (in-cluster) Kafka + +```yaml +apiVersion: milvus.io/v1alpha1 +kind: Milvus +metadata: + name: my-release + labels: + app: milvus +spec: + dependencies: + msgStreamType: "kafka" + kafka: + inCluster: + values: {} # see https://artifacthub.io/packages/helm/bitnami/kafka + components: {} + config: {} +``` + +Apply the configuration (assuming the file is `milvuscluster.yaml`): + +```bash +kubectl apply -f milvuscluster.yaml +``` + +### Uninstall + +```bash +kubectl delete milvus my-release +``` + +## Notes + +- **Upgrading from 2.5.x to 2.6.x:** {{fragments/mq_upgrade_limitation.md}} If you run Kafka and want to keep it, do not change the message queue during the upgrade. +- Only **Kafka 2.x and 3.x** are supported. +- For SASL/SSL connectivity, see [Connect to Kafka with SASL/SSL](connect_kafka_ssl.md). + +## What's next + +- [Woodpecker (default message queue)](woodpecker.md) +- [Switch MQ Type for Milvus Cluster](switch_milvus_cluster_mq_type-helm.md) diff --git a/site/en/adminGuide/mq_pulsar.md b/site/en/adminGuide/mq_pulsar.md index f7a4c4940..e9bb0d416 100644 --- a/site/en/adminGuide/mq_pulsar.md +++ b/site/en/adminGuide/mq_pulsar.md @@ -5,16 +5,22 @@ title: Pulsar # Use Pulsar as the Milvus Message Queue -Apache Pulsar is one of the message-queue (WAL) backends Milvus supports. Starting from Milvus 2.6.x, **Woodpecker** is the default and recommended message queue; Pulsar remains fully supported for users who prefer it. +Apache Pulsar is one of the message-queue (WAL) backends Milvus supports for **distributed (cluster)** deployments. Starting from Milvus 2.6.x, [Woodpecker](woodpecker.md) is the default message queue; Pulsar remains fully supported for users who prefer it. Pulsar is used with Milvus Distributed (cluster) only — standalone deployments use embedded Woodpecker or [RocksMQ](mq_rocksmq.md). -## Supported versions +## Version compatibility -- Milvus Distributed supports Pulsar as the message queue. -- Since Milvus 2.5, the Milvus Helm chart and Milvus Operator deploy Pulsar v3 by default, while Pulsar v2 remains compatible. See [Upgrade to Pulsar v3](upgrade-pulsar-v3.md) and [Continue Using Pulsar v2](use-pulsar-v2.md). +| Milvus version | Supported Pulsar version | Default | +| --- | --- | --- | +| 2.5.x and later | Pulsar v3 (recommended) or Pulsar v2 | Pulsar v3 (via Helm / Milvus Operator) | +| 2.4.x and earlier | Pulsar v2 | Pulsar v2 | -## Install Milvus with Pulsar (Helm) +Since Milvus 2.5, the Milvus Helm chart and Milvus Operator deploy **Pulsar v3** by default; Pulsar v2 remains compatible. See [Upgrade Pulsar from v2 to v3](upgrade-pulsar-v3.md) and [Continue Using Pulsar v2](use-pulsar-v2.md). -To deploy a Milvus cluster with Pulsar instead of Woodpecker, install the Helm chart without disabling Pulsar: +## Deploy a Milvus cluster with Pulsar using Helm + +### Install + +To deploy a Milvus cluster that uses the bundled Pulsar (instead of Woodpecker), install the Helm chart with the Streaming Node enabled: ```bash helm install my-release zilliztech/milvus \ @@ -23,9 +29,7 @@ helm install my-release zilliztech/milvus \ --set indexNode.enabled=false ``` -### Kubernetes v1.25+ PodDisruptionBudget workaround - -On Kubernetes v1.25 and later, the bundled Pulsar sub-chart may run into PodDisruptionBudget API issues. If you encounter them, disable the Pulsar PodDisruptionBudget policies: +On Kubernetes v1.25 and later, if you hit PodDisruptionBudget API issues from the bundled Pulsar sub-chart, disable the Pulsar PDB policies: ```bash helm install my-release zilliztech/milvus \ @@ -35,12 +39,111 @@ helm install my-release zilliztech/milvus \ --set pulsar.zookeeper.pdb.usePolicy=false ``` -## Configure Pulsar +### Configure + +To connect Milvus to an **external** Pulsar service, put the Pulsar connection settings in a `values.yaml` override and disable the bundled Pulsar: + +```yaml +extraConfigFiles: + user.yaml: |+ + pulsar: + address: localhost # Address of Pulsar + port: 6650 # Port of Pulsar + webport: 80 # Web port of Pulsar; use 8080 if connecting directly without a proxy + maxMessageSize: 5242880 # 5 MB, maximum size of each message + tenant: public + namespace: default +``` + +```bash +helm install my-release zilliztech/milvus --set pulsarv3.enabled=false -f values.yaml +``` + +### Uninstall + +```bash +helm uninstall my-release +``` + +If you used the bundled Pulsar and want to remove its persisted data, delete the Pulsar PVCs (named `my-release-pulsarv3-*`): + +```bash +kubectl get pvc | grep my-release-pulsarv3 +kubectl delete pvc ... +``` + +## Deploy a Milvus cluster with Pulsar using Milvus Operator + +With Milvus Operator, configure Pulsar under `spec.dependencies.pulsar` (supported for Milvus cluster only). `pulsar` supports `external` and `inCluster`. + +### External Pulsar + +```yaml +apiVersion: milvus.io/v1alpha1 +kind: Milvus +metadata: + name: my-release + labels: + app: milvus +spec: + dependencies: + pulsar: + external: true + endpoints: + - 192.168.1.1:6650 + components: {} + config: {} +``` + +### Internal (in-cluster) Pulsar + +```yaml +apiVersion: milvus.io/v1alpha1 +kind: Milvus +metadata: + name: my-release + labels: + app: milvus +spec: + dependencies: + pulsar: + inCluster: + values: + components: + autorecovery: false + zookeeper: + replicaCount: 1 + bookkeeper: + replicaCount: 1 + broker: + replicaCount: 1 + configData: + autoSkipNonRecoverableData: "true" + managedLedgerDefaultEnsembleSize: "1" + managedLedgerDefaultWriteQuorum: "1" + managedLedgerDefaultAckQuorum: "1" + proxy: + replicaCount: 1 + components: {} + config: {} +``` + +Apply the configuration (assuming the file is `milvuscluster.yaml`): + +```bash +kubectl apply -f milvuscluster.yaml +``` + +### Uninstall + +```bash +kubectl delete milvus my-release +``` -For detailed Pulsar parameters across deployment methods, see: +## Notes -- [Configure Pulsar with Docker Compose or Helm](deploy_pulsar.md) -- [Configure Pulsar with Milvus Operator](message_storage_operator.md#Configure-Pulsar) +- **Upgrading from 2.5.x to 2.6.x:** {{fragments/mq_upgrade_limitation.md}} If you run Pulsar and want to keep it, do not change the message queue during the upgrade. +- **Pulsar v2 → v3:** see [Upgrade Pulsar from v2 to v3](upgrade-pulsar-v3.md); to stay on v2, see [Continue Using Pulsar v2](use-pulsar-v2.md). ## What's next diff --git a/site/en/adminGuide/mq_rocksmq.md b/site/en/adminGuide/mq_rocksmq.md new file mode 100644 index 000000000..7e61ef37a --- /dev/null +++ b/site/en/adminGuide/mq_rocksmq.md @@ -0,0 +1,71 @@ +--- +id: mq_rocksmq.md +title: RocksMQ +--- + +# Use RocksMQ as the Milvus Message Queue + +RocksMQ is an embedded message queue (WAL) bundled with Milvus, available for **Milvus Standalone only**. It was the default message queue for standalone deployments up to 2.5.x; from 2.6.x, Milvus Standalone uses embedded [Woodpecker](woodpecker.md) by default. + +## Version compatibility + +- **Standalone only** — RocksMQ is **not** supported in Milvus Distributed (cluster). See the [message queue support matrix](mqtype-overview.md#Supported-message-queues). +- RocksMQ ships with Milvus, so there is no separate version to install. +- It was the default standalone message queue up to 2.5.x, and is superseded by embedded Woodpecker from 2.6.x onward. + +## Deploy Milvus Standalone with RocksMQ using Docker + +### Install + +Follow [Run Milvus in Docker](install_standalone-docker.md). On Milvus 2.6.x and later the standalone default is Woodpecker, so set the message-queue type to RocksMQ explicitly: + +```bash +mkdir milvus-rocksmq && cd milvus-rocksmq +curl -sfL https://raw.githubusercontent.com/milvus-io/milvus/master/scripts/standalone_embed.sh -o standalone_embed.sh + +# Create user.yaml to use RocksMQ +cat > user.yaml <<'EOF' +mq: + type: rocksmq +EOF + +bash standalone_embed.sh start +``` + +### Configure + +To tune RocksMQ, add a `rocksmq` section to `user.yaml` and restart the service: + +```yaml +mq: + type: rocksmq +rocksmq: + path: /var/lib/milvus/rdb_data # where messages are stored + lrucacheratio: 0.06 # rocksdb cache memory ratio + rocksmqPageSize: 67108864 # 64 MB, size of each message page + retentionTimeInMinutes: 4320 # 3 days + retentionSizeInMB: 8192 # 8 GB + compactionInterval: 86400 # 1 day, trigger rocksdb compaction + compressionTypes: [0, 0, 7, 7, 7] +``` + +```bash +bash standalone_embed.sh restart +``` + +### Uninstall + +```bash +bash standalone_embed.sh stop +bash standalone_embed.sh delete +``` + +## Notes + +- **Upgrading from 2.5.x to 2.6.x:** {{fragments/mq_upgrade_limitation.md}} Because 2.6.x changes the standalone default to Woodpecker, pin `mq.type: rocksmq` in your `user.yaml` **before** upgrading if you want to keep RocksMQ. +- To change the message queue of a running instance, see [Switch from RocksMQ to Woodpecker](switch_milvus_standalone_mq_type.md). + +## What's next + +- [Woodpecker (default message queue)](woodpecker.md) +- [Switch from RocksMQ to Woodpecker](switch_milvus_standalone_mq_type.md) diff --git a/site/en/adminGuide/mqtype-overview.md b/site/en/adminGuide/mqtype-overview.md new file mode 100644 index 000000000..0120e21f4 --- /dev/null +++ b/site/en/adminGuide/mqtype-overview.md @@ -0,0 +1,32 @@ +--- +id: mqtype-overview.md +title: Message Queue Overview +summary: Overview of the message queue (mqType) options Milvus supports, and which one to use for standalone vs. distributed deployments. +--- + +# Message Queue Overview + +Milvus relies on a message queue (write-ahead log, WAL) to manage logs of recent changes, output stream logs, and provide log subscriptions. Starting from Milvus 2.6.x, **Woodpecker** is the default message queue and requires no separate messaging infrastructure. Pulsar, Kafka, and RocksMQ remain supported for specific scenarios. + +## Supported message queues + +| Message queue | Milvus Standalone | Milvus Distributed (cluster) | Default in | Notes | +| --- | :---: | :---: | --- | --- | +| [Woodpecker](woodpecker.md) | ✔️ (embedded) | ✔️ (embedded or service) | **2.6.x and later** (both modes) | Default and recommended. Cloud-native WAL on object storage; no external service required. | +| [Pulsar](mq_pulsar.md) | ✔️ | ✔️ | ≤ 2.5.x (cluster default) | Supported, external or bundled. | +| [Kafka](mq_kafka.md) | ✔️ | ✔️ | — | Supported. Only Kafka 2.x or 3.x. | +| [RocksMQ](mq_rocksmq.md) | ✔️ | ✖️ | ≤ 2.5.x (standalone default) | Supported for **standalone only**. | + +
+ +- Each Milvus instance uses exactly one message queue. +- {{fragments/mq_upgrade_limitation.md}} +- To change the message queue of a running instance, see [Switch MQ Type](switch_milvus_cluster_mq_type-helm.md) (supported from v2.6.14). + +
+ +## Choosing a message queue + +- **New deployments (2.6.x / 3.x):** use **Woodpecker** (the default). Standalone runs it embedded; distributed can run it embedded or as a dedicated [service](woodpecker.md#Deployment-modes). +- **Existing Pulsar or Kafka users:** Pulsar and Kafka remain fully supported. Keep them, or [switch to Woodpecker](switch_milvus_cluster_mq_type-helm.md). +- **RocksMQ:** standalone only, and superseded by embedded Woodpecker from 2.6.x onward. diff --git a/site/en/menuStructure/en.json b/site/en/menuStructure/en.json index dbe4f8428..c4496e03a 100644 --- a/site/en/menuStructure/en.json +++ b/site/en/menuStructure/en.json @@ -1483,6 +1483,12 @@ "isMenu": true, "order": 4, "children": [ + { + "label": "Overview", + "id": "mqtype-overview.md", + "order": 0, + "children": [] + }, { "label": "Woodpecker", "id": "woodpecker.md", @@ -1494,6 +1500,18 @@ "id": "mq_pulsar.md", "order": 2, "children": [] + }, + { + "label": "Kafka", + "id": "mq_kafka.md", + "order": 3, + "children": [] + }, + { + "label": "RocksMQ", + "id": "mq_rocksmq.md", + "order": 4, + "children": [] } ] } From 40a839795e74737f147cbd84c8f6bcd8b1d4030d Mon Sep 17 00:00:00 2001 From: tinswzy Date: Thu, 11 Jun 2026 10:28:16 +0800 Subject: [PATCH 13/19] docs: add Data Infrastructure & Integration overview, etcd, and object storage pages; remove the old Manage Dependencies subtree Add the chapter overview, a concise Metadata > etcd page (version + Helm image override), and an Object Storage page (supported-store matrix plus the stricter S3 Conditional-Write requirement when running embedded Woodpecker). Re-parent the Pulsar v2/v3 pages under mqType > Pulsar and remove the legacy Manage Dependencies nav subtree; the detailed deploy_* and *_operator config pages remain and are linked from the new pages. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../data-infra-integration-overview.md | 15 +++ site/en/adminGuide/etcd.md | 24 ++++ site/en/adminGuide/object-storage.md | 29 +++++ site/en/menuStructure/en.json | 121 ++++++------------ 4 files changed, 108 insertions(+), 81 deletions(-) create mode 100644 site/en/adminGuide/data-infra-integration-overview.md create mode 100644 site/en/adminGuide/etcd.md create mode 100644 site/en/adminGuide/object-storage.md diff --git a/site/en/adminGuide/data-infra-integration-overview.md b/site/en/adminGuide/data-infra-integration-overview.md new file mode 100644 index 000000000..3691167ad --- /dev/null +++ b/site/en/adminGuide/data-infra-integration-overview.md @@ -0,0 +1,15 @@ +--- +id: data-infra-integration-overview.md +title: Data Infrastructure & Integration +summary: Overview of the third-party infrastructure Milvus integrates with — metadata, object storage, and message queues. +--- + +# Data Infrastructure & Integration + +Milvus builds on open data infrastructure for its core dependencies. This chapter covers the components you can plug in and configure: + +- **[Metadata](etcd.md)** — Milvus stores metadata (collection schemas, node status, consumption checkpoints) in etcd. +- **[Object Storage](object-storage.md)** — Milvus stores index files and binary logs in MinIO, AWS S3, or other S3-compatible / cloud object storage. +- **[Message Queue](mqtype-overview.md)** — Milvus uses a write-ahead log (WAL): Woodpecker (default), Pulsar, Kafka, or RocksMQ. + +By default, a new Milvus 2.6.x / 3.x deployment runs with **Woodpecker** as the message queue, **etcd** for metadata, and **MinIO** for object storage — no extra messaging infrastructure required. diff --git a/site/en/adminGuide/etcd.md b/site/en/adminGuide/etcd.md new file mode 100644 index 000000000..027eba05a --- /dev/null +++ b/site/en/adminGuide/etcd.md @@ -0,0 +1,24 @@ +--- +id: etcd.md +title: etcd +--- + +# etcd (Metadata) + +Milvus uses **etcd** to store metadata such as collection schemas, node status, and message-consumption checkpoints. + +## Version + +Milvus is validated against **etcd 3.5.x**. When you install Milvus with Helm, the bundled etcd image is `milvusdb/etcd:3.5.25-r1` by default. + +## Change the etcd image with Helm + +To pin or replace the etcd image version, override `etcd.image.tag` (and, if needed, `etcd.image.repository`) when installing or upgrading with Helm: + +```bash +helm install my-release zilliztech/milvus \ + --set etcd.image.repository=milvusdb/etcd \ + --set etcd.image.tag=3.5.25-r1 +``` + +For external etcd, or for detailed Docker Compose / Helm / Milvus Operator configuration, see [Configure Meta Storage with Docker Compose or Helm](deploy_etcd.md) and [Configure Meta Storage with Milvus Operator](meta_storage_operator.md). diff --git a/site/en/adminGuide/object-storage.md b/site/en/adminGuide/object-storage.md new file mode 100644 index 000000000..33021d1c0 --- /dev/null +++ b/site/en/adminGuide/object-storage.md @@ -0,0 +1,29 @@ +--- +id: object-storage.md +title: Object Storage +--- + +# Object Storage + +Milvus stores index files and binary logs — the bulk of its data — in object storage. Milvus supports MinIO and a range of S3-compatible and cloud object stores. + +## Supported object storage + +| Provider / service | Supported as Milvus object storage | +| --- | :---: | +| MinIO | ✔️ (default for self-hosted deployments) | +| AWS S3 | ✔️ | +| Azure Blob Storage | ✔️ | +| Google Cloud Storage (GCS) | ✔️ | +| Aliyun OSS | ✔️ | +| Tencent COS | ✔️ | +| Huawei Cloud OBS | ✔️ | +| Other S3-compatible storage | ✔️ | + +For configuration details, see [Configure Object Storage with Docker Compose or Helm](deploy_s3.md) and [Configure Object Storage with Milvus Operator](object_storage_operator.md). + +## Additional requirements when using embedded Woodpecker + +When you run the default **Woodpecker** message queue with its object-storage backend (`storage.type=minio`), Woodpecker writes its write-ahead log to the same object storage and requires **strict S3 Conditional-Write semantics**. Not every object store qualifies — for example, Huawei Cloud OBS is currently **unsupported** as a Woodpecker backend even though it works as regular Milvus object storage. + +See the object storage compatibility matrix on the [Woodpecker](woodpecker.md) page for the exact per-provider requirements. diff --git a/site/en/menuStructure/en.json b/site/en/menuStructure/en.json index c4496e03a..f757ce43a 100644 --- a/site/en/menuStructure/en.json +++ b/site/en/menuStructure/en.json @@ -1477,6 +1477,32 @@ "isMenu": true, "order": 9, "children": [ + { + "label": "Overview", + "id": "data-infra-integration-overview.md", + "order": 0, + "children": [] + }, + { + "label": "Metadata", + "id": "metadata", + "isMenu": true, + "order": 1, + "children": [ + { + "label": "etcd", + "id": "etcd.md", + "order": 0, + "children": [] + } + ] + }, + { + "label": "Object Storage", + "id": "object-storage.md", + "order": 2, + "children": [] + }, { "label": "Message Queue (mqType)", "id": "mqtype", @@ -1499,7 +1525,20 @@ "label": "Pulsar", "id": "mq_pulsar.md", "order": 2, - "children": [] + "children": [ + { + "label": "Upgrade to Pulsar v3", + "id": "upgrade-pulsar-v3.md", + "order": 0, + "children": [] + }, + { + "label": "Continue Using Pulsar v2", + "id": "use-pulsar-v2.md", + "order": 1, + "children": [] + } + ] }, { "label": "Kafka", @@ -1691,86 +1730,6 @@ } ] }, - { - "label": "Manage Dependencies", - "id": "manage_dependencies", - "order": 2, - "isMenu": true, - "children": [ - { - "label": "With Docker or Helm", - "id": "docker_helm_dependencies", - "order": 0, - "isMenu": true, - "children": [ - { - "label": "Object Storage", - "id": "deploy_s3.md", - "order": 0, - "children": [] - }, - { - "label": "Meta Storage", - "id": "deploy_etcd.md", - "order": 1, - "children": [] - }, - { - "label": "Message Storage", - "id": "deploy_pulsar.md", - "order": 2, - "children": [] - } - ] - }, - { - "label": "With Milvus Operator", - "id": "milvus_operator_dependencies", - "order": 1, - "isMenu": true, - "children": [ - { - "label": "Object Storage", - "id": "object_storage_operator.md", - "order": 0, - "children": [] - }, - { - "label": "Meta Storage", - "id": "meta_storage_operator.md", - "order": 1, - "children": [] - }, - { - "label": "Message Storage", - "id": "message_storage_operator.md", - "order": 2, - "children": [] - } - ] - }, - { - "label": "Use Pulsar with Milvus", - "id": "pulsar_dependencies", - "order": 2, - "isMenu": true, - "children": [ - { - "label": "Upgrade to Pulsar v3", - "id": "upgrade-pulsar-v3.md", - "order": 0, - "children": [] - }, - { - "label": "Continue Using Pulsar v2", - "id": "use-pulsar-v2.md", - "order": 1, - "children": [] - } - ] - } - ] - }, { "label": "Scaling", "id": "scaling", From 66271b636354a9a14dba504a2a1abd1d669917b4 Mon Sep 17 00:00:00 2001 From: tinswzy Date: Thu, 11 Jun 2026 12:01:09 +0800 Subject: [PATCH 14/19] docs: restructure Switch MQ Type into Woodpecker-paired pages MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace the deployment-organized switch docs (standalone / cluster-helm / cluster-operator) with an Overview plus three Woodpecker-paired pages — RocksMQ, Pulsar, and Kafka each <-> Woodpecker — covering both directions, Helm and Operator where applicable (WP->Pulsar/Kafka noted as Operator- unsupported). Pulsar<->Kafka is explicitly out of scope. Repoint inbound links and rebuild the Switch MQ Type nav. Co-Authored-By: Claude Opus 4.8 (1M context) --- site/en/adminGuide/mq_kafka.md | 2 +- site/en/adminGuide/mq_pulsar.md | 2 +- site/en/adminGuide/mq_rocksmq.md | 4 +- site/en/adminGuide/mqtype-overview.md | 4 +- site/en/adminGuide/switch-kafka-woodpecker.md | 145 ++++++++++ site/en/adminGuide/switch-mq-type.md | 60 ++++ .../en/adminGuide/switch-pulsar-woodpecker.md | 157 +++++++++++ .../adminGuide/switch-rocksmq-woodpecker.md | 110 ++++++++ .../switch_milvus_cluster_mq_type-helm.md | 260 ------------------ .../switch_milvus_cluster_mq_type-operator.md | 175 ------------ .../switch_milvus_standalone_mq_type.md | 203 -------------- .../install_standalone-docker-compose.md | 2 +- site/en/menuStructure/en.json | 35 ++- 13 files changed, 496 insertions(+), 663 deletions(-) create mode 100644 site/en/adminGuide/switch-kafka-woodpecker.md create mode 100644 site/en/adminGuide/switch-mq-type.md create mode 100644 site/en/adminGuide/switch-pulsar-woodpecker.md create mode 100644 site/en/adminGuide/switch-rocksmq-woodpecker.md delete mode 100644 site/en/adminGuide/switch_milvus_cluster_mq_type-helm.md delete mode 100644 site/en/adminGuide/switch_milvus_cluster_mq_type-operator.md delete mode 100644 site/en/adminGuide/switch_milvus_standalone_mq_type.md diff --git a/site/en/adminGuide/mq_kafka.md b/site/en/adminGuide/mq_kafka.md index c11b4bc94..58eaa99ed 100644 --- a/site/en/adminGuide/mq_kafka.md +++ b/site/en/adminGuide/mq_kafka.md @@ -119,4 +119,4 @@ kubectl delete milvus my-release ## What's next - [Woodpecker (default message queue)](woodpecker.md) -- [Switch MQ Type for Milvus Cluster](switch_milvus_cluster_mq_type-helm.md) +- [Switch MQ Type for Milvus Cluster](switch-kafka-woodpecker.md) diff --git a/site/en/adminGuide/mq_pulsar.md b/site/en/adminGuide/mq_pulsar.md index e9bb0d416..4c3c82a52 100644 --- a/site/en/adminGuide/mq_pulsar.md +++ b/site/en/adminGuide/mq_pulsar.md @@ -148,4 +148,4 @@ kubectl delete milvus my-release ## What's next - [Woodpecker (default message queue)](woodpecker.md) -- [Switch MQ Type for Milvus Cluster](switch_milvus_cluster_mq_type-helm.md) +- [Switch MQ Type for Milvus Cluster](switch-pulsar-woodpecker.md) diff --git a/site/en/adminGuide/mq_rocksmq.md b/site/en/adminGuide/mq_rocksmq.md index 7e61ef37a..d6f70e0cd 100644 --- a/site/en/adminGuide/mq_rocksmq.md +++ b/site/en/adminGuide/mq_rocksmq.md @@ -63,9 +63,9 @@ bash standalone_embed.sh delete ## Notes - **Upgrading from 2.5.x to 2.6.x:** {{fragments/mq_upgrade_limitation.md}} Because 2.6.x changes the standalone default to Woodpecker, pin `mq.type: rocksmq` in your `user.yaml` **before** upgrading if you want to keep RocksMQ. -- To change the message queue of a running instance, see [Switch from RocksMQ to Woodpecker](switch_milvus_standalone_mq_type.md). +- To change the message queue of a running instance, see [Switch from RocksMQ to Woodpecker](switch-rocksmq-woodpecker.md). ## What's next - [Woodpecker (default message queue)](woodpecker.md) -- [Switch from RocksMQ to Woodpecker](switch_milvus_standalone_mq_type.md) +- [Switch from RocksMQ to Woodpecker](switch-rocksmq-woodpecker.md) diff --git a/site/en/adminGuide/mqtype-overview.md b/site/en/adminGuide/mqtype-overview.md index 0120e21f4..50348b086 100644 --- a/site/en/adminGuide/mqtype-overview.md +++ b/site/en/adminGuide/mqtype-overview.md @@ -21,12 +21,12 @@ Milvus relies on a message queue (write-ahead log, WAL) to manage logs of recent - Each Milvus instance uses exactly one message queue. - {{fragments/mq_upgrade_limitation.md}} -- To change the message queue of a running instance, see [Switch MQ Type](switch_milvus_cluster_mq_type-helm.md) (supported from v2.6.14). +- To change the message queue of a running instance, see [Switch MQ Type](switch-mq-type.md) (supported from v2.6.14). ## Choosing a message queue - **New deployments (2.6.x / 3.x):** use **Woodpecker** (the default). Standalone runs it embedded; distributed can run it embedded or as a dedicated [service](woodpecker.md#Deployment-modes). -- **Existing Pulsar or Kafka users:** Pulsar and Kafka remain fully supported. Keep them, or [switch to Woodpecker](switch_milvus_cluster_mq_type-helm.md). +- **Existing Pulsar or Kafka users:** Pulsar and Kafka remain fully supported. Keep them, or [switch to Woodpecker](switch-mq-type.md). - **RocksMQ:** standalone only, and superseded by embedded Woodpecker from 2.6.x onward. diff --git a/site/en/adminGuide/switch-kafka-woodpecker.md b/site/en/adminGuide/switch-kafka-woodpecker.md new file mode 100644 index 000000000..485c49cb7 --- /dev/null +++ b/site/en/adminGuide/switch-kafka-woodpecker.md @@ -0,0 +1,145 @@ +--- +id: switch-kafka-woodpecker.md +title: Switch between Kafka and Woodpecker +summary: Switch the message queue of a Milvus cluster between Kafka and Woodpecker, with Helm or Milvus Operator. +--- + +# Switch between Kafka and Woodpecker + +This page describes how to switch the message queue (MQ) of a **Milvus cluster** between **Kafka** (builtin or external) and **Woodpecker** (MinIO backend), in both directions, for Helm and Milvus Operator deployments. For the general workflow and prerequisites, see [Switch MQ Type](switch-mq-type.md). + +## Switch from Kafka to Woodpecker + +### Step 1: Verify the Milvus instance is running + +Before switching, ensure your Milvus cluster is running properly — for example, by creating a test collection, inserting data, and running a query. + +### Step 2: Execute the MQ switch + +```shell +curl -X POST http://:/management/wal/alter \ + -H "Content-Type: application/json" \ + -d '{"target_wal_name": "woodpecker"}' +``` + +
+ +Expose the MixCoord management interface to reach it: + +- **Helm:** use `kubectl port-forward --address 0.0.0.0 service/my-release-milvus-mixcoord 29091:9091`; then `mixcoord_addr` is the machine's IP and `mixcoord_port` is `29091`. +- **Milvus Operator:** the MixCoord service is not exposed — use `kubectl exec` into the MixCoord pod and run the `curl` command from inside the pod. + +
+ +### Step 3: Verify the switch is complete + +```shell +kubectl logs | grep "successfully updated mq.type configuration in etcd" +``` + +A successful switch logs `[mqTypeValue=woodpecker]`. + +### Step 4 (Milvus Operator only): Update the MQ type in the Operator + +Update the MQ type in the Operator-managed configuration. Create `change_configmap.yaml`: + +```yaml +apiVersion: milvus.io/v1beta1 +kind: Milvus +metadata: + name: my-release + labels: + app: milvus +spec: + dependencies: + msgStreamType: woodpecker +``` + +```shell +kubectl patch -f change_configmap.yaml --patch-file change_configmap.yaml --type merge +``` + +### Step 5: (Optional) Stop Kafka and clean up + +**For builtin Kafka:** after confirming the switch, remove the Kafka pods and their PVCs. + +**For external Kafka:** clean up the Milvus topics in the external Kafka instance. Milvus topics follow the format `-dml__`. + +
+ +If you plan to switch back to Kafka later, clean up the data/topics first to avoid conflicts. + +
+ +## Switch from Woodpecker to Kafka + +
+ +This direction is supported with **Helm** only. With **Milvus Operator**, switching from Woodpecker to Pulsar/Kafka is **not supported** — the Operator does not allow pre-configuring the Kafka connection address that the switch requires. + +
+ +### Step 1: Verify the Milvus instance is running + +Ensure your Milvus cluster is running properly. + +### Step 2: Configure the target Kafka connection + +Render the Kafka access configuration into Milvus and enable the target MQ. The `streaming.enabled=true` flag is required for the Switch MQ feature. Create or edit `values.yaml`: + +```yaml +extraConfigFiles: + user.yaml: |+ + kafka: + brokerList: + - : + saslUsername: + saslPassword: + saslMechanisms: PLAIN + securityProtocol: SASL_SSL +``` + +```shell +helm upgrade -i my-release milvus/milvus \ + --set kafka.enabled=true \ + --set woodpecker.enabled=false \ + --set streaming.enabled=true \ + -f values.yaml +``` + +Wait for all pods to be ready, then verify that the Kafka access configuration has been rendered into the Milvus configuration. For SASL/SSL details, see [Connect to Kafka with SASL/SSL](connect_kafka_ssl.md). + +### Step 3: Execute the MQ switch + +
+ +Ensure the target Kafka does not contain Milvus topics from a previous configuration. If this is your first switch to Kafka, skip this note; otherwise clean up residual Milvus topics with the same names first. + +
+ +```shell +curl -X POST http://:/management/wal/alter \ + -H "Content-Type: application/json" \ + -d '{"target_wal_name": "kafka"}' +``` + +### Step 4: Verify the switch is complete + +```shell +kubectl logs | grep "successfully updated mq.type configuration in etcd" +``` + +A successful switch logs `[mqTypeValue=kafka]`. + +### Step 5: (Optional) Clean up Woodpecker data + +Delete the Woodpecker data on MinIO/S3 (under `/wp/...`, typically `files/wp/...`) and the Woodpecker metadata in etcd (`etcdctl get woodpecker --prefix`). If you plan to switch back to Woodpecker later, clean up these files first. + +## Supported scenarios + +| Source MQ | Target MQ | Helm | Milvus Operator | +|-----------|-----------|------|-----------------| +| Builtin Kafka | Woodpecker (MinIO) | **Supported** | **Supported** | +| External Kafka | Woodpecker (MinIO) | **Supported** | **Supported** | +| Woodpecker (MinIO) | External Kafka | **Supported** | **Not supported** | +| Kafka | Woodpecker (local) | **Supported but not recommended** (all pods need a shared FS) | **Not supported** | diff --git a/site/en/adminGuide/switch-mq-type.md b/site/en/adminGuide/switch-mq-type.md new file mode 100644 index 000000000..432892678 --- /dev/null +++ b/site/en/adminGuide/switch-mq-type.md @@ -0,0 +1,60 @@ +--- +id: switch-mq-type.md +title: Switch MQ Type +summary: Switch the message queue of an existing Milvus deployment between Woodpecker and another message queue without downtime. +--- + +# Switch MQ Type + +This guide describes how to switch the message queue (MQ) of an existing Milvus deployment **between Woodpecker and another message queue**, online and without downtime. + +
+ +This feature is pending release and is subject to change. Please reach out to Milvus support if you want to try it out or have any questions. + +
+ +## Prerequisites + +- The Milvus instance has been upgraded to a version that supports the Switch MQ feature (**v2.6.14 or later**). +- The instance is running properly. + +## Scope + +This guide covers switching **between Woodpecker and another message queue** only. Switching directly between Pulsar and Kafka is out of scope. + +- [Switch between RocksMQ and Woodpecker](switch-rocksmq-woodpecker.md) — Milvus Standalone (Docker Compose) +- [Switch between Pulsar and Woodpecker](switch-pulsar-woodpecker.md) — Milvus cluster (Helm / Milvus Operator) +- [Switch between Kafka and Woodpecker](switch-kafka-woodpecker.md) — Milvus cluster (Helm / Milvus Operator) + +## General workflow + +1. Ensure the Milvus instance is running properly. +2. Confirm the source MQ type and the target MQ type. +3. Render the target MQ's access settings into the Milvus configuration **without** changing the `mqType` value. +4. Trigger the switch by calling the WAL alter API on MixCoord. +5. Monitor the logs to confirm the switch has completed. + +
+ +Before switching, ensure that the target MQ does not contain topics with the same names as those used by the current Milvus instance. This is especially important if the target MQ has been used by another Milvus instance, as conflicting topic names can lead to unexpected behavior. + +
+ +## Support matrix + +| Source MQ | Target MQ | Deployment | Status | +|-----------|-----------|------------|--------| +| RocksMQ | Woodpecker (local/MinIO) | Standalone (Docker Compose) | **Supported** | +| Woodpecker (local/MinIO) | RocksMQ | Standalone (Docker Compose) | **Supported** | +| Pulsar (builtin/external) | Woodpecker (MinIO) | Cluster (Helm / Operator) | **Supported** | +| Woodpecker (MinIO) | Pulsar (external) | Cluster (Helm) | **Supported** (Operator: **not supported**) | +| Kafka (builtin/external) | Woodpecker (MinIO) | Cluster (Helm / Operator) | **Supported** | +| Woodpecker (MinIO) | Kafka (external) | Cluster (Helm) | **Supported** (Operator: **not supported**) | +| Woodpecker MinIO | Woodpecker local (or vice versa) | any | **Not supported** | + +
+ +Avoid switching MQ types back and forth repeatedly. If you do need to switch, make sure to clean up the related data before each switch — residual data may cause unexpected behavior. + +
diff --git a/site/en/adminGuide/switch-pulsar-woodpecker.md b/site/en/adminGuide/switch-pulsar-woodpecker.md new file mode 100644 index 000000000..8c6493614 --- /dev/null +++ b/site/en/adminGuide/switch-pulsar-woodpecker.md @@ -0,0 +1,157 @@ +--- +id: switch-pulsar-woodpecker.md +title: Switch between Pulsar and Woodpecker +summary: Switch the message queue of a Milvus cluster between Pulsar and Woodpecker, with Helm or Milvus Operator. +--- + +# Switch between Pulsar and Woodpecker + +This page describes how to switch the message queue (MQ) of a **Milvus cluster** between **Pulsar** (builtin or external) and **Woodpecker** (MinIO backend), in both directions, for Helm and Milvus Operator deployments. For the general workflow and prerequisites, see [Switch MQ Type](switch-mq-type.md). + +## Switch from Pulsar to Woodpecker + +### Step 1: Verify the Milvus instance is running + +Before switching, ensure your Milvus cluster is running properly — for example, by creating a test collection, inserting data, and running a query. + +### Step 2: Execute the MQ switch + +```shell +curl -X POST http://:/management/wal/alter \ + -H "Content-Type: application/json" \ + -d '{"target_wal_name": "woodpecker"}' +``` + +
+ +Expose the MixCoord management interface to reach it: + +- **Helm:** use `kubectl port-forward --address 0.0.0.0 service/my-release-milvus-mixcoord 29091:9091`; then `mixcoord_addr` is the machine's IP and `mixcoord_port` is `29091`. +- **Milvus Operator:** the MixCoord service is not exposed — use `kubectl exec` into the MixCoord pod and run the `curl` command from inside the pod. + +
+ +### Step 3: Verify the switch is complete + +```shell +kubectl logs | grep "successfully updated mq.type configuration in etcd" +``` + +A successful switch logs `[mqTypeValue=woodpecker]`. + +### Step 4 (Milvus Operator only): Update the MQ type in the Operator + +Update the MQ type in the Operator-managed configuration. Create `change_configmap.yaml`: + +```yaml +apiVersion: milvus.io/v1beta1 +kind: Milvus +metadata: + name: my-release + labels: + app: milvus +spec: + dependencies: + msgStreamType: woodpecker +``` + +```shell +kubectl patch -f change_configmap.yaml --patch-file change_configmap.yaml --type merge +``` + +### Step 5: (Optional) Stop Pulsar and clean up + +**For builtin Pulsar (Helm):** disable Pulsar and enable Woodpecker, then delete the Pulsar PVCs. + +```shell +helm upgrade my-release zilliztech/milvus \ + --set image.all.tag=v{{var.milvus_release_version}} \ + --set pulsarv3.enabled=false \ + --set woodpecker.enabled=true \ + --set streaming.enabled=true \ + --set indexNode.enabled=false +``` + +```shell +kubectl get pvc | grep my-release-pulsarv3 +kubectl delete pvc ... +``` + +**For builtin Pulsar (Operator):** `helm uninstall my-release-pulsar`, then delete the Pulsar PVCs. + +**For external Pulsar:** clean up the Milvus topics in the external Pulsar instance. Milvus topics follow the format `-dml__` (for example, `by-dev-rootcoord-dml_10_464633776992639586v0`). + +
+ +If you plan to switch back to Pulsar later, clean up the data/topics first to avoid conflicts. Due to Helm chart limitations, switching back to a **builtin** Pulsar instance is currently not possible. + +
+ +## Switch from Woodpecker to Pulsar + +
+ +This direction is supported with **Helm** only. With **Milvus Operator**, switching from Woodpecker to Pulsar/Kafka is **not supported** — the Operator does not allow pre-configuring the Pulsar connection address that the switch requires. + +
+ +### Step 1: Verify the Milvus instance is running + +Ensure your Milvus cluster is running properly. + +### Step 2: Configure the target Pulsar connection + +Render the Pulsar access configuration into Milvus and enable the target MQ. The `streaming.enabled=true` flag is required for the Switch MQ feature. Create or edit `values.yaml`: + +```yaml +extraConfigFiles: + user.yaml: |+ + pulsar: + address: + port: +``` + +```shell +helm upgrade -i my-release milvus/milvus \ + --set pulsarv3.enabled=true \ + --set woodpecker.enabled=false \ + --set streaming.enabled=true \ + -f values.yaml +``` + +Wait for all pods to be ready, then verify that the Pulsar access configuration has been rendered into the Milvus configuration. + +### Step 3: Execute the MQ switch + +
+ +Ensure the target Pulsar does not contain Milvus topics from a previous configuration. If this is your first switch to Pulsar, skip this note; otherwise clean up residual Milvus topics with the same names first. + +
+ +```shell +curl -X POST http://:/management/wal/alter \ + -H "Content-Type: application/json" \ + -d '{"target_wal_name": "pulsar"}' +``` + +### Step 4: Verify the switch is complete + +```shell +kubectl logs | grep "successfully updated mq.type configuration in etcd" +``` + +A successful switch logs `[mqTypeValue=pulsar]`. + +### Step 5: (Optional) Clean up Woodpecker data + +Delete the Woodpecker data on MinIO/S3 (under `/wp/...`, typically `files/wp/...`) and the Woodpecker metadata in etcd (`etcdctl get woodpecker --prefix`). If you plan to switch back to Woodpecker later, clean up these files first. + +## Supported scenarios + +| Source MQ | Target MQ | Helm | Milvus Operator | +|-----------|-----------|------|-----------------| +| Builtin Pulsar | Woodpecker (MinIO) | **Supported** | **Supported** | +| External Pulsar | Woodpecker (MinIO) | **Supported** | **Supported** | +| Woodpecker (MinIO) | External Pulsar | **Supported** | **Not supported** | +| Pulsar | Woodpecker (local) | **Supported but not recommended** (all pods need a shared FS) | **Not supported** | diff --git a/site/en/adminGuide/switch-rocksmq-woodpecker.md b/site/en/adminGuide/switch-rocksmq-woodpecker.md new file mode 100644 index 000000000..cbb2a8022 --- /dev/null +++ b/site/en/adminGuide/switch-rocksmq-woodpecker.md @@ -0,0 +1,110 @@ +--- +id: switch-rocksmq-woodpecker.md +title: Switch between RocksMQ and Woodpecker +summary: Switch the message queue of a Milvus Standalone (Docker Compose) deployment between RocksMQ and Woodpecker. +--- + +# Switch between RocksMQ and Woodpecker + +This page describes how to switch the message queue (MQ) of a **Milvus Standalone (Docker Compose)** deployment between **RocksMQ** and **Woodpecker** (local or MinIO backend), in both directions. For the general workflow and prerequisites, see [Switch MQ Type](switch-mq-type.md). + +
+ +MQ switching requires the Docker **Compose** deployment (which enables an etcd config source). The single-container Docker deployment does not support switching. + +
+ +## Switch from RocksMQ to Woodpecker + +### Step 1: Verify the Milvus instance is running + +Verify your Milvus Standalone Docker Compose instance is running properly — for example, by creating a test collection, inserting data, and running a query. + +### Step 2: Configure Woodpecker storage + +Add the Woodpecker settings to the Milvus configuration **without** changing the `mqType` value. Run `docker exec -it milvus-standalone bash` to enter the container, then edit `/milvus/configs/user.yaml`: + +```yaml +woodpecker: + storage: + type: minio # minio or local +``` + +Restart the Milvus instance to apply the configuration: + +```shell +docker-compose restart +``` + +### Step 3: Execute the MQ switch + +
+ +If this is your first time switching to Woodpecker, skip this note. Otherwise, clean up residual Woodpecker meta and data before switching again — residual data may cause unexpected behavior. + +
+ +```shell +curl -X POST http://:/management/wal/alter \ + -H "Content-Type: application/json" \ + -d '{"target_wal_name": "woodpecker"}' +``` + +The MixCoord port is typically `9091`. + +### Step 4: Verify the switch is complete + +```shell +docker logs milvus-standalone | grep "successfully updated mq.type configuration in etcd" +``` + +A successful switch logs `[mqTypeValue=woodpecker]`. + +### Step 5: (Optional) Clean up RocksMQ data + +RocksMQ data is in the `volumes/milvus/rdb_data` and `volumes/milvus/rdb_data_meta_kv` directories defined in `docker-compose.yaml`. If you plan to switch back to RocksMQ later, clean up these files first to avoid conflicts. + +## Switch from Woodpecker to RocksMQ + +### Step 1: Verify the Milvus instance is running + +Ensure your Milvus Standalone Docker Compose instance is running properly. + +### Step 2: Execute the MQ switch + +
+ +Ensure the instance has no residual RocksMQ data from a previous run. If this is your first time switching to RocksMQ, skip this note; otherwise clean up the related RocksMQ meta and data first. + +
+ +```shell +curl -X POST http://:/management/wal/alter \ + -H "Content-Type: application/json" \ + -d '{"target_wal_name": "rocksmq"}' +``` + +### Step 3: Verify the switch is complete + +```shell +docker logs milvus-standalone | grep "successfully updated mq.type configuration in etcd" +``` + +A successful switch logs `[mqTypeValue=rocksmq]`. + +### Step 4: (Optional) Clean up Woodpecker data + +- **Metadata (etcd):** the Woodpecker key prefix is typically `woodpecker/...`. View it with `etcdctl get woodpecker --prefix`, then delete it. +- **Storage data:** in **MinIO mode**, delete the log data under `/wp/...` (typically `files/wp/...`) in the bucket; in **local mode**, the data is on local disk at `volumes/milvus/data/wp/...`. + +If you plan to switch back to Woodpecker later, clean up these files first to avoid conflicts. + +## Supported scenarios + +| Source MQ | Target MQ | Status | Notes | +|-----------|-----------|--------|-------| +| RocksMQ | Woodpecker (MinIO/local) | **Supported** | | +| Woodpecker (MinIO/local) | RocksMQ | **Supported** | | +| Woodpecker MinIO | Woodpecker local | **Not supported** | Switching between Woodpecker storage modes requires additional metadata handling, which is not yet supported. | +| Woodpecker local | Woodpecker MinIO | **Not supported** | Same as above. | +| RocksMQ / Woodpecker | External Pulsar / Kafka | **Supported but not recommended** | Keep standalone instances as simple as possible. | diff --git a/site/en/adminGuide/switch_milvus_cluster_mq_type-helm.md b/site/en/adminGuide/switch_milvus_cluster_mq_type-helm.md deleted file mode 100644 index 13f7e3155..000000000 --- a/site/en/adminGuide/switch_milvus_cluster_mq_type-helm.md +++ /dev/null @@ -1,260 +0,0 @@ ---- -id: switch_milvus_cluster_mq_type-helm.md -summary: Learn how to switch the message queue type for a Milvus cluster. -title: Switch MQ Type for Milvus Cluster ---- - -# Switch MQ Type for Milvus Cluster - -This topic describes how to switch the message queue (MQ) type for an existing Milvus cluster deployment. Milvus supports online MQ switching between Pulsar, Kafka, and Woodpecker without downtime. - -
- -This feature is pending release and is subject to change. Please reach out to Milvus support if you want to try it out or have any questions. - -
- -## Prerequisites - -- A running Milvus cluster instance installed via [Helm](install_cluster-helm.md). -- The Milvus instance has been upgraded to the latest version that supports this Switch MQ feature (v2.6.14 or later). - -## Switch from builtin/external Pulsar/Kafka to Woodpecker (MinIO) - -Follow these steps to switch the MQ type from Pulsar or Kafka to Woodpecker with MinIO storage. - -### Step 1: Verify the Milvus instance is running - -Before switching, ensure that your Milvus cluster instance is running properly. You can verify this by creating a test collection, inserting data, and running a query. - -### Step 2: Execute the MQ switch - -
- -If you are switching to Woodpecker for the first time, you can skip this note. Otherwise, when switching repeatedly, make sure to clean up the related data before switching. Residual data may cause unexpected behavior. For example, switching from Woodpecker (MinIO/local) to another MQ and then back to Woodpecker requires cleaning up Woodpecker meta and data first. - -
- -Run the following command to trigger the switch to Woodpecker: - -```shell -curl -X POST http://:/management/wal/alter \ - -H "Content-Type: application/json" \ - -d '{"target_wal_name": "woodpecker"}' -``` - -
- -Replace `` and `` with the actual address of your MixCoord service. You can typically use `kubectl port-forward` to expose the MixCoord management interface: - -```shell -kubectl port-forward --address 0.0.0.0 service/my-release-milvus-mixcoord 29091:9091 -``` - -In this case, `mixcoord_addr` is the IP of the machine running the command, and `mixcoord_port` is the forwarded port `29091`. - -
- -### Step 3: Verify the switch is complete - -The switch process completes automatically. Monitor the Milvus logs for the following key messages to confirm the switch has finished: - -```shell -kubectl logs my-release-milvus-mixcoord-68d9889549-vdj8n | grep "successfully updated mq.type configuration in etcd" -``` - -If the switch is successful, the output should look similar to the following: - -``` -[INFO] [coordinator/wal_callbacks.go:90] ["successfully updated mq.type configuration in etcd"] -[targetWALName=WoodPecker] [config=null] [broadcastID=464632819924869973] [configKey=mq.type] [mqTypeValue=woodpecker] -``` - -### Step 4: [Optional] Stop Pulsar pods and clean up Pulsar data - -#### For builtin Pulsar instances - -After confirming the switch to Woodpecker is successful, you can remove the Pulsar pods and their associated PVC data. - -Run the following command to disable Pulsar and enable Woodpecker: - -```shell -helm upgrade my-release zilliztech/milvus \ - --set image.all.tag=v2.6.13 \ - --set pulsarv3.enabled=false \ - --set woodpecker.enabled=true \ - --set streaming.enabled=true \ - --set indexNode.enabled=false -``` - -Run the following commands to delete the Pulsar PVC storage: - -```shell -# Delete Pulsar bookie journal PVCs -kubectl delete pvc my-release-pulsarv3-bookie-journal-my-release-pulsarv3-bookie-0 -kubectl delete pvc my-release-pulsarv3-bookie-journal-my-release-pulsarv3-bookie-1 -kubectl delete pvc my-release-pulsarv3-bookie-journal-my-release-pulsarv3-bookie-2 - -# Delete Pulsar bookie ledger PVCs -kubectl delete pvc my-release-pulsarv3-bookie-ledgers-my-release-pulsarv3-bookie-0 -kubectl delete pvc my-release-pulsarv3-bookie-ledgers-my-release-pulsarv3-bookie-1 -kubectl delete pvc my-release-pulsarv3-bookie-ledgers-my-release-pulsarv3-bookie-2 - -# Delete Pulsar ZooKeeper PVCs -kubectl delete pvc my-release-pulsarv3-zookeeper-data-my-release-pulsarv3-zookeeper-0 -kubectl delete pvc my-release-pulsarv3-zookeeper-data-my-release-pulsarv3-zookeeper-1 -kubectl delete pvc my-release-pulsarv3-zookeeper-data-my-release-pulsarv3-zookeeper-2 -``` - -
- -If you plan to switch back to Pulsar in the future, you should clean up these data files first to avoid conflicts. Currently, due to Helm chart limitations, it is not possible to switch back to a builtin Pulsar instance. - -
- -#### For external Pulsar instances - -You can clean up the Milvus-related topics in the external Pulsar instance. The Milvus topic naming format is: - -``` --dml__ -``` - -For example: `by-dev-rootcoord-dml_10_464633776992639586v0` - -
- -If you plan to switch back to external Pulsar in the future, you should clean up these topics first to avoid conflicts. - -
- -## Switch from Woodpecker (MinIO) to external Pulsar/Kafka - -Follow these steps to switch the MQ type from Woodpecker back to external Pulsar or external Kafka. - -### Step 1: Verify the Milvus instance is running - -Before switching, ensure that your Milvus cluster instance is running properly. - -### Step 2: Configure the target MQ connection configuration - -Before triggering the switch, you need to ensure the target MQ service (Pulsar or Kafka) is available and its access configuration is rendered into the Milvus configuration. - -
- -The exact steps in this section depend on whether you are using an internal (bundled) or external MQ service. - -
- -If you are using the bundled Pulsar or Kafka deployed by Helm, update your Helm release to enable the target MQ service and disable Woodpecker. -The `streaming.enabled=true` flag is required to enable the Streaming Node, which is a prerequisite for the Switch MQ feature. For example, to switch to Pulsar: - -First, create or edit a `values.yaml` file: - -```yaml -extraConfigFiles: - user.yaml: |+ - pulsar: - address: - port: -``` - -```shell -helm upgrade -i my-release milvus/milvus \ - --set pulsarv3.enabled=true \ - --set woodpecker.enabled=false \ - --set streaming.enabled=true \ - -f values.yaml -``` - -After the upgrade, wait for all pods to be ready. Then, verify that the target MQ access configuration has been rendered into the Milvus configuration. - -### Step 3: Execute the MQ switch - -
- -Ensure that the target Pulsar/Kafka does not contain Milvus topics from a previous configuration. If this is your first time switching to Pulsar, you can skip this note. Otherwise, clean up the residual Milvus topics with the same names in Pulsar/Kafka first. - -
- -Run the following command to trigger the switch to Pulsar (replace `pulsar` with `kafka` if switching to Kafka): - -```shell -curl -X POST http://:/management/wal/alter \ - -H "Content-Type: application/json" \ - -d '{"target_wal_name": "pulsar"}' -``` - -
- -Replace `` and `` with the actual address of your MixCoord service. You can typically use `kubectl port-forward` to expose the MixCoord management interface: - -```shell -kubectl port-forward --address 0.0.0.0 service/my-release-milvus-mixcoord 29091:9091 -``` - -In this case, `mixcoord_addr` is the IP of the machine running the command, and `mixcoord_port` is the forwarded port `29091`. - -
- -### Step 4: Verify the switch is complete - -The switch process completes automatically. Monitor the Milvus logs for the following key messages to confirm the switch has finished: - -```shell -kubectl logs my-release-milvus-mixcoord-68d9889549-vdj8n | grep "successfully updated mq.type configuration in etcd" -``` - -If the switch is successful, the output should look similar to the following: - -``` -[INFO] [coordinator/wal_callbacks.go:90] ["successfully updated mq.type configuration in etcd"] -[targetWALName=pulsar] [config=null] [broadcastID=464632819924869973] [configKey=mq.type] [mqTypeValue=pulsar] -``` - -### Step 5: [Optional] Clean up Woodpecker data on MinIO/S3 and etcd - -Log in to MinIO/S3 and navigate to the corresponding bucket. Delete the Woodpecker data at the following path: - -- Format: `/wp//...` -- Example: typically `files/wp/...` - -Use an etcd tool to delete the Woodpecker metadata. The key-value prefix is as follows: - -- Prefix format: `/...` -- Example: typically `woodpecker/...` - -You can run the following command to view and then clean up the metadata: - -```shell -etcdctl get woodpecker --prefix -``` - -
- -If you plan to switch back to Woodpecker in the future, you should clean up these data files first to avoid conflicts. - -
- -## Supported MQ switch scenarios for Helm-installed Milvus Cluster - -| Source MQ | Target MQ | Status | Notes | -|-----------|-----------|--------|-------| -| Builtin Pulsar | Woodpecker (MinIO) | **Supported** | | -| Builtin Pulsar | External Kafka | **Supported** | | -| Builtin Pulsar | Builtin Kafka | **Not supported** | Helm does not support installing two builtin MQ systems simultaneously. | -| External Pulsar | Woodpecker (MinIO) | **Supported** | | -| External Pulsar | External Kafka | **Supported** | | -| Builtin Kafka | Woodpecker (MinIO) | **Supported** | | -| Builtin Kafka | External Pulsar | **Supported** | | -| Builtin Kafka | Builtin Pulsar | **Not supported** | Helm does not support installing two builtin MQ systems simultaneously. | -| External Kafka | Woodpecker (MinIO) | **Supported** | | -| External Kafka | External Pulsar | **Supported** | | -| Builtin/external Pulsar/Kafka | Woodpecker (local) | **Supported but not recommended** | All pods must access the same local disk (e.g., NFS mount), and additional configuration is required. | -| Woodpecker local | Woodpecker MinIO | **Not supported** | Switching between Woodpecker storage modes is not yet supported. May be supported in the future, but not recommended because Woodpecker local mode should not be used in cluster deployments. | - -
- -Avoid switching MQ types back and forth repeatedly. If you do need to switch, make sure to clean up the related data before each switch. Residual data may cause unexpected behavior. - -
diff --git a/site/en/adminGuide/switch_milvus_cluster_mq_type-operator.md b/site/en/adminGuide/switch_milvus_cluster_mq_type-operator.md deleted file mode 100644 index 3d2940326..000000000 --- a/site/en/adminGuide/switch_milvus_cluster_mq_type-operator.md +++ /dev/null @@ -1,175 +0,0 @@ ---- -id: switch_milvus_cluster_mq_type-operator.md -summary: Learn how to switch the message queue type for a Milvus cluster. -title: Switch MQ Type for Milvus Cluster ---- - -# Switch MQ Type for Milvus Cluster - -This topic describes how to switch the message queue (MQ) type for an existing Milvus cluster deployment. Milvus supports online MQ switching between Pulsar, Kafka, and Woodpecker without downtime. - -
- -This feature is pending release and is subject to change. Please reach out to Milvus support if you want to try it out or have any questions. - -
- -## Prerequisites - -- A running Milvus cluster instance installed via [Milvus Operator](install_cluster-milvusoperator.md). -- The Milvus instance has been upgraded to the latest version that supports this Switch MQ feature (v2.6.14 or later). - -## Switch from builtin/external Pulsar/Kafka to Woodpecker (MinIO) - -Follow these steps to switch the MQ type from Pulsar or Kafka to Woodpecker with MinIO storage. - -### Step 1: Verify the Milvus instance is running - -Before switching, ensure that your Milvus cluster instance is running properly. You can verify this by creating a test collection, inserting data, and running a query. - -### Step 2: Execute the MQ switch - -
- -If you are switching to Woodpecker for the first time, you can skip this note. Otherwise, when switching repeatedly, make sure to clean up the related data before switching. Residual data may cause unexpected behavior. For example, switching from Woodpecker (MinIO/local) to another MQ and then back to Woodpecker requires cleaning up Woodpecker meta and data first. - -
- -Run the following command to trigger the switch to Woodpecker: - -```shell -curl -X POST http://:/management/wal/alter \ - -H "Content-Type: application/json" \ - -d '{"target_wal_name": "woodpecker"}' -``` - -
- -Replace `` and `` with the actual address of your MixCoord service. You can typically use `kubectl port-forward` to expose the MixCoord management interface: - -```shell -kubectl port-forward --address 0.0.0.0 service/my-release-milvus-mixcoord 29091:9091 -``` - -In this case, `mixcoord_addr` is the IP of the machine running the command, and `mixcoord_port` is the forwarded port `29091`. - -However, Milvus Operator currently does not expose the MixCoord service. You need to use `kubectl exec` to enter the MixCoord pod and run the `curl` command from inside the pod. - -
- -### Step 3: Verify the switch is complete - -The switch process completes automatically. Monitor the Milvus logs for the following key messages to confirm the switch has finished: - -```shell -kubectl logs my-release-milvus-mixcoord-68d9889549-vdj8n | grep "successfully updated mq.type configuration in etcd" -``` - -If the switch is successful, the output should look similar to the following: - -``` -[INFO] [coordinator/wal_callbacks.go:90] ["successfully updated mq.type configuration in etcd"] -[targetWALName=WoodPecker] [config=null] [broadcastID=464632819924869973] [configKey=mq.type] [mqTypeValue=woodpecker] -``` - -### Step 4: Update the MQ type in the Operator ConfigMap - -Update the MQ type in the Operator ConfigMap to Woodpecker. First, create a file named `change_configmap.yaml`: - -```yaml -apiVersion: milvus.io/v1beta1 -kind: Milvus -metadata: - name: my-release - labels: - app: milvus -spec: - dependencies: - msgStreamType: woodpecker -``` - -Then apply the configuration update: - -```shell -kubectl patch -f change_configmap.yaml --patch-file change_configmap.yaml --type merge -``` - -### Step 5: [Optional] Stop Pulsar pods and clean up Pulsar data - -#### For builtin Pulsar instances - -After confirming the switch to Woodpecker is successful, you can remove the Pulsar pods and their associated PVC data. - -Run the following command to uninstall Pulsar: - -```shell -helm uninstall my-release-pulsar -``` - -Run the following commands to delete the Pulsar PVC storage: - -```shell -# Delete Pulsar bookie journal PVCs -kubectl delete pvc my-release-pulsarv3-bookie-journal-my-release-pulsarv3-bookie-0 -kubectl delete pvc my-release-pulsarv3-bookie-journal-my-release-pulsarv3-bookie-1 -kubectl delete pvc my-release-pulsarv3-bookie-journal-my-release-pulsarv3-bookie-2 - -# Delete Pulsar bookie ledger PVCs -kubectl delete pvc my-release-pulsarv3-bookie-ledgers-my-release-pulsarv3-bookie-0 -kubectl delete pvc my-release-pulsarv3-bookie-ledgers-my-release-pulsarv3-bookie-1 -kubectl delete pvc my-release-pulsarv3-bookie-ledgers-my-release-pulsarv3-bookie-2 - -# Delete Pulsar ZooKeeper PVCs -kubectl delete pvc my-release-pulsarv3-zookeeper-data-my-release-pulsarv3-zookeeper-0 -kubectl delete pvc my-release-pulsarv3-zookeeper-data-my-release-pulsarv3-zookeeper-1 -kubectl delete pvc my-release-pulsarv3-zookeeper-data-my-release-pulsarv3-zookeeper-2 -``` - -
- -If you plan to switch back to Pulsar in the future, you should clean up these data files first to avoid conflicts. Currently, due to Helm chart limitations, it is not possible to switch back to a builtin Pulsar instance. - -
- -#### For external Pulsar instances - -You can clean up the Milvus-related topics in the external Pulsar instance. The Milvus topic naming format is: - -``` --dml__ -``` - -For example: `by-dev-rootcoord-dml_10_464633776992639586v0` - -
- -If you plan to switch back to external Pulsar in the future, you should clean up these topics first to avoid conflicts. - -
- -## Switch from Woodpecker (MinIO) to external Pulsar/Kafka - -This is currently not supported. When Milvus Operator sets the MQ type to Woodpecker, it does not allow pre-configuring the Pulsar/Kafka connection address in the configuration file, which is required for the switch. - -## Supported MQ switch scenarios for Operator-installed Milvus Cluster - -| Source MQ | Target MQ | Status | Notes | -|-----------|-----------|--------|--------------------------------------------------------------------------------------------------------------------------| -| Builtin Pulsar | Woodpecker (MinIO) | **Supported** | | -| Builtin Pulsar | External Kafka | **Not supported** | Milvus Operator does not support pre-configuring the Kafka access address. | -| Builtin Pulsar | Builtin Kafka | **Not supported** | Helm does not support installing two builtin MQ systems simultaneously. | -| External Pulsar | Woodpecker (MinIO) | **Supported** | | -| External Pulsar | External Kafka | **Not supported** | Milvus Operator does not support pre-configuring the Kafka access address. | -| Builtin Kafka | Woodpecker (MinIO) | **Supported** | | -| Builtin Kafka | External Pulsar | **Not supported** | Milvus Operator does not support pre-configuring the Pulsar access address. | -| Builtin Kafka | Builtin Pulsar | **Not supported** | Helm does not support installing two builtin MQ systems simultaneously. | -| External Kafka | Woodpecker (MinIO) | **Supported** | | -| External Kafka | External Pulsar | **Not supported** | Milvus Operator does not support pre-configuring the Pulsar access address. | -| Builtin/external Pulsar/Kafka | Woodpecker (local) | **Supported but not recommended** | All pods must access the same local disk (e.g., NFS mount), and additional configuration is required. | -| Woodpecker local | Woodpecker MinIO | **Not supported** | Switching between Woodpecker storage modes is not yet supported. May be supported in the future, but not recommended because Woodpecker local mode should not be used in cluster deployments. | - -
- -Avoid switching MQ types back and forth repeatedly. If you do need to switch, make sure to clean up the related data before each switch. Residual data may cause unexpected behavior. - -
diff --git a/site/en/adminGuide/switch_milvus_standalone_mq_type.md b/site/en/adminGuide/switch_milvus_standalone_mq_type.md deleted file mode 100644 index 2e0086c42..000000000 --- a/site/en/adminGuide/switch_milvus_standalone_mq_type.md +++ /dev/null @@ -1,203 +0,0 @@ ---- -id: switch_milvus_standalone_mq_type.md -summary: Learn how to switch the message queue type for Milvus standalone. -title: Switch MQ Type for Milvus Standalone ---- - -# Switch MQ Type for Milvus Standalone - -This topic describes how to switch the message queue (MQ) type for an existing Milvus standalone deployment. Milvus supports online MQ switching without downtime. - -
- -This feature is pending release and is subject to change. Please reach out to Milvus support if you want to try it out or have any questions. - -
- -## Prerequisites - -- A running Milvus Standalone instance installed via [Docker Compose](install_standalone-docker-compose.md). -- The Milvus instance has been upgraded to the latest version that supports this Switch MQ feature (v2.6.14 or later). - -## General workflow - -The general workflow for switching the MQ type is as follows: - -1. Ensure the Milvus instance is running properly. -2. Confirm the source MQ type and the target MQ type. -3. Configure the target MQ's access settings into the Milvus configuration without changing the `mqType` value. -4. Trigger the switch by calling the WAL alter API. -5. Monitor the logs to verify the switch has completed successfully. - -
- -Before switching, ensure that the target MQ does not contain topics with the same names as those used by the current Milvus instance. -This is especially important if the target MQ service has been previously used by another Milvus instance, as conflicting topic names can lead to unexpected behavior. - -
- -## Docker Compose Standalone: Switch from RocksMQ to Woodpecker (Local/MinIO) - -This procedure applies to **Milvus Standalone Docker Compose** deployments that use RocksMQ by default. - -### Step 1: Verify the Milvus instance is running - -Ensure your Milvus Standalone docker compose instance is running properly. You can verify this by creating a test collection, inserting data, and running a query. - -### Step 2: Configure Woodpecker with local storage - -Update the Milvus configuration to add Woodpecker settings **without** changing the `mqType` value. Run `docker exec -it milvus-standalone bash` to enter the container, then edit the `/milvus/configs/user.yaml` file with the following key configuration: - -```yaml -woodpecker: - storage: - type: minio # minio or local -``` - -Then restart the Milvus instance to apply the configuration: - -```shell -docker-compose restart -``` - -### Step 3: Execute the MQ switch - -
- -If you are switching to Woodpecker for the first time, you can skip this note. Otherwise, when switching repeatedly, make sure to clean up the related data before switching. Residual data may cause unexpected behavior. For example, switching from Woodpecker (MinIO/local) to RocksMQ and then back to Woodpecker requires cleaning up Woodpecker meta and data first. - -
- -Run the following command to trigger the switch to Woodpecker: - -```shell -curl -X POST http://:/management/wal/alter \ - -H "Content-Type: application/json" \ - -d '{"target_wal_name": "woodpecker"}' -``` - -
- -This command sends a switch MQ type request to the internal MixCoord component in the standalone instance. Typically, the MixCoord port is `9091`. - -
- -### Step 4: Verify the switch is complete - -The switch process completes automatically. Monitor the Milvus logs for the following key messages to confirm the switch has finished. Run the following command to check the switch progress: - -```shell -docker logs milvus-standalone | grep "successfully updated mq.type configuration in etcd" -``` - -If the switch is successful, the output should look similar to the following: - -``` -[INFO] [coordinator/wal_callbacks.go:90] ["successfully updated mq.type configuration in etcd"] [targetWALName=WoodPecker] -[config=null] [broadcastID=464677797180736841] [configKey=mq.type] [mqTypeValue=woodpecker] -``` - -### Step 5: [Optional] Clean up RocksMQ data - -The RocksMQ data is located in the `volumes/milvus/rdb_data` and `volumes/milvus/rdb_data_meta_kv` directories defined in `docker-compose.yaml`. - -
- -If you plan to switch back to RocksMQ in the future, you should clean up these data files first to avoid conflicts. - -
- -## Docker Compose Standalone: Switch from Woodpecker (Local/MinIO) to RocksMQ - -This procedure applies to **Milvus Standalone Docker Compose** deployments that are currently using Woodpecker and want to switch to RocksMQ. - -### Step 1: Verify the Milvus instance is running - -Ensure your Milvus Standalone docker compose instance is running properly. You can verify this by creating a test collection, inserting data, and running a query. - -### Step 2: Execute the MQ switch - -
- -Ensure that the instance does not have residual RocksMQ data from a previous run. If this is your first time switching to RocksMQ, you can skip this note. Otherwise, clean up the related RocksMQ meta data and data first. - -
- -Run the following command to trigger the switch to RocksMQ: - -```shell -curl -X POST http://:/management/wal/alter \ - -H "Content-Type: application/json" \ - -d '{"target_wal_name": "rocksmq"}' -``` - -
- -This command sends a switch MQ type request to the internal MixCoord component in the standalone instance. Typically, the MixCoord port is `9091`. - -
- -### Step 3: Verify the switch is complete - -The switch process completes automatically. Monitor the Milvus logs for the following key messages to confirm the switch has finished. Run the following command to check the switch progress: - -```shell -docker logs milvus-standalone | grep "successfully updated mq.type configuration in etcd" -``` - -If the switch is successful, the output should look similar to the following: - -``` -[INFO] [coordinator/wal_callbacks.go:90] ["successfully updated mq.type configuration in etcd"] [targetWALName=rocksmq] -[config=null] [broadcastID=464677797180736841] [configKey=mq.type] [mqTypeValue=rocksmq] -``` - -### Step 4: [Optional] Clean up Woodpecker data - -#### Clean up Woodpecker metadata - -Use an etcd tool to delete the Woodpecker metadata. The key-value prefix is as follows: - -- Prefix format: `/...` -- Example: typically `woodpecker/...` - -You can run the following command to view and then clean up the metadata: - -```shell -etcdctl get woodpecker --prefix -``` - -#### Clean up Woodpecker storage data - -If Woodpecker was using **MinIO mode**, delete the log data on object storage. Log in to MinIO and navigate to the corresponding bucket. The data path is as follows: - -- Format: `/wp//...` -- Example: typically `files/wp/...` - -If Woodpecker was using **local mode**, the data is stored on the local disk at `volumes/milvus/data/wp/...`. - -
- -If you plan to switch back to Woodpecker in the future, you should clean up these data files first to avoid conflicts. - -
- -## Supported MQ switch scenarios for Milvus Standalone - -**Docker deployment (single container):** MQ switching is not supported because etcd source is not enabled and the `mq.type` config cannot be updated. - -**Docker Compose deployment:** - -| Source MQ | Target MQ | Status | Notes | -|-----------|-----------|--------|-------| -| RocksMQ | Woodpecker (MinIO/local) | **Supported** | | -| Woodpecker (MinIO/local) | RocksMQ | **Supported** | | -| Woodpecker MinIO | Woodpecker local | **Not supported** | Switching between Woodpecker storage modes requires additional metadata handling, which is not yet supported. May be supported in the future. | -| Woodpecker local | Woodpecker MinIO | **Not supported** | Same reason as above. | -| RocksMQ / Woodpecker (MinIO/local) | External Pulsar / Kafka | **Supported but not recommended** | Standalone instances should be kept as simple as possible for ease of operation and maintenance. | - -
- -Avoid switching MQ types back and forth repeatedly. If you do need to switch, make sure to clean up the related data before each switch. Residual data may cause unexpected behavior. - -
diff --git a/site/en/getstarted/run-milvus-docker/install_standalone-docker-compose.md b/site/en/getstarted/run-milvus-docker/install_standalone-docker-compose.md index f879d3e45..50450a764 100644 --- a/site/en/getstarted/run-milvus-docker/install_standalone-docker-compose.md +++ b/site/en/getstarted/run-milvus-docker/install_standalone-docker-compose.md @@ -119,7 +119,7 @@ mq: type: rocksmq ``` -To switch the message queue *after* upgrading, see [Switch MQ Type](switch_milvus_standalone_mq_type.md). +To switch the message queue *after* upgrading, see [Switch MQ Type](switch-mq-type.md). ## Optional dependencies diff --git a/site/en/menuStructure/en.json b/site/en/menuStructure/en.json index f757ce43a..33df2f637 100644 --- a/site/en/menuStructure/en.json +++ b/site/en/menuStructure/en.json @@ -1964,29 +1964,28 @@ "order": 14, "children": [ { - "label": "Milvus Cluster switch MQ Type", - "id": "switch_milvus_cluster_mq_type", + "label": "Overview", + "id": "switch-mq-type.md", "order": 0, - "children": [ - { - "label": "Helm Chart", - "id": "switch_milvus_cluster_mq_type-helm.md", - "order": 0, - "children": [] - }, - { - "label": "Milvus Operator", - "id": "switch_milvus_cluster_mq_type-operator.md", - "order": 1, - "children": [] - } - ] + "children": [] }, { - "label": "Milvus Standalone switch MQ type", - "id": "switch_milvus_standalone_mq_type.md", + "label": "RocksMQ ↔ Woodpecker", + "id": "switch-rocksmq-woodpecker.md", "order": 1, "children": [] + }, + { + "label": "Pulsar ↔ Woodpecker", + "id": "switch-pulsar-woodpecker.md", + "order": 2, + "children": [] + }, + { + "label": "Kafka ↔ Woodpecker", + "id": "switch-kafka-woodpecker.md", + "order": 3, + "children": [] } ] } From ef4b4f628f6274caebb0903626da011c8c0cb681 Mon Sep 17 00:00:00 2001 From: tinswzy Date: Thu, 11 Jun 2026 14:39:04 +0800 Subject: [PATCH 15/19] =?UTF-8?q?docs:=20cleanup=20pass=20=E2=80=94=20link?= =?UTF-8?q?=20text,=20version=20variables,=20stale=20default-MQ=20wording,?= =?UTF-8?q?=20glossary=20repoint?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Normalize '[Use Woodpecker]' link text to '[Woodpecker]', replace hardcoded v2.6.0 with the release-version variable in woodpecker.md, update the Switch-MQ link text on the Pulsar/Kafka pages, soften stale 'X is the default message storage' wording on the legacy config pages, and repoint the glossary Manage Dependencies reference to the new Data Infrastructure & Integration overview. Co-Authored-By: Claude Opus 4.8 (1M context) --- site/en/adminGuide/deploy_pulsar.md | 2 +- site/en/adminGuide/message_storage_operator.md | 6 +++--- site/en/adminGuide/mq_kafka.md | 2 +- site/en/adminGuide/mq_pulsar.md | 2 +- site/en/adminGuide/woodpecker.md | 6 +++--- .../run-milvus-docker/install_standalone-binary.md | 2 +- .../run-milvus-docker/install_standalone-docker.md | 2 +- .../run-milvus-docker/install_standalone-windows.md | 2 +- site/en/getstarted/run-milvus-k8s/install_cluster-helm.md | 2 +- site/en/reference/glossary.md | 2 +- 10 files changed, 14 insertions(+), 14 deletions(-) diff --git a/site/en/adminGuide/deploy_pulsar.md b/site/en/adminGuide/deploy_pulsar.md index c6d14d61c..6a30529cf 100644 --- a/site/en/adminGuide/deploy_pulsar.md +++ b/site/en/adminGuide/deploy_pulsar.md @@ -7,7 +7,7 @@ summary: Learn how to configure message storage with Docker Compose or Helm. # Configure Message Storage with Docker Compose or Helm -Milvus uses Pulsar or Kafka for managing logs of recent changes, outputting stream logs, and providing log subscriptions. Pulsar is the default message storage system. This topic introduces how to configure message storage with Docker Compose or Helm. +Milvus uses a message queue for managing logs of recent changes, outputting stream logs, and providing log subscriptions. Starting from Milvus 2.6.x, Woodpecker is the default message queue; this topic introduces how to configure Pulsar or Kafka with Docker Compose or Helm. You can configure Pulsar with [Docker Compose](https://docs.docker.com/get-started/overview/) or on K8s and configure Kafka on K8s. diff --git a/site/en/adminGuide/message_storage_operator.md b/site/en/adminGuide/message_storage_operator.md index 701ad5ebd..9a4b99dfd 100644 --- a/site/en/adminGuide/message_storage_operator.md +++ b/site/en/adminGuide/message_storage_operator.md @@ -38,7 +38,7 @@ There are also other limitations for specifying the message storage: - **Upgrade limitations**: {{fragments/mq_upgrade_limitation.md}} ## Configure RocksMQ -RocksMQ is the default message storage in Milvus standalone. +RocksMQ was the default message storage in Milvus standalone up to 2.5.x (superseded by Woodpecker from 2.6.x).
@@ -85,7 +85,7 @@ spec: ## Configure Woodpecker -Woodpecker is a cloud-native Write-Ahead Log (WAL) designed for object storage. It offers high throughput, low operational overhead, and seamless scalability. For more details, see [Use Woodpecker](woodpecker.md). +Woodpecker is a cloud-native Write-Ahead Log (WAL) designed for object storage. It offers high throughput, low operational overhead, and seamless scalability. For more details, see [Woodpecker](woodpecker.md). ## Configure Pulsar @@ -185,7 +185,7 @@ kubectl apply -f milvuscluster.yaml ## Configure Kafka -Pulsar is the default message storage in a Milvus cluster. If you want to use Kafka, add the optional field `msgStreamType` to configure Kafka. +Pulsar was the default message storage in a Milvus cluster up to 2.5.x (superseded by Woodpecker from 2.6.x). If you want to use Kafka, add the optional field `msgStreamType` to configure Kafka. `kafka` supports `external` and `inCluster`. diff --git a/site/en/adminGuide/mq_kafka.md b/site/en/adminGuide/mq_kafka.md index 58eaa99ed..59751f0ff 100644 --- a/site/en/adminGuide/mq_kafka.md +++ b/site/en/adminGuide/mq_kafka.md @@ -119,4 +119,4 @@ kubectl delete milvus my-release ## What's next - [Woodpecker (default message queue)](woodpecker.md) -- [Switch MQ Type for Milvus Cluster](switch-kafka-woodpecker.md) +- [Switch between Kafka and Woodpecker](switch-kafka-woodpecker.md) diff --git a/site/en/adminGuide/mq_pulsar.md b/site/en/adminGuide/mq_pulsar.md index 4c3c82a52..d621e0748 100644 --- a/site/en/adminGuide/mq_pulsar.md +++ b/site/en/adminGuide/mq_pulsar.md @@ -148,4 +148,4 @@ kubectl delete milvus my-release ## What's next - [Woodpecker (default message queue)](woodpecker.md) -- [Switch MQ Type for Milvus Cluster](switch-pulsar-woodpecker.md) +- [Switch between Pulsar and Woodpecker](switch-pulsar-woodpecker.md) diff --git a/site/en/adminGuide/woodpecker.md b/site/en/adminGuide/woodpecker.md index 5eacad8f3..64867f89e 100644 --- a/site/en/adminGuide/woodpecker.md +++ b/site/en/adminGuide/woodpecker.md @@ -169,7 +169,7 @@ Then deploy with one of the following examples: ```bash helm install my-release zilliztech/milvus \ - --set image.all.tag=v2.6.0 \ + --set image.all.tag=v{{var.milvus_release_version}} \ --set pulsarv3.enabled=false \ --set woodpecker.enabled=true \ --set streaming.enabled=true \ @@ -180,7 +180,7 @@ helm install my-release zilliztech/milvus \ ```bash helm install my-release zilliztech/milvus \ - --set image.all.tag=v2.6.0 \ + --set image.all.tag=v{{var.milvus_release_version}} \ --set cluster.enabled=false \ --set pulsarv3.enabled=false \ --set standalone.messageQueue=woodpecker \ @@ -219,7 +219,7 @@ Follow [Run Milvus with Docker Compose](install_standalone-docker-compose.md). E ```bash mkdir milvus-wp-compose && cd milvus-wp-compose -wget https://github.com/milvus-io/milvus/releases/download/v2.6.0/milvus-standalone-docker-compose.yml -O docker-compose.yml +wget https://github.com/milvus-io/milvus/releases/download/v{{var.milvus_release_version}}/milvus-standalone-docker-compose.yml -O docker-compose.yml # By default, the Docker Compose standalone uses Woodpecker sudo docker compose up -d # If you need to change Woodpecker parameters further, write an override: diff --git a/site/en/getstarted/run-milvus-docker/install_standalone-binary.md b/site/en/getstarted/run-milvus-docker/install_standalone-binary.md index d9119a289..ac8fe3bb3 100644 --- a/site/en/getstarted/run-milvus-docker/install_standalone-binary.md +++ b/site/en/getstarted/run-milvus-docker/install_standalone-binary.md @@ -76,7 +76,7 @@ You can find the Milvus binary at `/usr/bin/milvus`, the systemd service file at
-By default, Milvus Standalone runs **Woodpecker** (local filesystem) as its message queue with embedded etcd, so no external messaging or metadata service is required. See [Use Woodpecker](woodpecker.md). +By default, Milvus Standalone runs **Woodpecker** (local filesystem) as its message queue with embedded etcd, so no external messaging or metadata service is required. See [Woodpecker](woodpecker.md).
diff --git a/site/en/getstarted/run-milvus-docker/install_standalone-docker.md b/site/en/getstarted/run-milvus-docker/install_standalone-docker.md index fa3034be2..cd377bc4f 100644 --- a/site/en/getstarted/run-milvus-docker/install_standalone-docker.md +++ b/site/en/getstarted/run-milvus-docker/install_standalone-docker.md @@ -33,7 +33,7 @@ $ bash standalone_embed.sh start **What's new in v{{var.milvus_release_version}}:** - **Streaming Node**: Enhanced data processing capabilities -- **Woodpecker MQ (default)**: This Docker deployment runs Woodpecker as the message queue with the **local filesystem** as its WAL backend, so no external message-queue service is required. See [Use Woodpecker](woodpecker.md). +- **Woodpecker MQ (default)**: This Docker deployment runs Woodpecker as the message queue with the **local filesystem** as its WAL backend, so no external message-queue service is required. See [Woodpecker](woodpecker.md). - **Optimized Architecture**: Consolidated components for better performance Always download the latest script to ensure you get the most recent configurations and architecture improvements. diff --git a/site/en/getstarted/run-milvus-docker/install_standalone-windows.md b/site/en/getstarted/run-milvus-docker/install_standalone-windows.md index 069ab80eb..61080780c 100644 --- a/site/en/getstarted/run-milvus-docker/install_standalone-windows.md +++ b/site/en/getstarted/run-milvus-docker/install_standalone-windows.md @@ -8,7 +8,7 @@ title: Run Milvus in Docker (Windows) # Run Milvus in Docker (Windows) -> By default, this deployment runs **Woodpecker** (local filesystem) as its message queue, so no external message-queue service is required. See [Use Woodpecker](woodpecker.md). +> By default, this deployment runs **Woodpecker** (local filesystem) as its message queue, so no external message-queue service is required. See [Woodpecker](woodpecker.md). This page demonstrates how to run Milvus on Windows using Docker Desktop for Windows.​ diff --git a/site/en/getstarted/run-milvus-k8s/install_cluster-helm.md b/site/en/getstarted/run-milvus-k8s/install_cluster-helm.md index 3e6d5afe2..178001985 100644 --- a/site/en/getstarted/run-milvus-k8s/install_cluster-helm.md +++ b/site/en/getstarted/run-milvus-k8s/install_cluster-helm.md @@ -80,7 +80,7 @@ helm install my-release zilliztech/milvus \ --set streaming.enabled=true ``` -**Note**: Standalone mode uses Woodpecker as the default message queue and enables the Streaming Node component. Standalone deployments run Woodpecker **embedded** in the Milvus pod; the dedicated Woodpecker **service** (separate pods) is used for **distributed/cluster** deployments only. For details, refer to the [Architecture Overview](architecture_overview.md) and [Use Woodpecker](woodpecker.md). +**Note**: Standalone mode uses Woodpecker as the default message queue and enables the Streaming Node component. Standalone deployments run Woodpecker **embedded** in the Milvus pod; the dedicated Woodpecker **service** (separate pods) is used for **distributed/cluster** deployments only. For details, refer to the [Architecture Overview](architecture_overview.md) and [Woodpecker](woodpecker.md).
diff --git a/site/en/reference/glossary.md b/site/en/reference/glossary.md index f8a99e9ce..5faf5a1c3 100644 --- a/site/en/reference/glossary.md +++ b/site/en/reference/glossary.md @@ -43,7 +43,7 @@ In Milvus, a collection is equivalent to a table in a relational database manage ## Dependency -A dependency is a program that another program relies on to work. Milvus' dependencies include etcd (stores meta data), MinIO or S3 (object storage), and Pulsar (manages snapshot logs). For more information, refer to [Manage Dependencies](https://milvus.io/docs/manage_dependencies.md#Manage-Dependencies). +A dependency is a program that another program relies on to work. Milvus' dependencies include etcd (stores meta data), MinIO or S3 (object storage), and a message queue such as Woodpecker (manages snapshot logs). For more information, refer to [Data Infrastructure & Integration](https://milvus.io/docs/data-infra-integration-overview.md). ## Dynamic schema From 5f260cff3c71732af6e72d9c589433e606281f69 Mon Sep 17 00:00:00 2001 From: tinswzy Date: Fri, 12 Jun 2026 14:04:16 +0800 Subject: [PATCH 16/19] =?UTF-8?q?docs:=20fix=20message-queue=20doc=20bugs?= =?UTF-8?q?=20=E2=80=94=20anchors,=20stale=20dependency=20links,=20helm=20?= =?UTF-8?q?repo,=20demo=20script?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - woodpecker.md batch-insert demo: add missing 'import time'; fix client URI port 27017 -> 19530 (proxy serves 19530; 27017 is only a local port-forward); correct the maxFlushRetries inline comment. - install_cluster-helm.md: fix broken cross-ref anchor (#3-Connect-to-Milvus) and the #Optional-dependencies anchor case; repoint Kafka/RocksMQ optional- dependency links to mq_kafka.md / mq_rocksmq.md. - install_standalone-docker(.compose).md: repoint Pulsar/Kafka/RocksMQ optional-dependency links to the new mq_*.md pages. - switch-pulsar/kafka-woodpecker.md: fix helm repo milvus/milvus -> zilliztech/milvus. - switch-rocksmq-woodpecker.md: docker-compose -> docker compose (v2). - deploy_pulsar.md: soften stale 'standalone default is RocksMQ' and fix comment. Co-Authored-By: Claude Opus 4.8 (1M context) --- site/en/adminGuide/data-infra-integration-overview.md | 4 ++-- site/en/adminGuide/deploy_pulsar.md | 6 +++--- site/en/adminGuide/switch-kafka-woodpecker.md | 2 +- site/en/adminGuide/switch-pulsar-woodpecker.md | 2 +- site/en/adminGuide/switch-rocksmq-woodpecker.md | 2 +- site/en/adminGuide/woodpecker.md | 5 +++-- .../run-milvus-docker/install_standalone-docker-compose.md | 2 +- .../run-milvus-docker/install_standalone-docker.md | 2 +- site/en/getstarted/run-milvus-k8s/install_cluster-helm.md | 6 +++--- 9 files changed, 16 insertions(+), 15 deletions(-) diff --git a/site/en/adminGuide/data-infra-integration-overview.md b/site/en/adminGuide/data-infra-integration-overview.md index 3691167ad..10fb91546 100644 --- a/site/en/adminGuide/data-infra-integration-overview.md +++ b/site/en/adminGuide/data-infra-integration-overview.md @@ -10,6 +10,6 @@ Milvus builds on open data infrastructure for its core dependencies. This chapte - **[Metadata](etcd.md)** — Milvus stores metadata (collection schemas, node status, consumption checkpoints) in etcd. - **[Object Storage](object-storage.md)** — Milvus stores index files and binary logs in MinIO, AWS S3, or other S3-compatible / cloud object storage. -- **[Message Queue](mqtype-overview.md)** — Milvus uses a write-ahead log (WAL): Woodpecker (default), Pulsar, Kafka, or RocksMQ. +- **[Message Queue](mqtype-overview.md)** — Milvus uses a write-ahead log (WAL): Woodpecker (recommended), Pulsar, Kafka, or RocksMQ. -By default, a new Milvus 2.6.x / 3.x deployment runs with **Woodpecker** as the message queue, **etcd** for metadata, and **MinIO** for object storage — no extra messaging infrastructure required. +By default, a new Milvus 3.x deployment runs with **Woodpecker** as the recommended message queue, **etcd** for metadata, and **MinIO** for object storage — no extra infrastructure required. diff --git a/site/en/adminGuide/deploy_pulsar.md b/site/en/adminGuide/deploy_pulsar.md index 6a30529cf..5016d1399 100644 --- a/site/en/adminGuide/deploy_pulsar.md +++ b/site/en/adminGuide/deploy_pulsar.md @@ -7,7 +7,7 @@ summary: Learn how to configure message storage with Docker Compose or Helm. # Configure Message Storage with Docker Compose or Helm -Milvus uses a message queue for managing logs of recent changes, outputting stream logs, and providing log subscriptions. Starting from Milvus 2.6.x, Woodpecker is the default message queue; this topic introduces how to configure Pulsar or Kafka with Docker Compose or Helm. +Milvus uses a message queue for managing logs of recent changes, outputting stream logs, and providing log subscriptions. Starting from Milvus 3.0.x, Woodpecker is the default recommended message queue; this topic introduces how to configure Pulsar or Kafka with Docker Compose or Helm. You can configure Pulsar with [Docker Compose](https://docs.docker.com/get-started/overview/) or on K8s and configure Kafka on K8s. @@ -104,7 +104,7 @@ extraConfigFiles: maxIntervalForLocalStorage: 10ms # Maximum interval between two sync operations local storage backend, default is 10 milliseconds. maxBytes: 256M # Maximum size of write buffer in bytes. maxEntries: 10000 # Maximum entries number of write buffer. - maxFlushRetries: 5 # Maximum size of write buffer in bytes. + maxFlushRetries: 5 # Maximum number of flush retries. retryInterval: 1000ms # Maximum interval between two retries. default is 1000 milliseconds. maxFlushSize: 2M # Maximum size of a fragment in bytes to flush. maxFlushThreads: 32 # Maximum number of threads to flush data @@ -156,7 +156,7 @@ helm install milvus/milvus -f values.yaml ## Configure RocksMQ with Helm -Milvus standalone uses RocksMQ as the default message storage. For detailed steps on how to configure Milvus with Helm, refer to [Configure Milvus with Helm Charts](configure-helm.md). For details on RocksMQ-related configuration items, refer to [RocksMQ-related configurations](configure_rocksmq.md). +RocksMQ was the default message storage in Milvus standalone up to 2.5.x; from 2.6.x the default is Woodpecker. For detailed steps on how to configure Milvus with Helm, refer to [Configure Milvus with Helm Charts](configure-helm.md). For details on RocksMQ-related configuration items, refer to [RocksMQ-related configurations](configure_rocksmq.md). - If you start Milvus with RocksMQ and want to change its settings, you can run `helm upgrade -f ` with the changed settings in the following YAML file. diff --git a/site/en/adminGuide/switch-kafka-woodpecker.md b/site/en/adminGuide/switch-kafka-woodpecker.md index 485c49cb7..ca4daabf0 100644 --- a/site/en/adminGuide/switch-kafka-woodpecker.md +++ b/site/en/adminGuide/switch-kafka-woodpecker.md @@ -100,7 +100,7 @@ extraConfigFiles: ``` ```shell -helm upgrade -i my-release milvus/milvus \ +helm upgrade -i my-release zilliztech/milvus \ --set kafka.enabled=true \ --set woodpecker.enabled=false \ --set streaming.enabled=true \ diff --git a/site/en/adminGuide/switch-pulsar-woodpecker.md b/site/en/adminGuide/switch-pulsar-woodpecker.md index 8c6493614..bf3f1cd48 100644 --- a/site/en/adminGuide/switch-pulsar-woodpecker.md +++ b/site/en/adminGuide/switch-pulsar-woodpecker.md @@ -112,7 +112,7 @@ extraConfigFiles: ``` ```shell -helm upgrade -i my-release milvus/milvus \ +helm upgrade -i my-release zilliztech/milvus \ --set pulsarv3.enabled=true \ --set woodpecker.enabled=false \ --set streaming.enabled=true \ diff --git a/site/en/adminGuide/switch-rocksmq-woodpecker.md b/site/en/adminGuide/switch-rocksmq-woodpecker.md index cbb2a8022..b8d88cf1f 100644 --- a/site/en/adminGuide/switch-rocksmq-woodpecker.md +++ b/site/en/adminGuide/switch-rocksmq-woodpecker.md @@ -33,7 +33,7 @@ woodpecker: Restart the Milvus instance to apply the configuration: ```shell -docker-compose restart +docker compose restart ``` ### Step 3: Execute the MQ switch diff --git a/site/en/adminGuide/woodpecker.md b/site/en/adminGuide/woodpecker.md index 64867f89e..90ffa83a1 100644 --- a/site/en/adminGuide/woodpecker.md +++ b/site/en/adminGuide/woodpecker.md @@ -52,7 +52,7 @@ woodpecker: maxIntervalForLocalStorage: 10ms # Maximum interval between two sync operations local storage backend, default is 10 milliseconds. maxBytes: 256M # Maximum size of write buffer in bytes. maxEntries: 10000 # Maximum entries number of write buffer. - maxFlushRetries: 5 # Maximum size of write buffer in bytes. + maxFlushRetries: 5 # Maximum number of flush retries. retryInterval: 1000ms # Maximum interval between two retries. default is 1000 milliseconds. maxFlushSize: 2M # Maximum size of a fragment in bytes to flush. maxFlushThreads: 32 # Maximum number of threads to flush data @@ -286,10 +286,11 @@ Batch Insert Demo ```python from pymilvus import MilvusClient import random +import time # 1. Set up a Milvus client client = MilvusClient( - uri="http://:27017", + uri="http://:19530", ) # 2. Create a collection diff --git a/site/en/getstarted/run-milvus-docker/install_standalone-docker-compose.md b/site/en/getstarted/run-milvus-docker/install_standalone-docker-compose.md index 50450a764..1c2098b01 100644 --- a/site/en/getstarted/run-milvus-docker/install_standalone-docker-compose.md +++ b/site/en/getstarted/run-milvus-docker/install_standalone-docker-compose.md @@ -125,7 +125,7 @@ To switch the message queue *after* upgrading, see [Switch MQ Type](switch-mq-ty This deployment runs **Woodpecker** (local-filesystem WAL) for messaging, **etcd** for metadata, and **MinIO** for object storage. To use a different message queue or connect external object storage / metadata, see: -- Message queue: [Woodpecker](woodpecker.md) (default) · [Pulsar](deploy_pulsar.md) · [Kafka](deploy_pulsar.md#Configure-Kafka-with-Helm) · [RocksMQ](deploy_pulsar.md#Configure-RocksMQ-with-Helm) +- Message queue: [Woodpecker](woodpecker.md) (default) · [Pulsar](mq_pulsar.md) · [Kafka](mq_kafka.md) · [RocksMQ](mq_rocksmq.md) - Object storage: [MinIO](deploy_s3.md) (default) · [AWS S3](deploy_s3.md) · [Azure Blob](abs.md) · [GCP Cloud Storage](gcs.md) · [Aliyun OSS](deploy_s3.md) · [Tencent COS](deploy_s3.md) · [Huawei OBS](deploy_s3.md) · [S3-compatible](deploy_s3.md) - Metadata: [etcd](deploy_etcd.md) diff --git a/site/en/getstarted/run-milvus-docker/install_standalone-docker.md b/site/en/getstarted/run-milvus-docker/install_standalone-docker.md index cd377bc4f..65b14e7a5 100644 --- a/site/en/getstarted/run-milvus-docker/install_standalone-docker.md +++ b/site/en/getstarted/run-milvus-docker/install_standalone-docker.md @@ -110,7 +110,7 @@ $ bash standalone_embed.sh delete By default this deployment runs **Woodpecker** (local-filesystem WAL) as the message queue and an **embedded etcd** for metadata — nothing else to install. To use a different message queue or connect external object storage / metadata, see: -- Message queue: [Woodpecker](woodpecker.md) (default) · [Pulsar](deploy_pulsar.md) · [Kafka](deploy_pulsar.md#Configure-Kafka-with-Helm) · [RocksMQ](deploy_pulsar.md#Configure-RocksMQ-with-Helm) +- Message queue: [Woodpecker](woodpecker.md) (default) · [Pulsar](mq_pulsar.md) · [Kafka](mq_kafka.md) · [RocksMQ](mq_rocksmq.md) - Object storage: [MinIO](deploy_s3.md) (default) · [AWS S3](deploy_s3.md) · [Azure Blob](abs.md) · [GCP Cloud Storage](gcs.md) · [Aliyun OSS](deploy_s3.md) · [Tencent COS](deploy_s3.md) · [Huawei OBS](deploy_s3.md) · [S3-compatible](deploy_s3.md) - Metadata: [etcd](deploy_etcd.md) diff --git a/site/en/getstarted/run-milvus-k8s/install_cluster-helm.md b/site/en/getstarted/run-milvus-k8s/install_cluster-helm.md index 178001985..eac6a6502 100644 --- a/site/en/getstarted/run-milvus-k8s/install_cluster-helm.md +++ b/site/en/getstarted/run-milvus-k8s/install_cluster-helm.md @@ -117,7 +117,7 @@ For complete architecture details, refer to the [Architecture Overview](architec -**Alternative message queues:** To deploy with Pulsar, Kafka, or RocksMQ instead of Woodpecker, see [Optional dependencies](#optional-dependencies). +**Alternative message queues:** To deploy with Pulsar, Kafka, or RocksMQ instead of Woodpecker, see [Optional dependencies](#Optional-dependencies). **Next steps:** The command above deploys Milvus with recommended configurations. For production use: @@ -311,7 +311,7 @@ $ for image in $(find . -type f -name "*.tar.gz") ; do gunzip -c $image | docker $ kubectl apply -f milvus_manifest.yaml ``` -Till now, you can follow steps [2](#2-Check-Milvus-cluster-status) and [3](#3-Forward-a-local-port-to-Milvus) of the online install to check the cluster status and forward a local port to Milvus. +Till now, you can follow steps [2](#2-Check-Milvus-cluster-status) and [3](#3-Connect-to-Milvus) of the online install to check the cluster status and forward a local port to Milvus. ## Upgrade running Milvus cluster @@ -334,7 +334,7 @@ $ helm uninstall my-release This deployment runs **Woodpecker** as the message queue, **etcd** for metadata, and **MinIO** for object storage. To use a different message queue or connect external object storage / metadata, see: -- Message queue: [Woodpecker](woodpecker.md) (default) · [Pulsar](mq_pulsar.md) · [Kafka](deploy_pulsar.md#Configure-Kafka-with-Helm) · [RocksMQ](deploy_pulsar.md#Configure-RocksMQ-with-Helm) +- Message queue: [Woodpecker](woodpecker.md) (default) · [Pulsar](mq_pulsar.md) · [Kafka](mq_kafka.md) · [RocksMQ](mq_rocksmq.md) - Object storage: [MinIO](deploy_s3.md) (default) · [AWS S3](deploy_s3.md) · [Azure Blob](abs.md) · [GCP Cloud Storage](gcs.md) · [Aliyun OSS](deploy_s3.md) · [Tencent COS](deploy_s3.md) · [Huawei OBS](deploy_s3.md) · [S3-compatible](deploy_s3.md) - Metadata: [etcd](deploy_etcd.md) From 093cd26cff66b5e2cc4053d7761e84fb85b363a5 Mon Sep 17 00:00:00 2001 From: tinswzy Date: Fri, 12 Jun 2026 14:06:18 +0800 Subject: [PATCH 17/19] docs: revert stray edit to chapter overview, keep 'Woodpecker (default)' A review subagent inadvertently weakened 'Woodpecker (default)' to '(recommended)' and pre-empted the version-framing wording; restore the approved text. Version-framing (2.6.x vs 3.x) is handled deliberately in a follow-up. Co-Authored-By: Claude Opus 4.8 (1M context) --- site/en/adminGuide/data-infra-integration-overview.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/site/en/adminGuide/data-infra-integration-overview.md b/site/en/adminGuide/data-infra-integration-overview.md index 10fb91546..3691167ad 100644 --- a/site/en/adminGuide/data-infra-integration-overview.md +++ b/site/en/adminGuide/data-infra-integration-overview.md @@ -10,6 +10,6 @@ Milvus builds on open data infrastructure for its core dependencies. This chapte - **[Metadata](etcd.md)** — Milvus stores metadata (collection schemas, node status, consumption checkpoints) in etcd. - **[Object Storage](object-storage.md)** — Milvus stores index files and binary logs in MinIO, AWS S3, or other S3-compatible / cloud object storage. -- **[Message Queue](mqtype-overview.md)** — Milvus uses a write-ahead log (WAL): Woodpecker (recommended), Pulsar, Kafka, or RocksMQ. +- **[Message Queue](mqtype-overview.md)** — Milvus uses a write-ahead log (WAL): Woodpecker (default), Pulsar, Kafka, or RocksMQ. -By default, a new Milvus 3.x deployment runs with **Woodpecker** as the recommended message queue, **etcd** for metadata, and **MinIO** for object storage — no extra infrastructure required. +By default, a new Milvus 2.6.x / 3.x deployment runs with **Woodpecker** as the message queue, **etcd** for metadata, and **MinIO** for object storage — no extra messaging infrastructure required. From 14dddadccee680eea961b21884b9b47c2f6b245e Mon Sep 17 00:00:00 2001 From: tinswzy Date: Fri, 12 Jun 2026 14:51:23 +0800 Subject: [PATCH 18/19] docs: select MQ explicitly in per-MQ Helm commands, standardize on Milvus 3.x framing, update operator MQ priority - mq_pulsar.md / mq_kafka.md: use chart-native externalPulsar.enabled / externalKafka.enabled (+ explicit pulsarv3.enabled / woodpecker.enabled) so the install commands actually select the intended MQ instead of silently falling back to the chart-default Pulsar; soften 'cluster only' to 'primarily distributed' to match the support matrix. - Standardize the Woodpecker-default wording on 'Milvus 3.x' across the mqType chapter (kept the 2.5->2.6 upgrade notes, v2.6.14, and Pulsar v2/v3). - message_storage_operator.md: note Woodpecker as the default and add it to the MQ priority list (Operator supports embedded Woodpecker). Co-Authored-By: Claude Opus 4.8 (1M context) --- .../data-infra-integration-overview.md | 2 +- .../en/adminGuide/message_storage_operator.md | 6 ++--- site/en/adminGuide/mq_kafka.md | 23 ++++++++-------- site/en/adminGuide/mq_pulsar.md | 26 ++++++++++--------- site/en/adminGuide/mq_rocksmq.md | 6 ++--- site/en/adminGuide/mqtype-overview.md | 8 +++--- site/en/adminGuide/woodpecker.md | 4 +-- 7 files changed, 39 insertions(+), 36 deletions(-) diff --git a/site/en/adminGuide/data-infra-integration-overview.md b/site/en/adminGuide/data-infra-integration-overview.md index 3691167ad..86a2be0cc 100644 --- a/site/en/adminGuide/data-infra-integration-overview.md +++ b/site/en/adminGuide/data-infra-integration-overview.md @@ -12,4 +12,4 @@ Milvus builds on open data infrastructure for its core dependencies. This chapte - **[Object Storage](object-storage.md)** — Milvus stores index files and binary logs in MinIO, AWS S3, or other S3-compatible / cloud object storage. - **[Message Queue](mqtype-overview.md)** — Milvus uses a write-ahead log (WAL): Woodpecker (default), Pulsar, Kafka, or RocksMQ. -By default, a new Milvus 2.6.x / 3.x deployment runs with **Woodpecker** as the message queue, **etcd** for metadata, and **MinIO** for object storage — no extra messaging infrastructure required. +By default, a new Milvus 3.x deployment runs with **Woodpecker** as the message queue, **etcd** for metadata, and **MinIO** for object storage — no extra messaging infrastructure required. diff --git a/site/en/adminGuide/message_storage_operator.md b/site/en/adminGuide/message_storage_operator.md index 9a4b99dfd..c2542af55 100644 --- a/site/en/adminGuide/message_storage_operator.md +++ b/site/en/adminGuide/message_storage_operator.md @@ -7,7 +7,7 @@ summary: Learn how to configure message storage with Milvus Operator. # Configure Message Storage with Milvus Operator -Milvus uses RocksMQ, Pulsar or Kafka for managing logs of recent changes, outputting stream logs, and providing log subscriptions. This topic introduces how to configure message storage dependencies when you install Milvus with Milvus Operator. For more details, refer to [Configure Message Storage with Milvus Operator](https://github.com/zilliztech/milvus-operator/blob/main/docs/administration/manage-dependencies/message-storage.md) in the Milvus Operator repository. +In Milvus 3.x, Woodpecker is the default message queue (see [Woodpecker](woodpecker.md)). With Milvus Operator, you can also configure RocksMQ, Pulsar, or Kafka for managing logs of recent changes, outputting stream logs, and providing log subscriptions. This topic introduces how to configure message storage dependencies when you install Milvus with Milvus Operator. For more details, refer to [Configure Message Storage with Milvus Operator](https://github.com/zilliztech/milvus-operator/blob/main/docs/administration/manage-dependencies/message-storage.md) in the Milvus Operator repository. This topic assumes that you have deployed Milvus Operator. @@ -31,8 +31,8 @@ The table below shows whether RocksMQ, Pulsar, Kafka, and Woodpecker are support There are also other limitations for specifying the message storage: - Only one message storage for one Milvus instance is supported. However we still have backward compatibility with multiple message storages set for one instance. The priority is as follows: - - standalone mode: RocksMQ (default) > Pulsar > Kafka - - cluster mode: Pulsar (default) > Kafka + - standalone mode: Woodpecker (default) > RocksMQ > Pulsar > Kafka + - cluster mode: Woodpecker (default) > Pulsar > Kafka - The message storage cannot be changed while the Milvus system is running. - Only Kafka 2.x or 3.x verison is supported. - **Upgrade limitations**: {{fragments/mq_upgrade_limitation.md}} diff --git a/site/en/adminGuide/mq_kafka.md b/site/en/adminGuide/mq_kafka.md index 59751f0ff..60b090fe2 100644 --- a/site/en/adminGuide/mq_kafka.md +++ b/site/en/adminGuide/mq_kafka.md @@ -5,7 +5,7 @@ title: Kafka # Use Kafka as the Milvus Message Queue -Apache Kafka is one of the message-queue (WAL) backends Milvus supports for **distributed (cluster)** deployments. Starting from Milvus 2.6.x, [Woodpecker](woodpecker.md) is the default message queue; Kafka remains fully supported for users who prefer it. Kafka is used with Milvus Distributed (cluster) only — standalone deployments use embedded Woodpecker or [RocksMQ](mq_rocksmq.md). +Apache Kafka is one of the message-queue (WAL) backends Milvus supports. In Milvus 3.x, [Woodpecker](woodpecker.md) is the default message queue; Kafka remains fully supported for users who prefer it. Kafka is primarily used with Milvus Distributed (cluster); standalone deployments typically use embedded Woodpecker or [RocksMQ](mq_rocksmq.md). ## Version compatibility @@ -16,18 +16,19 @@ Apache Kafka is one of the message-queue (WAL) backends Milvus supports for **di ### Install and configure -To use an external Kafka service, put the Kafka connection settings in a `values.yaml` override, then install Milvus with it: +To use an external Kafka service, disable the bundled Pulsar and enable `externalKafka` in a `values.yaml` override, then install Milvus with it: ```yaml -extraConfigFiles: - user.yaml: |+ - kafka: - brokerList: - - : - saslUsername: - saslPassword: - saslMechanisms: PLAIN - securityProtocol: SASL_SSL +pulsarv3: + enabled: false +externalKafka: + enabled: true + brokerList: : + securityProtocol: SASL_SSL + sasl: + mechanisms: PLAIN + username: "" + password: "" ``` ```bash diff --git a/site/en/adminGuide/mq_pulsar.md b/site/en/adminGuide/mq_pulsar.md index d621e0748..bf8a1169a 100644 --- a/site/en/adminGuide/mq_pulsar.md +++ b/site/en/adminGuide/mq_pulsar.md @@ -5,7 +5,7 @@ title: Pulsar # Use Pulsar as the Milvus Message Queue -Apache Pulsar is one of the message-queue (WAL) backends Milvus supports for **distributed (cluster)** deployments. Starting from Milvus 2.6.x, [Woodpecker](woodpecker.md) is the default message queue; Pulsar remains fully supported for users who prefer it. Pulsar is used with Milvus Distributed (cluster) only — standalone deployments use embedded Woodpecker or [RocksMQ](mq_rocksmq.md). +Apache Pulsar is one of the message-queue (WAL) backends Milvus supports. In Milvus 3.x, [Woodpecker](woodpecker.md) is the default message queue; Pulsar remains fully supported for users who prefer it. Pulsar is primarily used with Milvus Distributed (cluster); standalone deployments typically use embedded Woodpecker or [RocksMQ](mq_rocksmq.md). ## Version compatibility @@ -25,6 +25,8 @@ To deploy a Milvus cluster that uses the bundled Pulsar (instead of Woodpecker), ```bash helm install my-release zilliztech/milvus \ --set image.all.tag=v{{var.milvus_release_version}} \ + --set pulsarv3.enabled=true \ + --set woodpecker.enabled=false \ --set streaming.enabled=true \ --set indexNode.enabled=false ``` @@ -41,22 +43,22 @@ helm install my-release zilliztech/milvus \ ### Configure -To connect Milvus to an **external** Pulsar service, put the Pulsar connection settings in a `values.yaml` override and disable the bundled Pulsar: +To connect Milvus to an **external** Pulsar service, disable the bundled Pulsar and enable `externalPulsar` in a `values.yaml` override: ```yaml -extraConfigFiles: - user.yaml: |+ - pulsar: - address: localhost # Address of Pulsar - port: 6650 # Port of Pulsar - webport: 80 # Web port of Pulsar; use 8080 if connecting directly without a proxy - maxMessageSize: 5242880 # 5 MB, maximum size of each message - tenant: public - namespace: default +pulsarv3: + enabled: false +externalPulsar: + enabled: true + host: + port: 6650 + maxMessageSize: "5242880" # 5 MB, maximum size of each message + tenant: public + namespace: default ``` ```bash -helm install my-release zilliztech/milvus --set pulsarv3.enabled=false -f values.yaml +helm install my-release zilliztech/milvus -f values.yaml ``` ### Uninstall diff --git a/site/en/adminGuide/mq_rocksmq.md b/site/en/adminGuide/mq_rocksmq.md index d6f70e0cd..e9330984a 100644 --- a/site/en/adminGuide/mq_rocksmq.md +++ b/site/en/adminGuide/mq_rocksmq.md @@ -5,19 +5,19 @@ title: RocksMQ # Use RocksMQ as the Milvus Message Queue -RocksMQ is an embedded message queue (WAL) bundled with Milvus, available for **Milvus Standalone only**. It was the default message queue for standalone deployments up to 2.5.x; from 2.6.x, Milvus Standalone uses embedded [Woodpecker](woodpecker.md) by default. +RocksMQ is an embedded message queue (WAL) bundled with Milvus, available for **Milvus Standalone only**. It was the default standalone message queue in earlier Milvus versions; in Milvus 3.x, Milvus Standalone uses embedded [Woodpecker](woodpecker.md) by default. ## Version compatibility - **Standalone only** — RocksMQ is **not** supported in Milvus Distributed (cluster). See the [message queue support matrix](mqtype-overview.md#Supported-message-queues). - RocksMQ ships with Milvus, so there is no separate version to install. -- It was the default standalone message queue up to 2.5.x, and is superseded by embedded Woodpecker from 2.6.x onward. +- It was the default standalone message queue in earlier Milvus versions, and is superseded by embedded Woodpecker in Milvus 3.x. ## Deploy Milvus Standalone with RocksMQ using Docker ### Install -Follow [Run Milvus in Docker](install_standalone-docker.md). On Milvus 2.6.x and later the standalone default is Woodpecker, so set the message-queue type to RocksMQ explicitly: +Follow [Run Milvus in Docker](install_standalone-docker.md). In Milvus 3.x the standalone default is Woodpecker, so set the message-queue type to RocksMQ explicitly: ```bash mkdir milvus-rocksmq && cd milvus-rocksmq diff --git a/site/en/adminGuide/mqtype-overview.md b/site/en/adminGuide/mqtype-overview.md index 50348b086..0c6c03e7c 100644 --- a/site/en/adminGuide/mqtype-overview.md +++ b/site/en/adminGuide/mqtype-overview.md @@ -6,13 +6,13 @@ summary: Overview of the message queue (mqType) options Milvus supports, and whi # Message Queue Overview -Milvus relies on a message queue (write-ahead log, WAL) to manage logs of recent changes, output stream logs, and provide log subscriptions. Starting from Milvus 2.6.x, **Woodpecker** is the default message queue and requires no separate messaging infrastructure. Pulsar, Kafka, and RocksMQ remain supported for specific scenarios. +Milvus relies on a message queue (write-ahead log, WAL) to manage logs of recent changes, output stream logs, and provide log subscriptions. In Milvus 3.x, **Woodpecker** is the default message queue and requires no separate messaging infrastructure. Pulsar, Kafka, and RocksMQ remain supported for specific scenarios. ## Supported message queues | Message queue | Milvus Standalone | Milvus Distributed (cluster) | Default in | Notes | | --- | :---: | :---: | --- | --- | -| [Woodpecker](woodpecker.md) | ✔️ (embedded) | ✔️ (embedded or service) | **2.6.x and later** (both modes) | Default and recommended. Cloud-native WAL on object storage; no external service required. | +| [Woodpecker](woodpecker.md) | ✔️ (embedded) | ✔️ (embedded or service) | **Milvus 3.x** (both modes) | Default and recommended. Cloud-native WAL on object storage; no external service required. | | [Pulsar](mq_pulsar.md) | ✔️ | ✔️ | ≤ 2.5.x (cluster default) | Supported, external or bundled. | | [Kafka](mq_kafka.md) | ✔️ | ✔️ | — | Supported. Only Kafka 2.x or 3.x. | | [RocksMQ](mq_rocksmq.md) | ✔️ | ✖️ | ≤ 2.5.x (standalone default) | Supported for **standalone only**. | @@ -27,6 +27,6 @@ Milvus relies on a message queue (write-ahead log, WAL) to manage logs of recent ## Choosing a message queue -- **New deployments (2.6.x / 3.x):** use **Woodpecker** (the default). Standalone runs it embedded; distributed can run it embedded or as a dedicated [service](woodpecker.md#Deployment-modes). +- **New deployments (Milvus 3.x):** use **Woodpecker** (the default). Standalone runs it embedded; for distributed (cluster), the recommended default is a dedicated [service](woodpecker.md#Deployment-modes) deployed with Helm, and embedded is also supported. - **Existing Pulsar or Kafka users:** Pulsar and Kafka remain fully supported. Keep them, or [switch to Woodpecker](switch-mq-type.md). -- **RocksMQ:** standalone only, and superseded by embedded Woodpecker from 2.6.x onward. +- **RocksMQ:** standalone only, and superseded by embedded Woodpecker in Milvus 3.x. diff --git a/site/en/adminGuide/woodpecker.md b/site/en/adminGuide/woodpecker.md index 90ffa83a1..5aede7e40 100644 --- a/site/en/adminGuide/woodpecker.md +++ b/site/en/adminGuide/woodpecker.md @@ -7,11 +7,11 @@ summary: Learn how Woodpecker works as the default message queue (WAL) in Milvus # Woodpecker -Woodpecker is the **default message queue (write-ahead log, WAL)** in Milvus starting from 2.6.x. It is a cloud‑native WAL designed for object storage, offering high throughput, low operational overhead, and seamless scalability. For architecture and benchmark details, see [Woodpecker](woodpecker_architecture.md). +Woodpecker is the **default message queue (write-ahead log, WAL)** in Milvus 3.x. It is a cloud‑native WAL designed for object storage, offering high throughput, low operational overhead, and seamless scalability. For architecture and benchmark details, see [Woodpecker](woodpecker_architecture.md). ## Overview -- Starting from Milvus 2.6, Woodpecker is the **default** WAL/message queue, providing ordered writes and recovery as the logging service. No external message-queue service (such as Pulsar or Kafka) is required. +- In Milvus 3.x, Woodpecker is the **default** WAL/message queue, providing ordered writes and recovery as the logging service. No external message-queue service (such as Pulsar or Kafka) is required. - Woodpecker can run **embedded** in the Milvus/streaming node (default), or as a **dedicated service** with its own pods (distributed/cluster only). - It supports three `storage.type` modes: object storage (`minio`, the default), local file system (`local`), and the dedicated `service`. See [Deployment modes](#Deployment-modes). From d5a5c87b8473b23768528fa5136538a2593a6838 Mon Sep 17 00:00:00 2001 From: tinswzy Date: Tue, 30 Jun 2026 17:00:46 +0800 Subject: [PATCH 19/19] docs: WP-default follow-ups for MQ install pages MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - mq_rocksmq: use start→edit→restart so the RocksMQ override survives run_embed's first-start user.yaml rewrite; note it targets a new instance - woodpecker: standalone-local example follows the Woodpecker default (no clobbered user.yaml); add the WP-service quorum note - install_cluster-helm: note the WP service needs a quorum of 3 (default 4) - install_standalone-docker-compose: correct the WP WAL backend wording to MinIO / object storage (compose standalone runs WP embedded over MinIO) Co-Authored-By: Claude Opus 4.8 (1M context) --- site/en/adminGuide/mq_rocksmq.md | 14 ++++++++--- site/en/adminGuide/woodpecker.md | 25 ++++++++----------- .../install_standalone-docker-compose.md | 2 +- .../run-milvus-k8s/install_cluster-helm.md | 2 +- 4 files changed, 24 insertions(+), 19 deletions(-) diff --git a/site/en/adminGuide/mq_rocksmq.md b/site/en/adminGuide/mq_rocksmq.md index e9330984a..d404ac44a 100644 --- a/site/en/adminGuide/mq_rocksmq.md +++ b/site/en/adminGuide/mq_rocksmq.md @@ -17,21 +17,29 @@ RocksMQ is an embedded message queue (WAL) bundled with Milvus, available for ** ### Install -Follow [Run Milvus in Docker](install_standalone-docker.md). In Milvus 3.x the standalone default is Woodpecker, so set the message-queue type to RocksMQ explicitly: +Follow [Run Milvus in Docker](install_standalone-docker.md). In Milvus 3.x the standalone default is Woodpecker, so switch the message-queue type to RocksMQ explicitly. The bootstrap script writes a fresh `user.yaml` on the **first** `start`, so set the type **after** that first start and then `restart` to apply (a `restart` preserves `user.yaml`): ```bash mkdir milvus-rocksmq && cd milvus-rocksmq curl -sfL https://raw.githubusercontent.com/milvus-io/milvus/master/scripts/standalone_embed.sh -o standalone_embed.sh -# Create user.yaml to use RocksMQ +# 1. First start — boots the container and writes a default user.yaml +bash standalone_embed.sh start + +# 2. Set the message queue to RocksMQ cat > user.yaml <<'EOF' mq: type: rocksmq EOF -bash standalone_embed.sh start +# 3. Restart to apply the change +bash standalone_embed.sh restart ``` +
+Switching mq.type this way is meant for a brand-new instance (no collections yet). To change the message queue of an instance that already holds data, follow the switch procedure instead. +
+ ### Configure To tune RocksMQ, add a `rocksmq` section to `user.yaml` and restart the service: diff --git a/site/en/adminGuide/woodpecker.md b/site/en/adminGuide/woodpecker.md index 5aede7e40..3386eb96b 100644 --- a/site/en/adminGuide/woodpecker.md +++ b/site/en/adminGuide/woodpecker.md @@ -192,26 +192,23 @@ After deployment, follow the docs to port‑forward and connect. To adjust Woodp ### Enable Woodpecker for Milvus Standalone in Docker (storage=local) -Follow [Run Milvus in Docker](install_standalone-docker.md). Example: +In Milvus 3.x, the Docker standalone deployment uses Woodpecker with the **local filesystem** as its WAL backend **by default** — no extra configuration is required. Follow [Run Milvus in Docker](install_standalone-docker.md): ```bash mkdir milvus-wp && cd milvus-wp curl -sfL https://raw.githubusercontent.com/milvus-io/milvus/master/scripts/standalone_embed.sh -o standalone_embed.sh - -# Create user.yaml to enable Woodpecker with local filesystem -cat > user.yaml <<'EOF' -mq: - type: woodpecker -woodpecker: - storage: - type: local - rootPath: /var/lib/milvus/woodpecker -EOF - bash standalone_embed.sh start ``` -To further change Woodpecker settings, update `user.yaml` and run `bash standalone_embed.sh restart`. +To tune Woodpecker, edit the generated `user.yaml` after the first start and run `bash standalone_embed.sh restart` to apply the changes (a fresh `start` regenerates `user.yaml`, so apply edits with `restart`): + +```yaml +# user.yaml +woodpecker: + logstore: + segmentSyncPolicy: + maxFlushThreads: 16 +``` ### Enable Woodpecker for Milvus Standalone with Docker Compose (storage=minio) @@ -252,7 +249,7 @@ helm install my-release zilliztech/milvus \ --set indexNode.enabled=false ``` -This deploys Woodpecker as a dedicated StatefulSet (`my-release-milvus-woodpecker`, 4 replicas by default) fronted by a headless service, gossip-clustered on ports `18080` (service), `17946` (gossip), and `9091` (metrics), with MinIO as its storage backend. The cluster then includes a separate `woodpecker` pod set: +This deploys Woodpecker as a dedicated StatefulSet (`my-release-milvus-woodpecker`, 4 replicas by default) fronted by a headless service, gossip-clustered on ports `18080` (service), `17946` (gossip), and `9091` (metrics), with MinIO as its storage backend. The service needs a quorum of **3** nodes; the default of **4** replicas keeps the quorum while tolerating a single node failure, so do not set `woodpecker.replicaCount` below 3. The cluster then includes a separate `woodpecker` pod set: ``` my-release-milvus-woodpecker-0 diff --git a/site/en/getstarted/run-milvus-docker/install_standalone-docker-compose.md b/site/en/getstarted/run-milvus-docker/install_standalone-docker-compose.md index 1c2098b01..e70e0847c 100644 --- a/site/en/getstarted/run-milvus-docker/install_standalone-docker-compose.md +++ b/site/en/getstarted/run-milvus-docker/install_standalone-docker-compose.md @@ -33,7 +33,7 @@ Creating milvus-standalone ... done
-**Default deployment (v{{var.milvus_release_version}}):** `docker compose up -d` starts three containers — `milvus-etcd` (metadata), `milvus-minio` (object storage), and `milvus-standalone`. The message queue is **Woodpecker (embedded, local-filesystem WAL backend)**, so no separate message-queue container is required. +**Default deployment (v{{var.milvus_release_version}}):** `docker compose up -d` starts three containers — `milvus-etcd` (metadata), `milvus-minio` (object storage), and `milvus-standalone`. The message queue is **Woodpecker (embedded, with MinIO / object storage as its WAL backend)**, so no separate message-queue container is required. **Message-queue default by version:** - **2.5.x** — default message queue is **RocksMQ**. diff --git a/site/en/getstarted/run-milvus-k8s/install_cluster-helm.md b/site/en/getstarted/run-milvus-k8s/install_cluster-helm.md index eac6a6502..c834e66cc 100644 --- a/site/en/getstarted/run-milvus-k8s/install_cluster-helm.md +++ b/site/en/getstarted/run-milvus-k8s/install_cluster-helm.md @@ -170,7 +170,7 @@ my-release-minio-3 1/1 Running 0 3m23s
-With `streaming.woodpecker.embedded=false`, Woodpecker runs as a **dedicated StatefulSet** (`my-release-milvus-woodpecker`, 4 replicas by default) fronted by a headless service, using MinIO as its storage backend — so the cluster has a separate `woodpecker` pod set, distinct from the streaming node. +With `streaming.woodpecker.embedded=false`, Woodpecker runs as a **dedicated StatefulSet** (`my-release-milvus-woodpecker`, 4 replicas by default — a quorum of 3 nodes plus one spare for fault tolerance; do not set `woodpecker.replicaCount` below 3) fronted by a headless service, using MinIO as its storage backend — so the cluster has a separate `woodpecker` pod set, distinct from the streaming node.