Skip to content

Commit 8ea453c

Browse files
Milvus-doc-botMilvus-doc-bot
Milvus-doc-bot
authored and
Milvus-doc-bot
committed
Release new docs to preview
1 parent 7bcf357 commit 8ea453c

6 files changed

Lines changed: 433 additions & 0 deletions

File tree

‎v3.0.x/site/en/adminGuide/data-infra-integration-overview.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
---
2+
id: data-infra-integration-overview.md
3+
title: Data Infrastructure & Integration
4+
summary: Overview of the third-party infrastructure Milvus integrates with — metadata, object storage, and message queues.
5+
---
6+
7+
# Data Infrastructure & Integration
8+
9+
Milvus builds on open data infrastructure for its core dependencies. This chapter covers the components you can plug in and configure:
10+
11+
- **[Metadata](etcd.md)** — Milvus stores metadata (collection schemas, node status, consumption checkpoints) in etcd.
12+
- **[Object Storage](object-storage.md)** — Milvus stores index files and binary logs in MinIO, AWS S3, or other S3-compatible / cloud object storage.
13+
- **[Message Queue](mqtype-overview.md)** — Milvus uses a write-ahead log (WAL): Woodpecker (default), Pulsar, Kafka, or RocksMQ.
14+
15+
By default, a new Milvus 3.x deployment runs with **Woodpecker** as the message queue, **etcd** for metadata, and **MinIO** for object storage — no extra messaging infrastructure required.

‎v3.0.x/site/en/adminGuide/mq_kafka.md‎

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
---
2+
id: mq_kafka.md
3+
title: Kafka
4+
---
5+
6+
# Use Kafka as the Milvus Message Queue
7+
8+
Apache Kafka is one of the message-queue (WAL) backends Milvus supports. In Milvus 3.x, [Woodpecker](woodpecker.md) is the default message queue; Kafka remains fully supported for users who prefer it. Kafka is primarily used with Milvus Distributed (cluster); standalone deployments typically use embedded Woodpecker or [RocksMQ](mq_rocksmq.md).
9+
10+
## Version compatibility
11+
12+
- Milvus supports **Kafka 2.x and 3.x** only.
13+
- Kafka is configured for Milvus Distributed (cluster) via Helm or Milvus Operator.
14+
15+
## Deploy a Milvus cluster with Kafka using Helm
16+
17+
### Install and configure
18+
19+
To use an external Kafka service, disable the bundled Pulsar and enable `externalKafka` in a `values.yaml` override, then install Milvus with it:
20+
21+
```yaml
22+
pulsarv3:
23+
enabled: false
24+
externalKafka:
25+
enabled: true
26+
brokerList: <your_kafka_address>:<your_kafka_port>
27+
securityProtocol: SASL_SSL
28+
sasl:
29+
mechanisms: PLAIN
30+
username: ""
31+
password: ""
32+
```
33+
34+
```bash
35+
helm install my-release zilliztech/milvus -f values.yaml
36+
```
37+
38+
For SASL/SSL authentication details, see [Connect to Kafka with SASL/SSL](connect_kafka_ssl.md).
39+
40+
### Uninstall
41+
42+
```bash
43+
helm uninstall my-release
44+
```
45+
46+
## Deploy a Milvus cluster with Kafka using Milvus Operator
47+
48+
With Milvus Operator, set `spec.dependencies.msgStreamType: "kafka"` and configure Kafka under `spec.dependencies.kafka` (cluster only). `kafka` supports `external` and `inCluster`.
49+
50+
### External Kafka
51+
52+
```yaml
53+
apiVersion: milvus.io/v1alpha1
54+
kind: Milvus
55+
metadata:
56+
name: my-release
57+
labels:
58+
app: milvus
59+
spec:
60+
config:
61+
kafka:
62+
# securityProtocol supports: PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL
63+
securityProtocol: PLAINTEXT
64+
# saslMechanisms supports: PLAIN, SCRAM-SHA-256, SCRAM-SHA-512
65+
saslMechanisms: PLAIN
66+
saslUsername: ""
67+
saslPassword: ""
68+
dependencies:
69+
msgStreamType: "kafka"
70+
kafka:
71+
external: true
72+
brokerList:
73+
- "kafkaBrokerAddr1:9092"
74+
- "kafkaBrokerAddr2:9092"
75+
```
76+
77+
<div class="alert note">
78+
79+
SASL configurations are supported in Milvus Operator v0.8.5 or later.
80+
81+
</div>
82+
83+
### Internal (in-cluster) Kafka
84+
85+
```yaml
86+
apiVersion: milvus.io/v1alpha1
87+
kind: Milvus
88+
metadata:
89+
name: my-release
90+
labels:
91+
app: milvus
92+
spec:
93+
dependencies:
94+
msgStreamType: "kafka"
95+
kafka:
96+
inCluster:
97+
values: {} # see https://artifacthub.io/packages/helm/bitnami/kafka
98+
components: {}
99+
config: {}
100+
```
101+
102+
Apply the configuration (assuming the file is `milvuscluster.yaml`):
103+
104+
```bash
105+
kubectl apply -f milvuscluster.yaml
106+
```
107+
108+
### Uninstall
109+
110+
```bash
111+
kubectl delete milvus my-release
112+
```
113+
114+
## Notes
115+
116+
- **Upgrading from 2.5.x to 2.6.x:** **Message Queue limitations**: When upgrading to Milvus v3.0-beta, you must maintain your current message queue choice. Switching between different message queue systems during the upgrade is not supported. Support for changing message queue systems will be available in future versions.
117+
If you run Kafka and want to keep it, do not change the message queue during the upgrade.
118+
- Only **Kafka 2.x and 3.x** are supported.
119+
- For SASL/SSL connectivity, see [Connect to Kafka with SASL/SSL](connect_kafka_ssl.md).
120+
121+
## What's next
122+
123+
- [Woodpecker (default message queue)](woodpecker.md)

‎v3.0.x/site/en/adminGuide/mq_pulsar.md‎

Lines changed: 153 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
---
2+
id: mq_pulsar.md
3+
title: Pulsar
4+
---
5+
6+
# Use Pulsar as the Milvus Message Queue
7+
8+
Apache Pulsar is one of the message-queue (WAL) backends Milvus supports. In Milvus 3.x, [Woodpecker](woodpecker.md) is the default message queue; Pulsar remains fully supported for users who prefer it. Pulsar is primarily used with Milvus Distributed (cluster); standalone deployments typically use embedded Woodpecker or [RocksMQ](mq_rocksmq.md).
9+
10+
## Version compatibility
11+
12+
| Milvus version | Supported Pulsar version | Default |
13+
| --- | --- | --- |
14+
| 2.5.x and later | Pulsar v3 (recommended) or Pulsar v2 | Pulsar v3 (via Helm / Milvus Operator) |
15+
| 2.4.x and earlier | Pulsar v2 | Pulsar v2 |
16+
17+
Since Milvus 2.5, the Milvus Helm chart and Milvus Operator deploy **Pulsar v3** by default; Pulsar v2 remains compatible. See [Upgrade Pulsar from v2 to v3](upgrade-pulsar-v3.md) and [Continue Using Pulsar v2](use-pulsar-v2.md).
18+
19+
## Deploy a Milvus cluster with Pulsar using Helm
20+
21+
### Install
22+
23+
To deploy a Milvus cluster that uses the bundled Pulsar (instead of Woodpecker), install the Helm chart with the Streaming Node enabled:
24+
25+
```bash
26+
helm install my-release zilliztech/milvus \
27+
--set image.all.tag=v3.0-beta \
28+
--set pulsarv3.enabled=true \
29+
--set woodpecker.enabled=false \
30+
--set streaming.enabled=true \
31+
--set indexNode.enabled=false
32+
```
33+
34+
On Kubernetes v1.25 and later, if you hit PodDisruptionBudget API issues from the bundled Pulsar sub-chart, disable the Pulsar PDB policies:
35+
36+
```bash
37+
helm install my-release zilliztech/milvus \
38+
--set pulsar.bookkeeper.pdb.usePolicy=false \
39+
--set pulsar.broker.pdb.usePolicy=false \
40+
--set pulsar.proxy.pdb.usePolicy=false \
41+
--set pulsar.zookeeper.pdb.usePolicy=false
42+
```
43+
44+
### Configure
45+
46+
To connect Milvus to an **external** Pulsar service, disable the bundled Pulsar and enable `externalPulsar` in a `values.yaml` override:
47+
48+
```yaml
49+
pulsarv3:
50+
enabled: false
51+
externalPulsar:
52+
enabled: true
53+
host: <your_pulsar_host>
54+
port: 6650
55+
maxMessageSize: "5242880" # 5 MB, maximum size of each message
56+
tenant: public
57+
namespace: default
58+
```
59+
60+
```bash
61+
helm install my-release zilliztech/milvus -f values.yaml
62+
```
63+
64+
### Uninstall
65+
66+
```bash
67+
helm uninstall my-release
68+
```
69+
70+
If you used the bundled Pulsar and want to remove its persisted data, delete the Pulsar PVCs (named `my-release-pulsarv3-*`):
71+
72+
```bash
73+
kubectl get pvc | grep my-release-pulsarv3
74+
kubectl delete pvc <pulsar-pvc-name> ...
75+
```
76+
77+
## Deploy a Milvus cluster with Pulsar using Milvus Operator
78+
79+
With Milvus Operator, configure Pulsar under `spec.dependencies.pulsar` (supported for Milvus cluster only). `pulsar` supports `external` and `inCluster`.
80+
81+
### External Pulsar
82+
83+
```yaml
84+
apiVersion: milvus.io/v1alpha1
85+
kind: Milvus
86+
metadata:
87+
name: my-release
88+
labels:
89+
app: milvus
90+
spec:
91+
dependencies:
92+
pulsar:
93+
external: true
94+
endpoints:
95+
- 192.168.1.1:6650
96+
components: {}
97+
config: {}
98+
```
99+
100+
### Internal (in-cluster) Pulsar
101+
102+
```yaml
103+
apiVersion: milvus.io/v1alpha1
104+
kind: Milvus
105+
metadata:
106+
name: my-release
107+
labels:
108+
app: milvus
109+
spec:
110+
dependencies:
111+
pulsar:
112+
inCluster:
113+
values:
114+
components:
115+
autorecovery: false
116+
zookeeper:
117+
replicaCount: 1
118+
bookkeeper:
119+
replicaCount: 1
120+
broker:
121+
replicaCount: 1
122+
configData:
123+
autoSkipNonRecoverableData: "true"
124+
managedLedgerDefaultEnsembleSize: "1"
125+
managedLedgerDefaultWriteQuorum: "1"
126+
managedLedgerDefaultAckQuorum: "1"
127+
proxy:
128+
replicaCount: 1
129+
components: {}
130+
config: {}
131+
```
132+
133+
Apply the configuration (assuming the file is `milvuscluster.yaml`):
134+
135+
```bash
136+
kubectl apply -f milvuscluster.yaml
137+
```
138+
139+
### Uninstall
140+
141+
```bash
142+
kubectl delete milvus my-release
143+
```
144+
145+
## Notes
146+
147+
- **Upgrading from 2.5.x to 2.6.x:** **Message Queue limitations**: When upgrading to Milvus v3.0-beta, you must maintain your current message queue choice. Switching between different message queue systems during the upgrade is not supported. Support for changing message queue systems will be available in future versions.
148+
If you run Pulsar and want to keep it, do not change the message queue during the upgrade.
149+
- **Pulsar v2 → v3:** see [Upgrade Pulsar from v2 to v3](upgrade-pulsar-v3.md); to stay on v2, see [Continue Using Pulsar v2](use-pulsar-v2.md).
150+
151+
## What's next
152+
153+
- [Woodpecker (default message queue)](woodpecker.md)

‎v3.0.x/site/en/adminGuide/mq_rocksmq.md‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
---
2+
id: mq_rocksmq.md
3+
title: RocksMQ
4+
---
5+
6+
# Use RocksMQ as the Milvus Message Queue
7+
8+
RocksMQ is an embedded message queue (WAL) bundled with Milvus, available for **Milvus Standalone only**. It was the default standalone message queue in earlier Milvus versions; in Milvus 3.x, Milvus Standalone uses embedded [Woodpecker](woodpecker.md) by default.
9+
10+
## Version compatibility
11+
12+
- **Standalone only** — RocksMQ is **not** supported in Milvus Distributed (cluster). See the [message queue support matrix](mqtype-overview.md#Supported-message-queues).
13+
- RocksMQ ships with Milvus, so there is no separate version to install.
14+
- It was the default standalone message queue in earlier Milvus versions, and is superseded by embedded Woodpecker in Milvus 3.x.
15+
16+
## Deploy Milvus Standalone with RocksMQ using Docker
17+
18+
### Install
19+
20+
Follow [Run Milvus in Docker](install_standalone-docker.md). In Milvus 3.x the standalone default is Woodpecker, so switch the message-queue type to RocksMQ explicitly. The bootstrap script writes a fresh `user.yaml` on the **first** `start`, so set the type **after** that first start and then `restart` to apply (a `restart` preserves `user.yaml`):
21+
22+
```bash
23+
mkdir milvus-rocksmq && cd milvus-rocksmq
24+
curl -sfL https://raw.githubusercontent.com/milvus-io/milvus/master/scripts/standalone_embed.sh -o standalone_embed.sh
25+
26+
# 1. First start — boots the container and writes a default user.yaml
27+
bash standalone_embed.sh start
28+
29+
# 2. Set the message queue to RocksMQ
30+
cat > user.yaml <<'EOF'
31+
mq:
32+
type: rocksmq
33+
EOF
34+
35+
# 3. Restart to apply the change
36+
bash standalone_embed.sh restart
37+
```
38+
39+
<div class="alert note">
40+
Switching <code>mq.type</code> this way is meant for a <b>brand-new</b> instance (no collections yet). To change the message queue of an instance that already holds data, follow the switch procedure instead.
41+
</div>
42+
43+
### Configure
44+
45+
To tune RocksMQ, add a `rocksmq` section to `user.yaml` and restart the service:
46+
47+
```yaml
48+
mq:
49+
type: rocksmq
50+
rocksmq:
51+
path: /var/lib/milvus/rdb_data # where messages are stored
52+
lrucacheratio: 0.06 # rocksdb cache memory ratio
53+
rocksmqPageSize: 67108864 # 64 MB, size of each message page
54+
retentionTimeInMinutes: 4320 # 3 days
55+
retentionSizeInMB: 8192 # 8 GB
56+
compactionInterval: 86400 # 1 day, trigger rocksdb compaction
57+
compressionTypes: [0, 0, 7, 7, 7]
58+
```
59+
60+
```bash
61+
bash standalone_embed.sh restart
62+
```
63+
64+
### Uninstall
65+
66+
```bash
67+
bash standalone_embed.sh stop
68+
bash standalone_embed.sh delete
69+
```
70+
71+
## Notes
72+
73+
- **Upgrading from 2.5.x to 2.6.x:** **Message Queue limitations**: When upgrading to Milvus v3.0-beta, you must maintain your current message queue choice. Switching between different message queue systems during the upgrade is not supported. Support for changing message queue systems will be available in future versions.
74+
Because 2.6.x changes the standalone default to Woodpecker, pin `mq.type: rocksmq` in your `user.yaml` **before** upgrading if you want to keep RocksMQ.
75+
- To change the message queue of a running instance, see Switch from RocksMQ to Woodpecker.
76+
77+
## What's next
78+
79+
- [Woodpecker (default message queue)](woodpecker.md)

0 commit comments

Comments
 (0)