Recommended storage backend for Milvus etcd on Kubernetes: Local NVMe RAID0 vs network NVMe persistent disk

[karantyagi-reltio]

Environment
Milvus version: v2.6.12
Deployment mode(standalone or cluster): cluster

We are evaluating the best storage option for Milvus etcd when running Milvus on Kubernetes across cloud providers.

Our current setup:

• Milvus etcd deployed on Kubernetes using StatefulSets
• Etcd runs on dedicated on-demand nodes

We are comparing two approaches:

Local NVMe SSD + RAID0 + local-static-volume-provisioner

• Highest disk performance and lowest latency
• PVCs are created using local PVs
• Data survives pod restart on the same node
• Data is lost if the node is deleted or replaced

Network-attached persistent NVMe disk

• Example: GCP Hyperdisk Balanced/Extreme
• Dynamically provisioned through StorageClass
• Data survives node deletion/replacement
• Slightly higher latency than Local NVMe

Question:
For Milvus etcd, which storage backend is recommended in production?

Specifically:

• Is Local NVMe RAID0 recommended for etcd if etcd runs on dedicated on-demand nodes?
• Or should etcd use durable network-attached storage to avoid data loss during node replacement?
• What are the recommended disk latency/IOPS targets for Milvus etcd?
• Are there any known issues with running Milvus etcd on local PVs backed by Local NVMe SSD?
• For multi-cloud deployments, should we standardize on durable disks for etcd and reserve Local NVMe only for cache/index/temp workloads?
• If we use Local NVMe RAID0 and two kubernetes went down together due to node failure will the milvus cluster recoverable as becuase of node failure we will lose the data in Local NVMe RAID0

Additionally, if we use Local NVMe SSD with RAID0 for etcd storage and two Kubernetes nodes fail simultaneously, would the Milvus cluster still be recoverable?

Our concern is that Local NVMe SSD is node-local and ephemeral. In the event of node failure or node replacement, the underlying Local NVMe RAID0 data would be permanently lost. Since etcd stores critical Milvus metadata, we would like to understand:

• How resilient Milvus is to simultaneous etcd member/node loss in this architecture
• Whether losing multiple etcd members backed by Local NVMe SSD can lead to unrecoverable cluster metadata loss
• Whether Milvus recommends using durable persistent disks instead of Local NVMe for etcd in production environments
• Whether Local NVMe RAID0 is recommended only for QueryNode/IndexNode cache/temp workloads rather than critical metadata components.

Would appreciate clarification on the recommended production architecture and recovery expectations in such failure scenarios.

Also, if we migrate existing PVC data to a new PVC backed by Local NVMe SSD, is there any risk of data inconsistency or data loss during or after the migration?

Specifically:

• If the original PVC is backed by durable storage and the new PVC is backed by Local NVMe RAID0/local PV storage
• Whether Milvus supports this migration pattern safely
• Whether any additional backup/snapshot/recovery steps are recommended before migration
• Whether there are known concerns around etcd consistency or metadata recovery after moving to node-local ephemeral NVMe storage

We would also like to understand the recommended migration and recovery strategy before moving existing Milvus workloads to Local NVMe-backed storage.