ETCD Scaleup failing in TLS mode enabled

[0xkc0de]

Name and Version

3.5.16-r1

What steps will reproduce the bug?

`
STS_NAME=etc-etcdtlson-869m27
NAMESPACE=dev
HOST=etc-etcdtlson-869m27-0
HOST_FQDN=etc-etcdtlson-869m27-0.etc-etcdtlson-869m27.dev.svc.cluster.local
MASTER_NODE=etc-etcdtlson-869m27-0
MASTER_FQDN=etc-etcdtlson-869m27-0.etc-etcdtlson-869m27.dev.svc.cluster.local
PORT=2379

ETCD_SCRIPTS=/opt/bitnami/scripts/etcd
ETCD_COMMON_SCRIPTS=/opt/bitnami/scripts
ETCD_HOME=/opt/bitnami
ETCD_DATA=/bitnami/etcd/data

APP_VERSION=3.5.16
BITNAMI_APP_NAME=etcd
ETCDCTL_API=3

ALLOW_NONE_AUTHENTICATION=yes
ETCD_CLUSTER_DOMAIN=
ETCD_START_FROM_SNAPSHOT=no
ETCD_DISASTER_RECOVERY=no
ETCD_ON_K8S=yes
ETCD_INIT_SNAPSHOT_FILENAME=
ETCD_LOG_LEVEL=info

ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://etc-etcdtlson-869m27-0.etc-etcdtlson-869m27.dev.svc.cluster.local:2380
ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379
ETCD_ADVERTISE_CLIENT_URLS=https://etc-etcdtlson-869m27-0.etc-etcdtlson-869m27.dev.svc.cluster.local:2379
ETCD_CERT_FILE=/srv/test/certs/tls.crt
ETCD_KEY_FILE=/srv/test/certs/tls.key
ETCD_TRUSTED_CA_FILE=/srv/test/certs/ca.crt
ETCD_CLIENT_CERT_AUTH=true
ETCD_PEER_CERT_FILE=/srv/test/certs/tls.crt
ETCD_PEER_KEY_FILE=/srv/test/certs/tls.key
ETCD_PEER_TRUSTED_CA_FILE=/srv/test/certs/ca.crt
ETCD_PEER_CLIENT_CERT_AUTH=true
ETCD_AUTO_TLS=false
ETCD_PEER_AUTO_TLS=false
PROTOCOL=https
TLSPARAMS=--cacert=/srv/test/certs/ca.crt --cert=/srv/test/certs/tls.crt --key=/srv/test/certs/tls.key

ETCD_INITIAL_CLUSTER_STATE=existing
ETCD_INITIAL_CLUSTER=etc-etcdtlson-869m27-0.etc-etcdtlson-869m27.dev.svc.cluster.local=https://etc-etcdtlson-869m27-0.etc-etcdtlson-869m27.dev.svc.cluster.local:2380,etc-etcdtlson-869m27-1.etc-etcdtlson-869m27.dev.svc.cluster.local=https://etc-etcdtlson-869m27-1.etc-etcdtlson-869m27.dev.svc.cluster.local:2380,etc-etcdtlson-869m27-2.etc-etcdtlson-869m27.dev.svc.cluster.local=https://etc-etcdtlson-869m27-2.etc-etcdtlson-869m27.dev.svc.cluster.local:2380
ETCD_NAME=etc-etcdtlson-869m27-0.etc-etcdtlson-869m27.dev.svc.cluster.local
ETCD_INITIAL_CLUSTER_TOKEN=etc-etcdtlson-869m27`

What is the expected behavior?

I have configured etcd tls enabled with above configuration. I was trying to scale up from 3 to 5. From the binary script it is checking the endpoint health from the peer using ENV ETCD_INITIAL_CLUSTER.

but ENV has port 2380, which is a peer-peer communication port while etcdctl is a client and it fails to identify the endpoints. Expectation is it should use the 2379 port using etcdctl

As per etcd official docs, port 2380 won't work with etcdctl client.

What do you see instead?

etcdctl --user root --password xxx --endpoints https://etc-etcdtlson-869m27-0.etc-etcdtlson-869m27.dev.svc.cluster.local:2380 --cacert=/srv/test/certs/ca.crt --cert=/srv/test/certs/tls.crt --key=/srv/test/certs/tls.key endpoint health

{"level":"warn","ts":"2025-06-25T06:18:05.250570Z","logger":"client","caller":"v3@v3.5.16/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000222000/etc-etcdtlson-869m27-0.etc-etcdtlson-869m27.dev.svc.cluster.local:2380","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = "error reading server preface: EOF""}
https://etc-etcdtlson-869m27-0.etc-etcdtlson-869m27.dev.svc.cluster.local:2380 is unhealthy: failed to commit proposal: context deadline exceeded
Error: unhealthy cluster

Additional information

Can you please review this scaleup scenario with tls enabled is a working usecase? or anything i missed in configuration need help on this

[LoveEachDay]

@karthiknathan06 port 2380 is used for etcd member communication, you should use 2379 for etcdctl command.
To scale etcd cluster from 3 members to 5, all you need to is change the replicas of etcd statefulset.

[0xkc0de]

Thanks @LoveEachDay for your response

But when i scale up with TLS enabled, below code check the health of peers using etcdctl cli. and the port was 2380 and faile. same scenario works for non TLS mode

https://github.com/milvus-io/bitnami-docker-etcd/blob/3.5.0-new/3.5/ubuntu22.04/rootfs/opt/bitnami/scripts/libetcd.sh#L223C1-L224C1

[xiaofan-luan]

could you change the ETCD_INIT_CLUSTER to ETCD_ADVERTISE_CLIENT_URLS and try?

[0xkc0de]

@xiaofan-luan for your suggestion.
ETCD_ADVERTISE_CLIENT_URLS doesn't contain all the peer details, it only have current peer endpoint details, to identify themselves to cluster.

In Scaleup scenario using ETCD_ADVERTISE_CLIENT_URLS here will ideally in fail

[0xkc0de]

etcdctl_get_endpoints() { local scheme="http" local port="2379" if [[ "${ETCD_CLIENT_CERT_AUTH}" == "true" || -n "${ETCD_CERT_FILE}" ]]; then scheme="https" fi echo "$ETCD_INITIAL_CLUSTER" \ | tr "," "\n" \ | awk -F"=" -v s="$scheme" -v p="$port" '{ sub(/^https?:\/\//, "", $2); split($2, parts, ":"); print s "://" parts[1] ":" p }' \ | paste -sd "," - }

Patched the etcdctl_get_endpoints() function to dynamically generate etcd client endpoints using the ETCD_INITIAL_CLUSTER environment variable. This addresses issues encountered during etcd scale-up operations, where static endpoint definitions were insufficient.

Can you please check if I’m missing something in the configuration, or if this is an actual bug? @LoveEachDay @xiaofan-luan

Thanks

[xiaofan-luan]

@karthiknathan06
I believe this a bug. can you create a pr for your fix?

[0xkc0de]

Hi @LoveEachDay @xiaofan-luan
I've raised a PR to address the issue. When you have a moment, I'd really appreciate a review. Please feel free to tag anyone else who might be appropriate to help get visibility on this PR.
Thanks in advance!

[haorenfsa]

Thank you @karthiknathan06. Released as 3.5.21-r2

[haorenfsa]

• update default version in https://github.com/zilliztech/milvus-helm

[0xkc0de]

Thank you @haorenfsa . I see that the Milvus Helm chart currently uses etcd:3.5.18-r1 as the default version (values.yaml link).

Just wanted to confirm, as 3.5.21-r2 is released, is there any action expected from my side to update the default etcd version in the Helm chart?
Happy to raise a PR if needed. Please advise.

[haorenfsa]

@karthiknathan06 Yes, please. You can open a PR for https://github.com/zilliztech/milvus-helm, updating the etcd default image version together with the milvus helm chart version