-
Notifications
You must be signed in to change notification settings - Fork 19
Description
Good morning, everyone.
I'm currently using an ETCD cluster deployed via Bitnami's Helm Chart to serve as a repository of ExternalDNS's generated DNS entries, being queried by external clients through a CoreDNS custom installation.
That Helm Chart allows you to specify a password for the root user (or to have it generated randomly and saved into a Secret.
Additionally, I've configured the ETCD cluster itself to avoid using TLS while communicating with clients, and I'm exposing it through an EnvoyGateway's GRPCRoute (with TLS termination at the Gateway level).
All that to say that both CoreDNS
etcd {
path /skydns
endpoint https://etcd.my.domain.com:2379
credentials root WhatAWonderfulDay
fallthrough
}and ExternalDNS
apiVersion: apps/v1
kind: Deployment
metadata:
name: externaldns-external-dns
namespace: externaldns
spec:
template:
spec:
containers:
- name: external-dns
env:
- name: ETCD_URLS
value: https://etcd.my.domain.com:2379were super easy to configure for working with that ETCD.
Now, given that we're distancing ourselves from Bitnami, I found out your ETCD Operator and wanted to replicate the previous setup, replacing Bitnami's ETCD with yours.
Only:
- should I want auth enabled, I must implement the whole TLS setup (internal and external)
- already got the directive to use for CoreDNS to "speak TLS" to ETCD
- I still have to understand how to do that with ExternalDNS
- I can't find any way to specify root user's password
- a simple interaction with etcdctl client got "permission denied" and these logs
[KS-Demo-admin|default] ➜ etcd-operator etcdctl --cacert=ca.pem --cert=tls.pem --key=key.pem --endpoints https://localhost:2379 put a b
Handling connection for 2379
{"level":"warn","ts":"2025-12-16T09:23:24.152548+0100","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x1400001c000/localhost:2379","attempt":0,"error":"rpc error: code = PermissionDenied desc = etcdserver: permission denied"}
Error: etcdserver: permission denied
{"level":"error","ts":"2025-12-16T08:23:24.181437Z","caller":"auth/store.go:884","msg":"cannot find a user for permission check","user-name":"store-etcd","stacktrace":"go.etcd.io/etcd/server/v3/auth.(*authStore).isOpPermitted\n\tgo.etcd.io/etcd/server/v3/auth/store.go:884\ngo.etcd.io/etcd/server/v3/auth.(*authStore).IsPutPermitted\n\tgo.etcd.io/etcd/server/v3/auth/store.go:901\ngo.etcd.io/etcd/server/v3/etcdserver/apply.(*authApplierV3).Put\n\tgo.etcd.io/etcd/server/v3/etcdserver/apply/apply_auth.go:66\ngo.etcd.io/etcd/server/v3/etcdserver/apply.(*uberApplier).dispatch\n\tgo.etcd.io/etcd/server/v3/etcdserver/apply/uber_applier.go:142\ngo.etcd.io/etcd/server/v3/etcdserver/apply.(*applierV3backend).Apply\n\tgo.etcd.io/etcd/server/v3/etcdserver/apply/apply.go:152\ngo.etcd.io/etcd/server/v3/etcdserver/apply.(*authApplierV3).Apply\n\tgo.etcd.io/etcd/server/v3/etcdserver/apply/apply_auth.go:59\ngo.etcd.io/etcd/server/v3/etcdserver/apply.(*uberApplier).Apply\n\tgo.etcd.io/etcd/server/v3/etcdserver/apply/uber_applier.go:119\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyInternalRaftRequest\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:2016\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyEntryNormal\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:1975\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).apply\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:1894\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyEntries\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:1194\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyAll\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:979\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).run.func6\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:855\ngo.etcd.io/etcd/pkg/v3/schedule.job.Do\n\tgo.etcd.io/etcd/pkg/[email protected]/schedule/schedule.go:41\ngo.etcd.io/etcd/pkg/v3/schedule.(*fifo).executeJob\n\tgo.etcd.io/etcd/pkg/[email protected]/schedule/schedule.go:206\ngo.etcd.io/etcd/pkg/v3/schedule.(*fifo).run\n\tgo.etcd.io/etcd/pkg/[email protected]/schedule/schedule.go:187"}
{"level":"warn","ts":"2025-12-16T08:23:24.181613Z","caller":"txn/util.go:46","msg":"failed to apply request","took":"147.609µs","request":"header:<ID:2875155919132451270 username:\"store-etcd\" auth_revision:4 > put:<key:\"a\" value_size:1 >","response":"","error":"auth: permission denied"}
{"level":"error","ts":"2025-12-16T08:23:24.176564Z","caller":"auth/store.go:884","msg":"cannot find a user for permission check","user-name":"store-etcd","stacktrace":"go.etcd.io/etcd/server/v3/auth.(*authStore).isOpPermitted\n\tgo.etcd.io/etcd/server/v3/auth/store.go:884\ngo.etcd.io/etcd/server/v3/auth.(*authStore).IsPutPermitted\n\tgo.etcd.io/etcd/server/v3/auth/store.go:901\ngo.etcd.io/etcd/server/v3/etcdserver/apply.(*authApplierV3).Put\n\tgo.etcd.io/etcd/server/v3/etcdserver/apply/apply_auth.go:66\ngo.etcd.io/etcd/server/v3/etcdserver/apply.(*uberApplier).dispatch\n\tgo.etcd.io/etcd/server/v3/etcdserver/apply/uber_applier.go:142\ngo.etcd.io/etcd/server/v3/etcdserver/apply.(*applierV3backend).Apply\n\tgo.etcd.io/etcd/server/v3/etcdserver/apply/apply.go:152\ngo.etcd.io/etcd/server/v3/etcdserver/apply.(*authApplierV3).Apply\n\tgo.etcd.io/etcd/server/v3/etcdserver/apply/apply_auth.go:59\ngo.etcd.io/etcd/server/v3/etcdserver/apply.(*uberApplier).Apply\n\tgo.etcd.io/etcd/server/v3/etcdserver/apply/uber_applier.go:119\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyInternalRaftRequest\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:2016\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyEntryNormal\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:1975\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).apply\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:1894\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyEntries\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:1194\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyAll\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:979\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).run.func6\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:855\ngo.etcd.io/etcd/pkg/v3/schedule.job.Do\n\tgo.etcd.io/etcd/pkg/[email protected]/schedule/schedule.go:41\ngo.etcd.io/etcd/pkg/v3/schedule.(*fifo).executeJob\n\tgo.etcd.io/etcd/pkg/[email protected]/schedule/schedule.go:206\ngo.etcd.io/etcd/pkg/v3/schedule.(*fifo).run\n\tgo.etcd.io/etcd/pkg/[email protected]/schedule/schedule.go:187"}
{"level":"warn","ts":"2025-12-16T08:23:24.176789Z","caller":"txn/util.go:46","msg":"failed to apply request","took":"195.102µs","request":"header:<ID:2875155919132451270 username:\"store-etcd\" auth_revision:4 > put:<key:\"a\" value_size:1 >","response":"","error":"auth: permission denied"}
{"level":"error","ts":"2025-12-16T08:23:24.183176Z","caller":"auth/store.go:884","msg":"cannot find a user for permission check","user-name":"store-etcd","stacktrace":"go.etcd.io/etcd/server/v3/auth.(*authStore).isOpPermitted\n\tgo.etcd.io/etcd/server/v3/auth/store.go:884\ngo.etcd.io/etcd/server/v3/auth.(*authStore).IsPutPermitted\n\tgo.etcd.io/etcd/server/v3/auth/store.go:901\ngo.etcd.io/etcd/server/v3/etcdserver/apply.(*authApplierV3).Put\n\tgo.etcd.io/etcd/server/v3/etcdserver/apply/apply_auth.go:66\ngo.etcd.io/etcd/server/v3/etcdserver/apply.(*uberApplier).dispatch\n\tgo.etcd.io/etcd/server/v3/etcdserver/apply/uber_applier.go:142\ngo.etcd.io/etcd/server/v3/etcdserver/apply.(*applierV3backend).Apply\n\tgo.etcd.io/etcd/server/v3/etcdserver/apply/apply.go:152\ngo.etcd.io/etcd/server/v3/etcdserver/apply.(*authApplierV3).Apply\n\tgo.etcd.io/etcd/server/v3/etcdserver/apply/apply_auth.go:59\ngo.etcd.io/etcd/server/v3/etcdserver/apply.(*uberApplier).Apply\n\tgo.etcd.io/etcd/server/v3/etcdserver/apply/uber_applier.go:119\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyInternalRaftRequest\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:2016\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyEntryNormal\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:1975\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).apply\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:1894\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyEntries\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:1194\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyAll\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:979\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).run.func6\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:855\ngo.etcd.io/etcd/pkg/v3/schedule.job.Do\n\tgo.etcd.io/etcd/pkg/[email protected]/schedule/schedule.go:41\ngo.etcd.io/etcd/pkg/v3/schedule.(*fifo).executeJob\n\tgo.etcd.io/etcd/pkg/[email protected]/schedule/schedule.go:206\ngo.etcd.io/etcd/pkg/v3/schedule.(*fifo).run\n\tgo.etcd.io/etcd/pkg/[email protected]/schedule/schedule.go:187"}
{"level":"warn","ts":"2025-12-16T08:23:24.183333Z","caller":"txn/util.go:46","msg":"failed to apply request","took":"136.302µs","request":"header:<ID:2875155919132451270 username:\"store-etcd\" auth_revision:4 > put:<key:\"a\" value_size:1 >","response":"","error":"auth: permission denied"}
Help? I am probably missing something really obvious, but I can't find out where.