Skip to content

How to use an EtcdCluster as a CoreDNS + ExternalDNS endpoint? #302

@ferdinandosimonetti

Description

@ferdinandosimonetti

cluster.yml

Good morning, everyone.
I'm currently using an ETCD cluster deployed via Bitnami's Helm Chart to serve as a repository of ExternalDNS's generated DNS entries, being queried by external clients through a CoreDNS custom installation.

That Helm Chart allows you to specify a password for the root user (or to have it generated randomly and saved into a Secret.
Additionally, I've configured the ETCD cluster itself to avoid using TLS while communicating with clients, and I'm exposing it through an EnvoyGateway's GRPCRoute (with TLS termination at the Gateway level).

All that to say that both CoreDNS

      etcd {
        path /skydns
        endpoint https://etcd.my.domain.com:2379
        credentials root WhatAWonderfulDay
        fallthrough
      }

and ExternalDNS

apiVersion: apps/v1
kind: Deployment
metadata:
  name: externaldns-external-dns
  namespace: externaldns
spec:
  template:
    spec:
      containers:
      - name: external-dns
        env:
        - name: ETCD_URLS
          value: https://etcd.my.domain.com:2379

were super easy to configure for working with that ETCD.

Now, given that we're distancing ourselves from Bitnami, I found out your ETCD Operator and wanted to replicate the previous setup, replacing Bitnami's ETCD with yours.

Only:

  • should I want auth enabled, I must implement the whole TLS setup (internal and external)
  • already got the directive to use for CoreDNS to "speak TLS" to ETCD
  • I still have to understand how to do that with ExternalDNS
  • I can't find any way to specify root user's password
  • a simple interaction with etcdctl client got "permission denied" and these logs
[KS-Demo-admin|default] ➜  etcd-operator etcdctl --cacert=ca.pem --cert=tls.pem --key=key.pem --endpoints https://localhost:2379 put a b

Handling connection for 2379
{"level":"warn","ts":"2025-12-16T09:23:24.152548+0100","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x1400001c000/localhost:2379","attempt":0,"error":"rpc error: code = PermissionDenied desc = etcdserver: permission denied"}
Error: etcdserver: permission denied
{"level":"error","ts":"2025-12-16T08:23:24.181437Z","caller":"auth/store.go:884","msg":"cannot find a user for permission check","user-name":"store-etcd","stacktrace":"go.etcd.io/etcd/server/v3/auth.(*authStore).isOpPermitted\n\tgo.etcd.io/etcd/server/v3/auth/store.go:884\ngo.etcd.io/etcd/server/v3/auth.(*authStore).IsPutPermitted\n\tgo.etcd.io/etcd/server/v3/auth/store.go:901\ngo.etcd.io/etcd/server/v3/etcdserver/apply.(*authApplierV3).Put\n\tgo.etcd.io/etcd/server/v3/etcdserver/apply/apply_auth.go:66\ngo.etcd.io/etcd/server/v3/etcdserver/apply.(*uberApplier).dispatch\n\tgo.etcd.io/etcd/server/v3/etcdserver/apply/uber_applier.go:142\ngo.etcd.io/etcd/server/v3/etcdserver/apply.(*applierV3backend).Apply\n\tgo.etcd.io/etcd/server/v3/etcdserver/apply/apply.go:152\ngo.etcd.io/etcd/server/v3/etcdserver/apply.(*authApplierV3).Apply\n\tgo.etcd.io/etcd/server/v3/etcdserver/apply/apply_auth.go:59\ngo.etcd.io/etcd/server/v3/etcdserver/apply.(*uberApplier).Apply\n\tgo.etcd.io/etcd/server/v3/etcdserver/apply/uber_applier.go:119\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyInternalRaftRequest\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:2016\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyEntryNormal\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:1975\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).apply\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:1894\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyEntries\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:1194\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyAll\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:979\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).run.func6\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:855\ngo.etcd.io/etcd/pkg/v3/schedule.job.Do\n\tgo.etcd.io/etcd/pkg/[email protected]/schedule/schedule.go:41\ngo.etcd.io/etcd/pkg/v3/schedule.(*fifo).executeJob\n\tgo.etcd.io/etcd/pkg/[email protected]/schedule/schedule.go:206\ngo.etcd.io/etcd/pkg/v3/schedule.(*fifo).run\n\tgo.etcd.io/etcd/pkg/[email protected]/schedule/schedule.go:187"}
{"level":"warn","ts":"2025-12-16T08:23:24.181613Z","caller":"txn/util.go:46","msg":"failed to apply request","took":"147.609µs","request":"header:<ID:2875155919132451270 username:\"store-etcd\" auth_revision:4 > put:<key:\"a\" value_size:1 >","response":"","error":"auth: permission denied"}
{"level":"error","ts":"2025-12-16T08:23:24.176564Z","caller":"auth/store.go:884","msg":"cannot find a user for permission check","user-name":"store-etcd","stacktrace":"go.etcd.io/etcd/server/v3/auth.(*authStore).isOpPermitted\n\tgo.etcd.io/etcd/server/v3/auth/store.go:884\ngo.etcd.io/etcd/server/v3/auth.(*authStore).IsPutPermitted\n\tgo.etcd.io/etcd/server/v3/auth/store.go:901\ngo.etcd.io/etcd/server/v3/etcdserver/apply.(*authApplierV3).Put\n\tgo.etcd.io/etcd/server/v3/etcdserver/apply/apply_auth.go:66\ngo.etcd.io/etcd/server/v3/etcdserver/apply.(*uberApplier).dispatch\n\tgo.etcd.io/etcd/server/v3/etcdserver/apply/uber_applier.go:142\ngo.etcd.io/etcd/server/v3/etcdserver/apply.(*applierV3backend).Apply\n\tgo.etcd.io/etcd/server/v3/etcdserver/apply/apply.go:152\ngo.etcd.io/etcd/server/v3/etcdserver/apply.(*authApplierV3).Apply\n\tgo.etcd.io/etcd/server/v3/etcdserver/apply/apply_auth.go:59\ngo.etcd.io/etcd/server/v3/etcdserver/apply.(*uberApplier).Apply\n\tgo.etcd.io/etcd/server/v3/etcdserver/apply/uber_applier.go:119\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyInternalRaftRequest\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:2016\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyEntryNormal\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:1975\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).apply\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:1894\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyEntries\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:1194\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyAll\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:979\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).run.func6\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:855\ngo.etcd.io/etcd/pkg/v3/schedule.job.Do\n\tgo.etcd.io/etcd/pkg/[email protected]/schedule/schedule.go:41\ngo.etcd.io/etcd/pkg/v3/schedule.(*fifo).executeJob\n\tgo.etcd.io/etcd/pkg/[email protected]/schedule/schedule.go:206\ngo.etcd.io/etcd/pkg/v3/schedule.(*fifo).run\n\tgo.etcd.io/etcd/pkg/[email protected]/schedule/schedule.go:187"}
{"level":"warn","ts":"2025-12-16T08:23:24.176789Z","caller":"txn/util.go:46","msg":"failed to apply request","took":"195.102µs","request":"header:<ID:2875155919132451270 username:\"store-etcd\" auth_revision:4 > put:<key:\"a\" value_size:1 >","response":"","error":"auth: permission denied"}
{"level":"error","ts":"2025-12-16T08:23:24.183176Z","caller":"auth/store.go:884","msg":"cannot find a user for permission check","user-name":"store-etcd","stacktrace":"go.etcd.io/etcd/server/v3/auth.(*authStore).isOpPermitted\n\tgo.etcd.io/etcd/server/v3/auth/store.go:884\ngo.etcd.io/etcd/server/v3/auth.(*authStore).IsPutPermitted\n\tgo.etcd.io/etcd/server/v3/auth/store.go:901\ngo.etcd.io/etcd/server/v3/etcdserver/apply.(*authApplierV3).Put\n\tgo.etcd.io/etcd/server/v3/etcdserver/apply/apply_auth.go:66\ngo.etcd.io/etcd/server/v3/etcdserver/apply.(*uberApplier).dispatch\n\tgo.etcd.io/etcd/server/v3/etcdserver/apply/uber_applier.go:142\ngo.etcd.io/etcd/server/v3/etcdserver/apply.(*applierV3backend).Apply\n\tgo.etcd.io/etcd/server/v3/etcdserver/apply/apply.go:152\ngo.etcd.io/etcd/server/v3/etcdserver/apply.(*authApplierV3).Apply\n\tgo.etcd.io/etcd/server/v3/etcdserver/apply/apply_auth.go:59\ngo.etcd.io/etcd/server/v3/etcdserver/apply.(*uberApplier).Apply\n\tgo.etcd.io/etcd/server/v3/etcdserver/apply/uber_applier.go:119\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyInternalRaftRequest\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:2016\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyEntryNormal\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:1975\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).apply\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:1894\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyEntries\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:1194\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).applyAll\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:979\ngo.etcd.io/etcd/server/v3/etcdserver.(*EtcdServer).run.func6\n\tgo.etcd.io/etcd/server/v3/etcdserver/server.go:855\ngo.etcd.io/etcd/pkg/v3/schedule.job.Do\n\tgo.etcd.io/etcd/pkg/[email protected]/schedule/schedule.go:41\ngo.etcd.io/etcd/pkg/v3/schedule.(*fifo).executeJob\n\tgo.etcd.io/etcd/pkg/[email protected]/schedule/schedule.go:206\ngo.etcd.io/etcd/pkg/v3/schedule.(*fifo).run\n\tgo.etcd.io/etcd/pkg/[email protected]/schedule/schedule.go:187"}
{"level":"warn","ts":"2025-12-16T08:23:24.183333Z","caller":"txn/util.go:46","msg":"failed to apply request","took":"136.302µs","request":"header:<ID:2875155919132451270 username:\"store-etcd\" auth_revision:4 > put:<key:\"a\" value_size:1 >","response":"","error":"auth: permission denied"}

Help? I am probably missing something really obvious, but I can't find out where.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions