[Bug]: ExternalDNS webhook provider needs an image repository

[alexohneander]

📝 Description

Hey! I'm trying to enable ExternalDNS on an existing setup, but ArgoCD won't deploy it.
Not sure how to cleanly reproduce this yet, so I included my configs below.

So far, I have found that in my generated values.yaml template, the variable external-dns.webhookImage is not set at all. The question here is what should be set in it.

🧪 Steps to Reproduce

kubara Config:

version: v1alpha4
clusters:
  - name: dev-null
    stage: dev
    type: hub
    dnsName: ske.PRIVATE_DOMAIN
    ssoOrg: none
    ssoTeam: none
    ingressClassName: traefik
    terraform:
      provider: none
      projectId: none
      kubernetesType: edge
      kubernetesVersion: "1.34"
      dns:
        name: ske.PRIVATE_DOMAIN
        email: moin@PRIVATE_DOMAIN
    argocd:
      selfManaged: enabled
      repo:
        https:
          configs:
            url: http://PRIVATE_DOMAIN/alex/kubara.git
            targetRevision: main
          components:
            url: http://PRIVATE_DOMAIN/alex/kubara.git
            targetRevision: main
    catalogs:
      - oci://ghcr.io/kubara-io/catalogs/general:1.1.0
    services:
      cert-manager:
        status: disabled
      external-dns:
        status: enabled
      external-secrets:
        status: enabled
      homer-dashboard:
        status: enabled
      kube-prometheus-stack:
        status: enabled
      kyverno:
        status: disabled
      kyverno-policies:
        status: disabled
      kyverno-policy-reporter:
        status: disabled
      loki:
        status: enabled
      longhorn:
        status: disabled
      metallb:
        status: disabled
      metrics-server:
        status: enabled
      oauth2-proxy:
        status: disabled
      reloader:
        status: disabled
      traefik:
        status: enabled
      velero:
        status: disabled

💻 Environment

• Kubara version: v0.13.0
• Kubernetes version: v1.36.0 (Talos v1.13.8)
• Helm version: v4.2.2
• Platform: Talos
• OS/Arch: 6.18.42-talos (amd64)

📎 Additional Context / Logs

ArgoCD Logs:

Failed to load target state: failed to generate manifest for source 2 of 2: rpc error: code = Unknown desc = Manifest generation error (cached): failed to execute helm template command: failed running helm: `helm template . --name-template external-dns --namespace external-dns --kube-version 1.36.0 --values <path to cached source>/platform-components/helm/external-dns/values.yaml --values <path to cached source>/platform-configs/dev-null/helm/external-dns/values.generated.yaml <api versions removed> --include-crds` failed exit status 1: Error: execution error at (external-dns/charts/external-dns/templates/deployment.yaml:193:20): ERROR: webhook provider needs an image repository and a tag Use --debug flag to render out invalid YAML

helm template logs:

Error: execution error at (external-dns/charts/external-dns/templates/deployment.yaml:193:20): ERROR: webhook provider needs an image repository and a tag

deployment.yaml:

        - name: webhook
          image: {{ include "external-dns.webhookImage" . }}
          imagePullPolicy: {{ .image.pullPolicy }}

generated values.yaml:

---
# creates an ExternalSecret that looks for the specified remote-keys (remote secret names) to inject into webhook pod
externalSecrets: {}
namespace:
  annotations: {}
  labels: {}
external-dns:
  sources: [ingress]
  policy: sync
  serviceMonitor:
    enabled: false
  resources:
    requests:
      cpu: 10m
      memory: 100Mi
    limits:
      memory: 100Mi
  podSecurityContext:
    runAsNonRoot: true
    fsGroup: 65534
    seccompProfile:
      type: RuntimeDefault
  securityContext:
    privileged: false
    allowPrivilegeEscalation: false
    readOnlyRootFilesystem: true
    runAsNonRoot: true
    runAsUser: 65532
    runAsGroup: 65532
    capabilities:
      drop: [ALL]
  provider:
    name: webhook
    webhook:
      resources:
        requests:
          cpu: 10m
          memory: 100Mi
        limits:
          memory: 100Mi
      securityContext:
        privileged: false
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        runAsNonRoot: true
        runAsUser: 65534
        runAsGroup: 65534
        capabilities:
          drop: [ALL]
      livenessProbe:
        httpGet:
          port: http-webhook
      readinessProbe:
        httpGet:
          port: http-webhook

✅ Checklist

• I have searched existing issues
• I am using the latest version of Kubara
• The issue is reproducible
• This is not a duplicate

[alexohneander]

This issue could be resolved if we add the following to the upstream chart (catalog):

  provider:
    name: webhook
    webhook:
      image:
        repository: 'registry.k8s.io/external-dns/external-dns'
        tag: v0.21.0

Whether you want full URLs in the values is, of course, a different question.

I don't fully understand why the error is happening in the first place. This section actually exists in the upstream chart, and it isn't being overwritten in the catalog chart unless I'm missing something.

[Matthiator]

Thanks for reporting this — this is a bug in the current kubara flow.

The provider-specific configuration step itself is necessary because kubara cannot infer which DNS provider an existing cluster uses. However, kubara should not generate an incomplete ExternalDNS configuration and let it fail later during Argo CD reconciliation. We also need to make this requirement clearer in our documentation.

ExternalDNS watches Kubernetes resources such as Ingresses and Services and creates or updates the corresponding DNS records through the configured DNS provider.

To do that, ExternalDNS needs:

• a concrete DNS provider implementation;
• authorization to access and modify the DNS zone, for example credentials or a workload identity;
• when using the webhook provider, the provider-specific webhook image.

The regular registry.k8s.io/external-dns/external-dns image is the main ExternalDNS controller image. It is not a generic provider webhook and therefore should not be used as the webhook image.

Webhook implementations are provider-specific. For example, STACKIT and T Cloud Public require different webhook images and authentication configurations. ExternalDNS maintains an overview of available providers and community webhook implementations in its provider documentation and project documentation.

As a quick workaround, you can disable ExternalDNS if you do not currently need automatic DNS management:

services:
  external-dns:
    status: disabled

Then regenerate and push the generated changes.

If you want to use ExternalDNS, add an additional provider-specific values file:

# platform-configs/<cluster-name>/helm/external-dns/values-provider.yaml
#
# In your case:
# platform-configs/dev-null/helm/external-dns/values-provider.yaml

external-dns:
  provider:
    name: webhook
    webhook:
      image:
        repository: <your-provider-webhook-image>
        tag: <compatible-version>

Depending on the provider, this overlay may also need provider-specific environment variables, Secret references, or volume mounts. Please do not edit values.generated.yaml, as kubara will overwrite it during the next generation.

If your DNS provider is supported directly by ExternalDNS, a webhook image may not be necessary. In that case, configure the native provider instead:

external-dns:
  provider:
    name: <provider-name>

Please let us know which DNS provider you use, and we can help with the exact configuration.

We will improve both the documentation and the generation/validation flow so that kubara detects this incomplete configuration earlier and returns an actionable error instead of producing an Argo CD Helm rendering failure.

[alexohneander]

Hi @Matthiator,
Your explanation helps me a lot and also explains why I have this error.
Thank you for the quick response.