feat(source)!: introduce optional force-default-targets

[alen-z]

Description

⚠️ BREAKING CHANGE, but with mitigation strategy.

Improved default-targets behavior to allow empty CRD targets section in case default targets are set. Default targets are overridden if it's set anywhere else. Since it introduces breaking change in situations where targets are set, we offer force-default-targets flag to keep current behavior (hopefully easing migration).

Breaking change should be stated in release documentation.

Fixes #3163

Checklist

• Unit tests updated
• End user documentation updated

[k8s-ci-robot]

Welcome @alen-z!

It looks like this is your first PR to kubernetes-sigs/external-dns 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/external-dns has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @alen-z. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ivankatliarchuk]

As far as I know, that's not in our current plans, but things might have shifted. A breaking CRD change usually means a new API version, and we're not there yet.

To explore this further, could you help us understand the underlying issue by providing example manifests and any alternative solutions you've considered?

Breaking changes are generally something we try to avoid, so I'm not going to review that.

[alen-z]

I can switch the script to keep current behavior and allow new flag (Default: --prefer-source-targets=false, remove --force-default-targets=false) that needs to be enabled to get new behavior. What do you think?

With set default-targets I need to always set targets:

apiVersion: externaldns.k8s.io/v1alpha1
kind: DNSEndpoint
metadata:
  name: demo
  labels:
    demo: foo
  annotations:
    external-dns.demo.io/access: public
spec:
  endpoints:
  - dnsName: demo.example.com
    recordTTL: 300
    recordType: CNAME
    targets:
      - placeholder

I'd like to avoid placeholders while allowing to override default-targets if targets is set in CRD.

Expected CRD that should be allowed with default-targets set:

apiVersion: externaldns.k8s.io/v1alpha1
kind: DNSEndpoint
metadata:
  name: demo
  labels:
    demo: foo
  annotations:
    external-dns.demo.io/access: public
spec:
  endpoints:
  - dnsName: demo.example.com
    recordTTL: 300
    recordType: CNAME

[ivankatliarchuk]

I'm working on path to beta at the moment #5243

[ivankatliarchuk]

For this case

apiVersion: externaldns.k8s.io/v1alpha1
kind: DNSEndpoint
metadata:
  name: demo
  labels:
    demo: foo
  annotations:
    external-dns.demo.io/access: public
spec:
  endpoints:
  - dnsName: demo.example.com
    recordTTL: 300
    recordType: CNAME

Which provider you using, and what the result DNS is expected

[alen-z]

It's Cloudflare:

- args:
  - --metrics-address=:7979
  - --log-level=info
  - --log-format=json
  - --events
  - --policy=sync
  - --provider=cloudflare
  - --registry=txt
  - --interval=1m
  - --txt-owner-id=bar
  - --txt-prefix=b-
  - --annotation-filter=external-dns.demo.io/access in (public)
  - --source=crd
  - --cloudflare-proxied
  - --cloudflare-dns-records-per-page=5000
  - --default-targets=ingress-external.example.com

Result DNS with default-targets set is expected to be:

demo.example.com CNAME ingress-external.example.com

In PR, you can see if we do set targets in CRD, then I'd expect those targets to be respected.

[ivankatliarchuk]

Not sure about practicality, may need some time to re-think.

How this DNSEndpoint(s) created? I'm using helm, example relevant part

  {{- range $hostNames }}
  {{- if .name }}
  {{- $target := printf "eks-cluster-%s-ingress-%s.%s" $.Values.cluster.env $accessType $defaultDomainName }}
  {{- $subdomain := printf "%s-%s-eks" .name $accessType }}
  {{- if and (eq (include "this.allPortsAreGrpc" $) "false") (ne .protocol "gRPC") }}
  - dnsName: {{ $subdomain }}.{{ $defaultDomainName }}
    recordTTL: 60
    recordType: CNAME
    targets:
    - {{ $target }}
  {{- end }}
  {{- if and (eq $accessType "internal") (eq (include "this.http2Enabled" $) "true") }}
  - dnsName: {{ $subdomain }}.http2.{{ $defaultDomainName }}
    recordTTL: 60
    recordType: CNAME
    targets:
    - {{ $target }}
  {{- end }}
  {{- end }}
  {{- end }}

or
    - dnsName: {{ .Values.dns.host }}
      recordTTL: 60
      recordType: CNAME
      targets:
        - {{ include "common.dns.target" . }} # target or default target

I may missing some other use case

Or similar with kustomize or kyverno admission webhooks.

[alen-z]

It's simple:

1. Using Helm.
2. Running multiple ExternalDNS instances for different providers and public/private settings.
3. Each instance has default-targets flag set that points to cluster load balancer created during cluster initialization with Terraform.
4. DNSEndpoint is losing point in setting targets as it's using set default-targets.

This is PR to improve 4th step.

Another approach I'm using is different starting with 3rd step, which is setting annotations on Service of type LoadBalancer to create DNS for cluster load balancer. I configure default-targets with load balancer DNS I've set in annotation and again DNSEndpoint targets currently need to contain some arbitrary placeholder value.

Both approaches have product teams using DNSEndpoint resources, while load balancer DNS records and ExternalDNS instances are maintained by operations team. Product team just creates proper DNSEndpoint with annotation for public or private (again, bummer is it needs to contain targets placeholder).

[ivankatliarchuk]

Another approach I'm using is different starting with 3rd step, which is setting annotations on Service of type LoadBalancer to create DNS for cluster load balancer. I configure default-targets with load balancer DNS I've set in annotation and again DNSEndpoint targets currently need to contain some arbitrary placeholder value.

If I understand correctly, we have exactly same cases in environments I look after.

For the use case you shared, I'm not sure how you setting annotations, but if annotations on ALB are set with helm, you could add a template for DNSEndpoint as well and set all required values, use kustomize and any other tools. The targets are static, so it works.

If teams using Crossplane, it could manage DNSEndpoint targets as well at runtime.

Regardless of how DNSEndpoint manifests are generated (Helm, Kustomize, Admission WebHooks, ArgoCD, Jsonnet, etc.), any DNSEndpoint with the external-dns.demo.io/access annotation can have its necessary targets defined at manifest apply time.

I apologise, but I don't understand the proposed approach and don't see any clear benefits. However, other contributors or maintainers might have a different perspective and see value in this feature.

[alen-z]

Again, it's simple proposition: No need for targets in CRD if default-targets are set. Common sense. Rest is going around it, which is always possible, of course.

Additional comment on your suggestion that it's easy to add targets to annotations or CRD: What if you have 100 products and targets change? One simple central change in default-targets or redeploy 100 products owned by product teams with new template values?

[ivankatliarchuk]

We have differing opinions on this. In my view, the number of DNSEndpoints shouldn't matter – automation tools exist to handle scaling so 1, 100, 10000 should not be matter. If other maintainers or contributors believe external-dns should support this specific scenario, I don't see a fundamental conflict as consensus is not needed. However, from my perspective, this problem seems too niche and is better addressed with dedicated tools. This is just my personal opinion, and others may have different views.

[ivankatliarchuk]

Maybe CRD should support FQDN templating --fqdn-template=""?

[alen-z]

All good. I'm not attached to this, only offered a fix to make it better for our use case. I can close the PR.

[ivankatliarchuk]

@alen-z do you wanna try FQDN approach?

@mloiseleur wdyt?

[mloiseleur]

@ivankatliarchuk To me, this use case is quite straightforward and valid.

We are using the same pattern on some clusters: a single target (the LB) with as many CNAME as we need. So, setting it once in the CLI arg looks better to me than set the same value in all the CR.

It's also consistent: that's how other sources works.

Since there is no CEL validation ATM in the CRD, I'm not sure if this is really a breaking change. With current version, it will fail without target so, to me, a warning in release notes should be enough. But maybe I missed something, I need to double check that.

[alen-z]

@mloiseleur, in terms of looking at breaking change, depends on the approach (current PR or one other possible non-breaking).

Challenge is, to make current ExternalDNS work, many have put placeholders in CRD targets (to satisfy the validation) while using default-targets argument. If dummy CRD targets start to suddenly apply, this could have wide consequences.

To me, it's best to be able to upgrade to new version, clean DNSEndpoint under the flag, remove the flag. This actually requires this PR to be a bit more improved to allow empty targets in legacy mode that constantly enforces default-targets. Because this PR proposes overriding default-targets with CRD targets if set. To me, this makes sense.

If you folks align on the approach, I can prepare PR.

[mloiseleur]

@alen-z We took the time to discuss it with @szuecs @Raffo and @ivankatliarchuk .

We are aligned on this approach. You can prepare the PR.

[mloiseleur]

Feel free to re-open this one or open a new one.

[alen-z]

Hey @mloiseleur and all involved,

sounds good. Appreciate you folks spending time on this one. I'll re-open the PR.

To me, it's best to be able to upgrade to new version, clean DNSEndpoint under the flag, remove the flag. This actually requires this PR to be a bit more improved to allow empty targets in legacy mode that constantly enforces default-targets. Because this PR proposes overriding default-targets with CRD targets if set. To me, this makes sense.

I'll follow through with this approach and ask for review. ⚾ is in my court now. Will keep you posted.

[ivankatliarchuk]

Let us know when ready. What else is required, share results of a smoke tests with flag enabled/disabled, my understanding this are the maniests #5316 (comment). This should help to speed up a review.

[alekc]

Hijacking a bit this thread. Currently (unless I am mistaken) endpoints generated from annotations on ingress object behave in a completely opposite way where anything you set on the ingress is overriden if defaultTarget is set which is counter intuitive.

Is there a way to have all of this uniformed? As things are right now, its not possible to have an override for an individual ingress.

[ivankatliarchuk]

Hijacking a bit this thread. Currently (unless I am mistaken) endpoints generated from annotations on ingress object behave in a completely opposite way where anything you set on the ingress is overriden if defaultTarget is set which is counter intuitive.

Is there a way to have all of this uniformed? As things are right now, its not possible to have an override for an individual ingress.

This fix is realed to CRD, not an ingress. CRDs do not support annotations.

The best way is to create an issue and provide kubernetes manifests alongside with current and desired behaviour.

[ivankatliarchuk]

Hi @alen-z we are planning to do a release soonish. Do you have a capacity to rebase and address this comment #5316 (comment)

[alen-z]

Let's see what I can do today, you can then plan around it.

[alen-z]

Hey @alekc, I think your comment is valid. I can look into it if decision makers decide to expand current scope. This should flow to new multisource logic, which would see hasSourceTargets as false and correctly apply the default targets.

Code example...

diff --git a/source/ingress.go b/source/ingress.go
index d3b8f8d0..ac7314da 100644
--- a/source/ingress.go
+++ b/source/ingress.go
@@ -291,51 +291,47 @@ func endpointsFromIngress(ing *networkv1.Ingress, ignoreHostnameAnnotation bool,
 
    providerSpecific, setIdentifier := annotations.ProviderSpecificAnnotations(ing.Annotations)
 
-   // Gather endpoints defined on hosts sections of the ingress
-   var definedHostsEndpoints []*endpoint.Endpoint
-   // Skip endpoints if we do not want entries from Rules section
+   var definedHosts, annotationHosts []string
+
+   // Gather hostnames from rules section
    if !ignoreIngressRulesSpec {
        for _, rule := range ing.Spec.Rules {
-           if rule.Host == "" {
-               continue
+           if rule.Host != "" {
+               definedHosts = append(definedHosts, rule.Host)
            }
-           definedHostsEndpoints = append(definedHostsEndpoints, endpointsForHostname(rule.Host, targets, ttl, providerSpecific, setIdentifier, resource)...)
        }
    }
 
-   // Skip endpoints if we do not want entries from tls spec section
+   // Gather hostnames from TLS section
    if !ignoreIngressTLSSpec {
        for _, tls := range ing.Spec.TLS {
            for _, host := range tls.Hosts {
-               if host == "" {
-                   continue
+               if host != "" {
+                   definedHosts = append(definedHosts, host)
                }
-               definedHostsEndpoints = append(definedHostsEndpoints, endpointsForHostname(host, targets, ttl, providerSpecific, setIdentifier, resource)...)
            }
        }
    }
 
-   // Gather endpoints defined on annotations in the ingress
-   var annotationEndpoints []*endpoint.Endpoint
+   // Gather hostnames from annotation
    if !ignoreHostnameAnnotation {
-       for _, hostname := range annotations.HostnamesFromAnnotations(ing.Annotations) {
-           annotationEndpoints = append(annotationEndpoints, endpointsForHostname(hostname, targets, ttl, providerSpecific, setIdentifier, resource)...)
-       }
+       annotationHosts = annotations.HostnamesFromAnnotations(ing.Annotations)
    }
 
-   // Determine which hostnames to consider in our final list
+   // Determine which hostnames to consider
+   var hostnames []string
    hostnameSourceAnnotation, hostnameSourceAnnotationExists := ing.Annotations[ingressHostnameSourceKey]
    if !hostnameSourceAnnotationExists {
-       return append(definedHostsEndpoints, annotationEndpoints...)
+       hostnames = append(definedHosts, annotationHosts...)
+   } else if strings.ToLower(hostnameSourceAnnotation) == IngressHostnameSourceDefinedHostsOnlyValue {
+       hostnames = definedHosts
+   } else if strings.ToLower(hostnameSourceAnnotation) == IngressHostnameSourceAnnotationOnlyValue {
+       hostnames = annotationHosts
    }
 
-   // Include endpoints according to the hostname source annotation in our final list
    var endpoints []*endpoint.Endpoint
-   if strings.ToLower(hostnameSourceAnnotation) == IngressHostnameSourceDefinedHostsOnlyValue {
-       endpoints = append(endpoints, definedHostsEndpoints...)
-   }
-   if strings.ToLower(hostnameSourceAnnotation) == IngressHostnameSourceAnnotationOnlyValue {
-       endpoints = append(endpoints, annotationEndpoints...)
+   for _, hostname := range hostnames {
+       endpoints = append(endpoints, endpointsForHostname(hostname, targets, ttl, providerSpecific, setIdentifier, resource)...)
    }
    return endpoints
 }

---

Smoke test

Note: It'd be nice to provide smoke test infrastructure since it's required in PRs. Let me know if you would like me to create one.

I've pushed staging image to my infrastructure (make release.staging), configured and deployed new ExternalDNS to work with Cloudflare, tested 2 manifests with different configurations.

---
# Working as expected, w/o force-default-targets, w/ default-targets = creates records, adds CRD targets since not empty, new behavior
# Working as expected, w/ force-default-targets, w/ default-targets = creates records, sets only default targets, legacy behavior
apiVersion: externaldns.k8s.io/v1alpha1
kind: DNSEndpoint
metadata:
  name: targets
  namespace: default
  labels:
    demo: foo
  annotations:
    external-dns-smoke.relaymonkey.io/access: public
spec:
  endpoints:
  - dnsName: smoke-t.relaymonkey.com
    recordTTL: 300
    recordType: CNAME
    targets:
      - placeholder
---
# Working as expected, w/o force-default-targets, w/ default-targets = creates records, sets only default targets, new behavior (not failing on empty targets), debug log
# Working as expected, w/ force-default-targets, w/ default-targets = creates records, sets only default targets, new behavior (not failing on empty targets), debug log
apiVersion: externaldns.k8s.io/v1alpha1
kind: DNSEndpoint
metadata:
  name: no-targets
  namespace: default
  labels:
    demo: foo
  annotations:
    external-dns-smoke.relaymonkey.io/access: public
spec:
  endpoints:
  - dnsName: smoke-nt.relaymonkey.com
    recordTTL: 300
    recordType: CNAME

Debug log for 2nd manifest:

{"level":"debug","msg":"Endpoint no-targets with DNSName smoke-nt.relaymonkey.com has an empty list of targets, allowing it to pass through for default-targets processing","time":"2025-06-17T19:55:05Z"}

1st manifest w/o force-default-targets, w/ default-targets (default target was set if flag was set):

All seems as expected ✅

[ivankatliarchuk]

/lgtm

[ivankatliarchuk]

To keep things manageble, the preffered approach is to scope PR to a single problem. If there is follow-up pull request with the fix, we could discuss it there.

cc @mloiseleur for final review

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mloiseleur

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [mloiseleur]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment