BREAKING CHANGE: Improve default targets management

[alen-z]

Description

⚠️ BREAKING CHANGE, but with mitigation strategy.

Improved default-targets behavior to allow empty CRD targets section in case default targets are set. Default targets are overridden if it's set anywhere else. Since it introduces breaking change in situations where targets are set, we offer force-default-targets flag to keep current behavior (hopefully easing migration).

Breaking change should be stated in release documentation.

Fixes #3163

Checklist

• Unit tests updated
• End user documentation updated

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign mloiseleur for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

Welcome @alen-z!

It looks like this is your first PR to kubernetes-sigs/external-dns 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/external-dns has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @alen-z. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ivankatliarchuk]

As far as I know, that's not in our current plans, but things might have shifted. A breaking CRD change usually means a new API version, and we're not there yet.

To explore this further, could you help us understand the underlying issue by providing example manifests and any alternative solutions you've considered?

Breaking changes are generally something we try to avoid, so I'm not going to review that.

[alen-z]

I can switch the script to keep current behavior and allow new flag (Default: --prefer-source-targets=false, remove --force-default-targets=false) that needs to be enabled to get new behavior. What do you think?

With set default-targets I need to always set targets:

apiVersion: externaldns.k8s.io/v1alpha1
kind: DNSEndpoint
metadata:
  name: demo
  labels:
    demo: foo
  annotations:
    external-dns.demo.io/access: public
spec:
  endpoints:
  - dnsName: demo.example.com
    recordTTL: 300
    recordType: CNAME
    targets:
      - placeholder

I'd like to avoid placeholders while allowing to override default-targets if targets is set in CRD.

Expected CRD that should be allowed with default-targets set:

apiVersion: externaldns.k8s.io/v1alpha1
kind: DNSEndpoint
metadata:
  name: demo
  labels:
    demo: foo
  annotations:
    external-dns.demo.io/access: public
spec:
  endpoints:
  - dnsName: demo.example.com
    recordTTL: 300
    recordType: CNAME

[ivankatliarchuk]

I'm working on path to beta at the moment #5243

[ivankatliarchuk]

For this case

apiVersion: externaldns.k8s.io/v1alpha1
kind: DNSEndpoint
metadata:
  name: demo
  labels:
    demo: foo
  annotations:
    external-dns.demo.io/access: public
spec:
  endpoints:
  - dnsName: demo.example.com
    recordTTL: 300
    recordType: CNAME

Which provider you using, and what the result DNS is expected

[alen-z]

It's Cloudflare:

- args:
  - --metrics-address=:7979
  - --log-level=info
  - --log-format=json
  - --events
  - --policy=sync
  - --provider=cloudflare
  - --registry=txt
  - --interval=1m
  - --txt-owner-id=bar
  - --txt-prefix=b-
  - --annotation-filter=external-dns.demo.io/access in (public)
  - --source=crd
  - --cloudflare-proxied
  - --cloudflare-dns-records-per-page=5000
  - --default-targets=ingress-external.example.com

Result DNS with default-targets set is expected to be:

demo.example.com CNAME ingress-external.example.com

In PR, you can see if we do set targets in CRD, then I'd expect those targets to be respected.

[ivankatliarchuk]

Not sure about practicality, may need some time to re-think.

How this DNSEndpoint(s) created? I'm using helm, example relevant part

  {{- range $hostNames }}
  {{- if .name }}
  {{- $target := printf "eks-cluster-%s-ingress-%s.%s" $.Values.cluster.env $accessType $defaultDomainName }}
  {{- $subdomain := printf "%s-%s-eks" .name $accessType }}
  {{- if and (eq (include "this.allPortsAreGrpc" $) "false") (ne .protocol "gRPC") }}
  - dnsName: {{ $subdomain }}.{{ $defaultDomainName }}
    recordTTL: 60
    recordType: CNAME
    targets:
    - {{ $target }}
  {{- end }}
  {{- if and (eq $accessType "internal") (eq (include "this.http2Enabled" $) "true") }}
  - dnsName: {{ $subdomain }}.http2.{{ $defaultDomainName }}
    recordTTL: 60
    recordType: CNAME
    targets:
    - {{ $target }}
  {{- end }}
  {{- end }}
  {{- end }}

or
    - dnsName: {{ .Values.dns.host }}
      recordTTL: 60
      recordType: CNAME
      targets:
        - {{ include "common.dns.target" . }} # target or default target

I may missing some other use case

Or similar with kustomize or kyverno admission webhooks.

[alen-z]

It's simple:

1. Using Helm.
2. Running multiple ExternalDNS instances for different providers and public/private settings.
3. Each instance has default-targets flag set that points to cluster load balancer created during cluster initialization with Terraform.
4. DNSEndpoint is losing point in setting targets as it's using set default-targets.

This is PR to improve 4th step.

Another approach I'm using is different starting with 3rd step, which is setting annotations on Service of type LoadBalancer to create DNS for cluster load balancer. I configure default-targets with load balancer DNS I've set in annotation and again DNSEndpoint targets currently need to contain some arbitrary placeholder value.

Both approaches have product teams using DNSEndpoint resources, while load balancer DNS records and ExternalDNS instances are maintained by operations team. Product team just creates proper DNSEndpoint with annotation for public or private (again, bummer is it needs to contain targets placeholder).

[ivankatliarchuk]

Another approach I'm using is different starting with 3rd step, which is setting annotations on Service of type LoadBalancer to create DNS for cluster load balancer. I configure default-targets with load balancer DNS I've set in annotation and again DNSEndpoint targets currently need to contain some arbitrary placeholder value.

If I understand correctly, we have exactly same cases in environments I look after.

For the use case you shared, I'm not sure how you setting annotations, but if annotations on ALB are set with helm, you could add a template for DNSEndpoint as well and set all required values, use kustomize and any other tools. The targets are static, so it works.

If teams using Crossplane, it could manage DNSEndpoint targets as well at runtime.

Regardless of how DNSEndpoint manifests are generated (Helm, Kustomize, Admission WebHooks, ArgoCD, Jsonnet, etc.), any DNSEndpoint with the external-dns.demo.io/access annotation can have its necessary targets defined at manifest apply time.

I apologise, but I don't understand the proposed approach and don't see any clear benefits. However, other contributors or maintainers might have a different perspective and see value in this feature.

[alen-z]

Again, it's simple proposition: No need for targets in CRD if default-targets are set. Common sense. Rest is going around it, which is always possible, of course.

Additional comment on your suggestion that it's easy to add targets to annotations or CRD: What if you have 100 products and targets change? One simple central change in default-targets or redeploy 100 products owned by product teams with new template values?

[ivankatliarchuk]

We have differing opinions on this. In my view, the number of DNSEndpoints shouldn't matter – automation tools exist to handle scaling so 1, 100, 10000 should not be matter. If other maintainers or contributors believe external-dns should support this specific scenario, I don't see a fundamental conflict as consensus is not needed. However, from my perspective, this problem seems too niche and is better addressed with dedicated tools. This is just my personal opinion, and others may have different views.

[ivankatliarchuk]

Maybe CRD should support FQDN templating --fqdn-template=""?

[alen-z]

All good. I'm not attached to this, only offered a fix to make it better for our use case. I can close the PR.

[ivankatliarchuk]

@alen-z do you wanna try FQDN approach?

@mloiseleur wdyt?

[mloiseleur]

@ivankatliarchuk To me, this use case is quite straightforward and valid.

We are using the same pattern on some clusters: a single target (the LB) with as many CNAME as we need. So, setting it once in the CLI arg looks better to me than set the same value in all the CR.

It's also consistent: that's how other sources works.

Since there is no CEL validation ATM in the CRD, I'm not sure if this is really a breaking change. With current version, it will fail without target so, to me, a warning in release notes should be enough. But maybe I missed something, I need to double check that.

[alen-z]

@mloiseleur, in terms of looking at breaking change, depends on the approach (current PR or one other possible non-breaking).

Challenge is, to make current ExternalDNS work, many have put placeholders in CRD targets (to satisfy the validation) while using default-targets argument. If dummy CRD targets start to suddenly apply, this could have wide consequences.

To me, it's best to be able to upgrade to new version, clean DNSEndpoint under the flag, remove the flag. This actually requires this PR to be a bit more improved to allow empty targets in legacy mode that constantly enforces default-targets. Because this PR proposes overriding default-targets with CRD targets if set. To me, this makes sense.

If you folks align on the approach, I can prepare PR.