Pod crashes when being rate-limited by AWS Route53

[johnjeffers]

external-dns pods crash when throttled by Route53 rate limiting. I understand that this was supposed to have been resolved by #4166 but the problem is still occurring

What happened:

Pod crashes with this error:

time="2025-03-17T22:36:25Z" level=fatal msg="Failed to do run once: failed to list resource records sets for zone /hostedzone/REDACTED using aws profile \"default\ │
│ ": Throttling: Rate exceeded\n\tstatus code: 400, request id: 97e2dc96-fe67-4da8-8cd3-b64f6f2d23e0"

What you expected to happen:

Pod reports the failure due to rate limiting, but does not crash (soft error).

How to reproduce it (as minimally and precisely as possible):

Intermittent, depends on traffic from external-dns pods running in other clusters.

Anything else we need to know?:

I have already enabled the --interval=10m and --events flags to try to reduce the possibility of rate limiting, but I can't seem to eliminate it entirely.

Environment:

• External-DNS version (use external-dns --version): 0.15.0
• DNS provider: Route53
• Others:

[ivankatliarchuk]

Hi. Could you share the size of the estate. Number oh hosted zones, number of records. Is it reconciliation or create/update/delete?

[ivankatliarchuk]

Any chance you share relevant debug logs as well as all arguments. I'll try to replicate. I would assume external-dns is not the only service that manage Route53.

What I currently do for large estates, I stop external dns, while do maintenance. This is not correct. If I could find a pattern, may be able to submit a fix or proposal.

[johnjeffers]

Hi. Could you share the size of the estate. Number oh hosted zones, number of records. Is it reconciliation or create/update/delete?

3 zones. One of the zones is large, around 5,500 records. The other two zones are very small, less than 20 records. It seems to be happening on reconciliation.

I would assume external-dns is not the only service that manage Route53.

It's not the only thing that hits the route53 API, but it's by far the most common. I run 14 EKS clusters, each in a different region, in a single AWS account. Each cluster is running external-dns, so each of those 14 external-dns services is contributing to the account-based rate limit.

Any chance you share relevant debug logs as well as all arguments.

I don't have debug logs, but the startup args are:

      --log-level=info
      --log-format=text
      --interval=10m
      --events
      --source=service
      --source=ingress
      --policy=sync
      --registry=txt
      --txt-owner-id=prod.eu-west-3.company.co
      --txt-prefix=ext-dns-
      --domain-filter=company.cloud
      --domain-filter=company.io
      --domain-filter=prod.eu-west-3.company.co
      --provider=aws
      --aws-zone-type=public

Zone names above are fake, of course, but the pattern is correct. In this example, company.io would be the zone with 5,500 records.

I feel like it's important to stress that I'm not trying to solve the problem of rate limiting. I understand that will happen. I just don't understand why the pod is crashing when it is rate limited, instead of continuing to run and trying again later.

[ivankatliarchuk]

/assign

I keep getting throttling quite often as well. Most likely we need a way to configure API rate limit https://github.com/aws/aws-sdk-go-v2/blob/main/aws/ratelimit/token_rate_limit.go

[johnjeffers]

Can't it just be fixed so that being rate limited doesn't cause the pod to crash?