Records created without ownership TXTs when a same-named TXT owned by a different owner ID is present

[dnraitzyk]

Records created without ownership TXTs when a same-named TXT owned by a different owner ID is present

Summary

Two external-dns instances share a Route53 zone (policy=sync, TXT registry, distinct --txt-owner-id). During a transfer of 14 hostnames from cluster-a to cluster-b, cluster-b created all 14 CNAMEs and wrote no ownership TXTs for them. Every reconcile since logs "All records are already up to date", including ones that do a fresh provider read. I recovered by creating the TXT records by hand.

What I think is happening

This is our reading of the code, and I may be missing context on why it works this way.

existingTXTs.isAbsent gates ownership TXT creation on the TXT record's name alone, with no reference to the owner value (registry/txt/registry.go, master 13368cde, same logic since v0.19.0):

func (im *existingTXTs) isAbsent(ep *endpoint.Endpoint) bool {
	key := recordKey{dnsName: ep.DNSName, setIdentifier: ep.SetIdentifier}
	return !im.entries.Has(key)   // entries is a sets.Set[recordKey], no owner stored
}

// ApplyChanges, line 372
filteredChanges.Create = append(filteredChanges.Create, im.generateTXTRecordWithFilter(r, im.existingTXTs.isAbsent)...)

existingTXTs is filled in Records() from every TXT the provider returns, regardless of owner. So when cluster-b went to create a record and the provider response still held cluster-a's TXT at that registry name, isAbsent returned false and cluster-b skipped writing its own.

It doesn't self-correct. Once cluster-a's TXTs were gone I could not find a path that writes the missing TXT: the record now matches desired state so plan.appendEndpointUpdates emits nothing, it has no owner label so FilterEndpointsByOwnerID would drop it from UpdateNew/Delete anyway, and no Create is generated because the record exists. If there is a mechanism that should have caught this, I would be glad to hear what I am missing.

Timeline

20:11:20  cluster-b  Records cache provider: refreshing records list cache
                     Last real Route53 read. cluster-a's 14 TXTs present. Cache TTL to ~20:14:20.
20:13:22  cluster-b  All records are already up to date        (served from cache, no refresh line)
20:14:13  cluster-a  Desired change: DELETE ... x28            (14 CNAME + 14 TXT, one batch)
20:15:22  cluster-b  Records cache provider: refreshing records list cache
                     Cache expired, fresh Route53 read, 69.2s after cluster-a's batch.
20:15:23  cluster-b  Desired change: CREATE ... x14            (14 CNAME, zero TXT)
20:17:23  cluster-b  Records cache provider: refreshing records list cache
20:17:24  cluster-b  All records are already up to date        (fresh read, cluster-a's TXTs gone)

Desired change: is logged by the AWS provider in submitChanges per change actually submitted. cluster-a's output shows its TXT deletes, cluster-b's shows no TXT lines at all, which points to the TXTs being dropped before the batch reached the provider rather than lost at Route53.

The specific sequence needed unlucky timing, but the condition it depends on looks broader: any TXT at the registry name owned by someone else, present in the response the incoming instance reads. An orphaned TXT from a removed or renamed instance, or a partially applied batch as in #6582, would look the same.

Possible directions

Make the filter owner-aware: skip TXT creation when the existing TXT is already owned by im.ownerID, rather than whenever one exists. That means storing the parsed owner alongside the key in existingTXTs. It looks contained, since the owner is already parsed a few lines above the add() call, but I may be underestimating the blast radius.

A caveat if the filter is loosened. The generated TXT goes into changes.Create, and at least the AWS provider submits creates as ChangeActionCreate rather than upsert (provider/aws/aws.go:715). A CREATE against an existing RRset is rejected with InvalidChangeBatch, and the batch is one transaction, so I would expect that to fail the other records too. If that is right, a filter change alone may not be enough, and the write would need the upsert path or a delete plus create. Other providers may differ in ways I have not looked at.

On the behavior change. Writing over another instance's TXT changes semantics, and you will have a far better sense than I do of who relies on the current behavior. Setups like #1441 share names deliberately, so a flag may be safer if this is contentious.

A smaller option: warn when a record is created while its ownership TXT was filtered out due to a different owner. Even with no behavior change, that turns this into a quick diagnosis. As it stood the only output was "14 record(s) were successfully updated", which read as success.

Workaround, and a question on the docs

docs/registry/txt.md, "When to avoid owner migration" warns against --txt-owner-id-old in shared zones with policy=sync, and suggests "per-cluster zones, manual TXT record adjustment, or fully coordinated migration of all clusters".

Manual adjustment is what I do now: rewrite the TXT value to the incoming owner before the config change, wait a reconcile interval, then make the change. The incoming instance finds the record already claimed and issues a target update instead of a create, which also avoids a resolution gap.

Is a supported path for cluster-to-cluster ownership transfer something you would consider, or is manual adjustment the intended answer?

Environment and collected data

Process flags

From kubectl get pod -o jsonpath='{.spec.containers[*].args}'. Both instances identical apart from --txt-owner-id.

--log-level=info
--log-format=json
--interval=2m
--source=crd
--policy=sync
--registry=txt
--txt-owner-id=cluster-b
--txt-prefix=_external-dns.
--provider=aws
--aws-zone-type=public
--provider-cache-time=3m
--domain-filter=example.com
--crd-source-apiversion=example.io/v1
--crd-source-kind=DNSEndpoint

Source resource (source=crd)

One of the seven objects that moved between instances, trimmed to the relevant fields. Each contributes a plain CNAME and a wildcard CNAME, 14 records in total. The transfer was a config change moving these from cluster-a to cluster-b.

apiVersion: example.io/v1
kind: DNSEndpoint
metadata:
  name: svc-1-dns-endpoint
  namespace: svc-1
spec:
  endpoints:
    - dnsName: svc-1.example.com
      recordType: CNAME
      targets:
        - cluster-b.example.com
    - dnsName: '*.svc-1.example.com'
      recordType: CNAME
      targets:
        - cluster-b.example.com
status:
  observedGeneration: 1

DNS records: actual vs expected

State immediately after the transfer, for each of the seven name pairs:

Record	Type	Value	Expected?
svc-1.example.com	CNAME	cluster-b.example.com	yes
*.svc-1.example.com	CNAME	cluster-b.example.com	yes
_external-dns.cname-svc-1.example.com	TXT	absent	no, expected "heritage=external-dns,external-dns/owner=cluster-b,external-dns/resource=crd/svc-1/svc-1-dns-endpoint"
_external-dns.cname-*.svc-1.example.com	TXT	absent	no, same as above

Before the transfer all four existed, with the TXT values carrying external-dns/owner=cluster-a.

Verbatim logs

Hostnames redacted to svc-N.example.com, everything else as logged.

cluster-a, deleting the CNAMEs and its own ownership TXTs in one batch:

2026-08-06T20:14:13.644Z  Desired change: DELETE svc-1.example.com CNAME
2026-08-06T20:14:13.644Z  Desired change: DELETE *.svc-1.example.com CNAME
2026-08-06T20:14:13.644Z  Desired change: DELETE _external-dns.cname-svc-1.example.com TXT
2026-08-06T20:14:13.644Z  Desired change: DELETE _external-dns.cname-*.svc-1.example.com TXT
   ... the same four lines for svc-2 through svc-7 ...
2026-08-06T20:14:13.703Z  28 record(s) were successfully updated

cluster-b, 69 seconds later:

2026-08-06T20:15:22.885Z  Records cache provider: refreshing records list cache
2026-08-06T20:15:23.818Z  Applying provider record filter for domains: [...]
2026-08-06T20:15:23.864Z  Desired change: CREATE svc-1.example.com CNAME
2026-08-06T20:15:23.864Z  Desired change: CREATE *.svc-1.example.com CNAME
2026-08-06T20:15:23.864Z  Desired change: CREATE svc-2.example.com CNAME
2026-08-06T20:15:23.864Z  Desired change: CREATE *.svc-2.example.com CNAME
   ... 10 more CNAME creates, no TXT lines at all ...
2026-08-06T20:15:23.910Z  14 record(s) were successfully updated

Two details behind the timeline

The provider cache refreshes only on a real read, and is reset only when changes are applied. CachedProvider.ApplyChanges returns early without calling Reset() when !changes.HasChanges(), so an idle instance does a fresh read about once per --provider-cache-time. That would make the read at 20:15:22 cluster-b's first look at the zone after the deletes at 20:14:13, with no chance to see an intermediate state.

The read looks internally inconsistent. Both the CNAME deletes and the TXT deletes were in one batch, but cluster-b's read seems to have reflected the CNAME deletes and not the TXT deletes: it created all 14 CNAMEs, which I think plan.appendTakenDNSNameChanges would have refused had they still appeared present under cluster-a's ownership, while suppressing all 14 TXTs. ListResourceRecordSets is paginated and Route53 reads are eventually consistent, so a response straddling a propagating change seems plausible, though I cannot confirm that from the logs.

Checklist

• I have searched existing issues and tried to find a fix myself
• I am using the latest release. I am on v0.19.0. The code path appears unchanged through v0.21.0 and master, but I have not reproduced on the latest release. Happy to attempt a kind-based reproduction if useful, though the trigger involves provider read timing that may be awkward to stage locally.
• I have provided the actual process flags (not Helm values)
• I have provided the source resource YAML
• I have provided debug logs. Logs are at --log-level=info, since this was an unanticipated production incident.
• I have described what DNS records exist and what I expected

Related

• #5459 / #4914: where the filter came from.
• #6368: ownership TXT deleted and not recreated after upgrading to v0.20.0. Same end state, different trigger.
• #6430: record created without its ownership TXT when the registry name overflows 63 characters. Same end state, different cause.
• #5457: ownership data overwritten when multiple TXT records exist for a name. Similar setup, different mechanism.
• #6582: Create-batch ordering in the same ApplyChanges loop.
• #5465: TXT registry meta-issue.