RFC2136/GSS-TSIG against Active Directory broken on external-dns >= 0.18.0 (works fine on 0.15.0)

[Mick46]

Summary

RFC2136 provider with GSS-TSIG authentication against a Windows Active Directory
DNS server works correctly on external-dns v0.15.0, but fails with a GSS-API
error on v0.18.0 and v0.21.0 (haven't tested versions in between). This
prevents upgrading past 0.15.0 for any AD-integrated RFC2136 setup.

Environment

• external-dns versions tested: v0.15.0 (works), v0.18.0 (fails), v0.21.0 (fails)
• DNS server: Windows Server Active Directory-integrated DNS (dynamic updates
  set to "Secure only")
• Auth: --rfc2136-gss-tsig with --rfc2136-kerberos-username /
  --rfc2136-kerberos-password / --rfc2136-kerberos-realm
• Provider config:
• --provider=rfc2136
  --rfc2136-gss-tsig
  --rfc2136-tsig-axfr
  --rfc2136-kerberos-realm=
  --rfc2136-kerberos-username=
  --rfc2136-kerberos-password=$(PASSWORD)
  --rfc2136-min-ttl=15m
  --rfc2136-zone=

Error (v0.21.0, --log-level=debug)

{"level":"error","msg":"RFC2136 create record failed: soft error"}
...
unexpected acceptor flag is not set: expecting a token from the acceptor, not in the initiator

Root cause analysis

This traces to github.com/bodgit/tsig (the GSS-TSIG library used by
external-dns's rfc2136 provider), specifically to
gss/gokrb5.go:

apreq, err := spnego.NewKRB5TokenAPREQ(cl, tkt, key,
    []int{gssapi.ContextFlagMutual, gssapi.ContextFlagReplay, gssapi.ContextFlagInteg},
    []int{flags.APOptionMutualRequired})

This is a known, long-standing issue: bodgit/tsig#54
(bodgit/tsig#54), open since 2021, specifically
reproducible against Active Directory DNS servers. The issue reporter traced
it to these exact parameters, and found that the ns1/tsig fork
(https://github.com/ns1/tsig/blob/master/gss/gokrb5.go#L150) uses different
values and does not trigger the error:

apreq, err := spnego.NewKRB5TokenAPREQ(cl, tkt, key,
    []int{gssapi.ContextFlagInteg, gssapi.ContextFlagMutual},
    []int{gssapi.ContextFlagMutual})

I don't know why this resolves the issue, but I can confirm it's reproducible:
external-dns v0.15.0 works against our AD DNS server, and v0.18.0/v0.21.0 do
not, with the exact error above. Something changed between external-dns'
usage/bundled version of bodgit/tsig in 0.15.0 vs later releases that
surfaces this pre-existing library bug.

Related issues

• RFC2136 GSS-TSIG verify checksum fails #5123 — different symptom (checksum mismatch
  during verify, not during AP_REQ negotiation), but likely related root
  cause in the same GSS-TSIG/AD code path
• Error updating DNS record: unexpected acceptor flag is not set: expecting a token from the acceptor, not in the initiator bodgit/tsig#54 — original upstream report of this exact "acceptor flag"
  error, open since 2021, not yet fixed

Impact

Anyone using --provider=rfc2136 --rfc2136-gss-tsig against Windows AD is
stuck on external-dns v0.15.0 and cannot safely upgrade (missing newer
features, Gateway API v1 support, other bugfixes, etc.).

Ask

• Could the external-dns maintainers evaluate pinning/patching bodgit/tsig
  with the flag fix from the ns1/tsig fork, or track/escalate
  bodgit/tsig#54 upstream?
• Alternatively, is there a supported workaround (e.g. a webhook provider
  path) recommended for AD-backed RFC2136 users on current external-dns
  releases?

Happy to provide full debug logs / test any patched build against our AD
environment.

[bodgit]

Said library maintainer here. The linked issue isn't a "longstanding, known, unfixed bug" as that error gets thrown for lots of reasons and has quite a simple meaning. I should just close that issue as there's nothing really to fix.

When using GSS-TSIG, signatures aren't symmetrical, there's very much a concept of my signatures, (the initiator/client), and their signatures, (the acceptor/server), using MIC tokens, which have a flag indicating the source/direction using what's known as the acceptor flag. When sending requests to the server, a signature is sent with the acceptor flag cleared. When the server replies, it should send a signature with the acceptor flag set, it's that simple. I should never get a response with the acceptor flag unset, that's what that error means, nothing more.

The problem is that Windows for many reasons will reply and send your signature back to you. This then violates the above logic. Reasons I've so far seen:

• Sending an update to create a wildcard DNS record.
• Sending an update to create record types other than A, PTR or CNAME.
• Sending an update to a zone configured for both non-secure and secure updates, (the dumbest combination). In this particular case, Windows ignores the security settings but honours the update as if you didn't sign the request in the first place.

This is exacerbated because everyone (correctly IMO) writes something like the following idiomatic code:

rr, _, err := client.Exchange(msg, host)
if err != nil {
        panic(err)
}

// Do something with the response
if rr.Rcode != dns.RcodeSuccess {
        fmt.Printf("DNS error: %s (%d)\n", dns.RcodeToString[rr.Rcode], rr.Rcode)
}

The problem is the response signature is checked during the Exchange() call before the caller has chance to look at the actual response itself so when Windows encounters one of the above scenarios, (or another), your error handling fires first and you never check the response code which would be "refused" for the first two cases and is the actual error.

The context flags are negotiated before the update exchange happens using TKEY records, so if the server didn't want to provide replay protection, (the flag being removed), the negotiation would fail before you even tried to send the actual update. Those flags are set identically to the nsupdate utility which for good or bad is the standard by which other GSS-TSIG implementations are tested & measured. This code also works fine for me, and I feel many other people also, as I'd no doubt hear otherwise as this library gets used in a few projects as well as this one. The occasional person having problems points to a misconfiguration in their AD/DNS settings, the first thing I'd check is the zone configuration and whether the AD principal has the necessary permissions. PCAP captures can also be illuminating.

[Mick46]

Update: packet capture confirms this is not a config/data issue on our side

Followed up with tcpdump against our AD DNS server to see what's actually
happening on the wire, independent of how the Go client surfaces the error.

Tested with --rfc2136-batch-change-size=1 (ruling out batch size as a
factor) and plain A record creates only.

Every single UPDATE, regardless of hostname, gets this exchange:

Request (client -> server): OPCODE=UPDATE, one A record, PRCOUNT=0,
TSIG present with error=NOERROR.

Response (server -> client): OPCODE=UPDATE, RCODE=YXRRSET, TSIG
present with error=NOERROR.

This is the actual wire-level RCODE, decoded directly from the packet capture
— it's not something the miekg/dns or bodgit/tsig client library is inventing;
the Windows AD DNS server is sending this back. But YXRRSET (RFC 2136 §3.2)
is only supposed to be returned in response to a prerequisite check, and
PRCOUNT=0 in every one of these messages — there's no prerequisite section
for the server to be evaluating a conflict against.

I reproduced this identically across 3 different single-record UPDATE
messages (3 different hostnames, all fresh records not previously present in
the zone), 3/3 got YXRRSET.

I've raised this with the bodgit/tsig maintainer directly
(bodgit/tsig#54), who's confirmed the "unexpected acceptor flag" error is a
client-side signature-verification failure that happens before the caller
ever gets to see the actual RCODE — which matches what I'm seeing: the
YXRRSET never reaches external-dns's own logs, only the acceptor-flag error
does. Per the maintainer, this class of issue is a known Windows-side
GSS-TSIG quirk in certain scenarios, though the specific trigger for our case
isn't identified yet.

What this rules out on the external-dns side:

• Batch size (tested --rfc2136-batch-change-size=1)
• Record type (plain A, not AAAA/wildcard/unusual types)
• Prerequisites (none sent)
• Zone security config (confirmed "Secure only", not the "Nonsecure and
  secure" misconfiguration the bodgit/tsig maintainer flagged as a common
  cause)

At this point I don't think there's anything actionable on external-dns's
side beyond what's already tracked here — this looks like it needs a fix (or
at least a documented workaround) in bodgit/tsig itself, or a config-level
workaround (relaxing to nonsecure updates) for affected AD environments. Happy
to keep testing if there's something specific worth ruling in/out.

[Mick46]

Update: found the practical fix — --force-default-targets was the missing piece

Followed up extensively on this (real-server testing, protocol-level experiments). Posting a consolidated summary since it changes the picture quite a bit.

TL;DR

For our environment, upgrading past v0.15.0 works fine once --force-default-targets is added. Without it, v0.21.0's new default-targets behavior makes it think almost the entire zone is "missing" (existing records are CNAMEs, it now wants direct A/AAAA records instead) and tries to recreate hundreds of records in one go. That self-inflicted bulk burst is what was actually breaking things for us — not a fundamental GSS-TSIG protocol bug.

What we ruled out (real-server testing, not just reasoning)

Built a standalone Go repro (bypassing external-dns entirely, talking to bodgit/tsig/gokrb5 directly) to test the acceptor-flag/signature theory in isolation against our AD DC. All of the following were implemented and tested for real, and none of them fixed it on their own:

• Removing gssapi.ContextFlagReplay (the fix suggested earlier in Error updating DNS record: unexpected acceptor flag is not set: expecting a token from the acceptor, not in the initiator bodgit/tsig#54)
• Wrapping the AP-REQ in SPNEGO (nsupdate/Heimdal does this; the Go stack doesn't)
• A real bug found in jcmturner/gokrb5: the Authenticator's EncryptedData carries a spurious kvno it shouldn't (RFC 4120 §5.3), and no GSS subkey gets generated (RFC 4121 §4.1.1) — fixed both, still no change
• Full GSS credential delegation (GSS_C_DELEG_FLAG, KRB-CRED) — implemented end-to-end, the AD KDC itself rejects the forwarded-ticket request with KDC_ERR_BADOPTION (a policy restriction on the service account), and since nsupdate succeeds without delegation, this ruled itself out too

Also confirmed via packet capture + ASN.1 diffing that nsupdate/Heimdal against the same server/account/zone succeeds cleanly — so this really is something specific to the Go client stack's interaction with this DC, just not any of the above.

The actual "why does 0.15.0 work" answer

Ran a controlled test: with --force-default-targets in place (avoiding the self-inflicted bulk resync above), isolated single-record writes succeeded reliably — 4/4 in a row, confirmed via live DNS lookups each time. Separately, bulk bursts of many simultaneous GSS-TSIG negotiations (11–1000+ in quick succession, e.g. right after a version switch when a lot of "drift" needs reconciling) showed degraded, inconsistent success rates (0%–50% across different test runs). So the acceptor-flag issue does still occur, but appears correlated with negotiation burst/concurrency rather than being deterministic per-request — and normal steady-state reconciliation (one ingress changed at a time, the realistic day-to-day pattern) essentially never hits that regime. This matches @bodgit's framing that Windows will sometimes reflect the client's own signature back for reasons that aren't fully pinned down, rather than there being one single deterministic trigger.

Practical takeaway for anyone hitting this

• Add --force-default-targets when upgrading past v0.15.x if your existing records are CNAMEs to a default target — otherwise expect a large one-time resync attempt that can look exactly like widespread GSS-TSIG failure.
• If you ever do need a large bulk resync against an AD DC that's prone to this, expect a non-trivial failure rate on that specific operation even though steady-state usage is fine — worth monitoring rather than assuming total failure.

Happy to share the standalone repro tooling / packet captures if useful for anyone still chasing the actual Windows-side root cause of the acceptor-flag reflection under bursty conditions.

[bodgit]

I'd be interested to know what the gokrb5 bugs/fixes were. Sadly the maintainer has gone AWOL and doesn't appear to be active any longer, however there is a fork which might make sense to submit the fixes to.

Each context negotiated via TKEY records maintains its own sequence numbers, and this is implemented in the same way that is implemented in the MIT codebase, (I've not checked Heimdal), in that it tracks a sliding window of the last 64 sequence numbers for replay protection. There is a much stricter sequence check that can be negotiated that ensures a monotonic increasing number with no gaps but we don't ever use that, (nsupdate doesn't either).

I don't know how Windows tracks the context, i.e. 1,000 contexts in parallel would have 1,000 separate starting sequence numbers keyed against each TKEY key name on my side, but does Windows assume that they're all for the same AD principal and key them against that? That would confuse the sequence tracking as every TKEY negotiation would clobber/reset the expected sequence number. Just a wild theory, might be possible to test that?

If it's useful as another data point, my code can also be built against the Kerberos/GSSAPI C libraries rather than use the gokrb5 package by passing the build tag apcera, the downside to this is that it has no way to acquire Kerberos credentials so you have to perform a kinit before starting any software. It also makes cross-compiling difficult/impossible but it was the first implementation I got working so I've never removed it.