cluster setup: Response failed TSIG signature verification (Client RCODE=FormatError, Client TSIG Error=NoError). while zone transfer

[audricd]

Hello,

This is a "please help me" kind of topic, and right off the bat: apologies for my lack of general network knowledge. but this is how i learn, setting up things on my homelab.

in the past, i used to have a working cluster at home. ive been doing changes (as usualy in a homelab. constantly evolving) and now im doing a hybrid setup. one server is on the cloud, the other is at home. i mention this because i think it may be relevant. as its a signifant change in the architecture

dns-01.lab is 10.10.10.2 is on cloud
dns-02.lab is 192.168.236.102 is at home

root@dns-01:~# cat /etc/hosts
127.0.0.1       localhost
::1             localhost ip6-localhost ip6-loopback
ff02::1         ip6-allnodes
ff02::2         ip6-allrouters
# --- BEGIN PVE ---
10.10.10.2 dns-01.lab dns-01
# --- END PVE ---
192.168.236.102 dns-02
192.168.236.102 dns-02.lab
10.10.10.2      dns-01
10.10.10.2      dns-01.lab
10.10.10.3      ca
root@dns-01:~# ping dns-02.lab
PING dns-02.lab (192.168.236.102) 56(84) bytes of data.
64 bytes from dns-02 (192.168.236.102): icmp_seq=1 ttl=62 time=94.2 ms
64 bytes from dns-02 (192.168.236.102): icmp_seq=2 ttl=62 time=95.1 ms
^C
--- dns-02.lab ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 94.200/94.674/95.148/0.474 ms
root@dns-01:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: eth0@if297: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether bc:24:11:10:f6:07 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.10.10.2/24 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::be24:11ff:fe10:f607/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever
3: tailscale0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none 
    inet6 fe80::4ceb:5fea:dd5e:27aa/64 scope link stable-privacy proto kernel_ll 
       valid_lft forever preferred_lft forever

root@dns-02:~# cat /etc/hosts
127.0.0.1       localhost
::1             localhost ip6-localhost ip6-loopback
ff02::1         ip6-allnodes
ff02::2         ip6-allrouters
# --- BEGIN PVE ---
192.168.236.102 dns-02.lab dns-02
# --- END PVE ---
192.168.236.102 dns-02
192.168.236.102 dns-02.lab 
10.10.10.2      dns-01
10.10.10.2      dns-01.lab
10.10.10.3      ca.lab
root@dns-02:~# ping dns-01.lab
PING dns-01.lab (10.10.10.2) 56(84) bytes of data.
64 bytes from dns-01 (10.10.10.2): icmp_seq=1 ttl=62 time=91.4 ms
64 bytes from dns-01 (10.10.10.2): icmp_seq=2 ttl=62 time=95.0 ms
^C
--- dns-01.lab ping statistics ---
3 packets transmitted, 2 received, 33.3333% packet loss, time 2002ms
rtt min/avg/max/mdev = 91.374/93.172/94.970/1.798 ms
root@dns-02:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: eth0@if37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether bc:24:11:cc:4d:cf brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.236.102/24 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::be24:11ff:fecc:4dcf/64 scope link proto kernel_ll 
       valid_lft forever preferred_lft forever

the cluster itself, apparently, works:
from node1:

from node2:

but it fails at transferring zones
and ive tried everything i can thing of. custom certificates (for joining cluster). custom tsig. but im doing this the most "out of the box" as i can, but i cant NOT get past this:

[2026-03-15 15:58:28 UTC] DNS Server failed to refresh 'cluster-catalog.lab' Secondary Catalog zone from: dns-01.lab (10.10.10.2)
TechnitiumLibrary.Net.Dns.DnsClientTsigResponseVerificationException: Response failed TSIG signature verification (Client RCODE=FormatError, Client TSIG Error=NoError).
   at TechnitiumLibrary.Net.Dns.DnsClient.<>c__DisplayClass97_0.<TsigResolveAsync>b__0(DnsDatagram signedResponse, CancellationToken cancellationToken1) in Z:\Technitium\Projects\TechnitiumLibrary\TechnitiumLibrary.Net\Dns\DnsClient.cs:line 5261
   at TechnitiumLibrary.Net.Dns.DnsClient.<>c__DisplayClass90_0.<<InternalResolveAsync>g__DoResolveAsync|1>d.MoveNext() in Z:\Technitium\Projects\TechnitiumLibrary\TechnitiumLibrary.Net\Dns\DnsClient.cs:line 4603
--- End of stack trace from previous location ---
   at TechnitiumLibrary.Net.Dns.DnsClient.<>c__DisplayClass90_0.<<InternalResolveAsync>g__DoResolveAsync|1>d.MoveNext() in Z:\Technitium\Projects\TechnitiumLibrary\TechnitiumLibrary.Net\Dns\DnsClient.cs:line 4772
--- End of stack trace from previous location ---
   at TechnitiumLibrary.Net.Dns.DnsClient.<>c__DisplayClass90_0.<<InternalResolveAsync>g__DoResolveAsync|1>d.MoveNext() in Z:\Technitium\Projects\TechnitiumLibrary\TechnitiumLibrary.Net\Dns\DnsClient.cs:line 4462
--- End of stack trace from previous location ---
   at TechnitiumLibrary.Net.Dns.DnsClient.InternalResolveAsync(DnsDatagram request, Func`3 getValidatedResponseAsync, Boolean doNotReorderNameServers, CancellationToken cancellationToken) in Z:\Technitium\Projects\TechnitiumLibrary\TechnitiumLibrary.Net\Dns\DnsClient.cs:line 4934
   at TechnitiumLibrary.Net.Dns.DnsClient.TsigResolveAsync(DnsDatagram request, TsigKey key, UInt16 fudge, CancellationToken cancellationToken) in Z:\Technitium\Projects\TechnitiumLibrary\TechnitiumLibrary.Net\Dns\DnsClient.cs:line 5254
   at DnsServerCore.Dns.Zones.SecondaryZone.RefreshZoneAsync(IReadOnlyList`1 primaryNameServers, DnsTransportProtocol zoneTransferProtocol, TsigKey key, Boolean validateZone) in Z:\Technitium\Projects\DnsServer\DnsServerCore\Dns\Zones\SecondaryZone.cs:line 447

i also tried; even for just troubleshooting sake; to NOT require TSIG. but i still throws me the same error.
its obviously not a connection issue; since they have connectivity in between them and the cluster itself does not throw any error. its "only" the zone transfer failing.

the time is the same on both nodes.

root@dns-01:~# timedatectl
               Local time: Sun 2026-03-15 17:13:20 CET
           Universal time: Sun 2026-03-15 16:13:20 UTC
                 RTC time: n/a
                Time zone: Europe/Madrid (CET, +0100)
System clock synchronized: yes
              NTP service: inactive
          RTC in local TZ: no
root@dns-02:~# timedatectl
               Local time: Sun 2026-03-15 17:13:12 CET
           Universal time: Sun 2026-03-15 16:13:12 UTC
                 RTC time: n/a
                Time zone: Europe/Madrid (CET, +0100)
System clock synchronized: yes
              NTP service: inactive
          RTC in local TZ: no

what i think is weird... and probably causing hell, is my SOA entry:

which im not able to edit, even if i add my user (admin) to edit and delete! i get the forbidden icon when trying to edit the entry. clearly "invalid" is well, not valid.

[ShreyasZare]

Thanks for the details.

TechnitiumLibrary.Net.Dns.DnsClientTsigResponseVerificationException: Response failed TSIG signature verification (Client RCODE=FormatError, Client TSIG Error=NoError).

From this error message, it looks like the DNS request is not reaching the primary node directly. Some network equipment is probably intercepting it and does not support TSIG causing this issue. The response being received probably has no TSIG record or there is some TSIG parameter mismatch.

So just make sure that something like your router does not have DNS related security feature enabled.

[audricd]

hello,
thank you for your answer.
okay. even without decent knowledge, i am not really convinced thats the issue. i will look for something on my network that strips tsig?

but how does that explain that, without exchanging zones, on origin dns-01, the SOA record says invalid? surely that cant be a firewall doing that.

isnt possible the connection happens, but when node2 tries to replicate the zone it stops at reading invalid? i dont blame it. i would too. even if that doesnt add up with the reported error message. but it wouldnt be the first or last time that happens, on any software, lets be honest.

[audricd]

okay, you were right.
i disabled dns over https on my router. my zone was able to transfer.

i still get the "invalid" entry for my SOA, and also, the zone failed to notify

from master:

[2026-03-16 17:30:03 UTC] DNS Server failed to notify name server 'dns-02.labaudric' (RCODE=Refused) for zone: labaudric
[2026-03-16 17:30:08 UTC] [192.168.236.1:2726] [TCP] DNS Server received zone transfer request for zone: cluster-catalog.lab

from slave:

[2026-03-16 17:25:12 UTC] Saved zone file for domain: cluster-catalog.lab
[2026-03-16 17:25:12 UTC] Saved zone file for domain: lab
[2026-03-16 17:25:13 UTC] DNS Server auth config file was saved: /etc/dns/auth.config
[2026-03-16 17:25:13 UTC] DNS Server auth config file was saved: /etc/dns/auth.config
[2026-03-16 17:25:13 UTC] Server configuration was synced from the Primary node successfully.
[2026-03-16 17:25:18 UTC] DNS Server Cluster config file was saved: /etc/dns/cluster.config
[2026-03-16 17:25:18 UTC] Saved zone file for domain: lab
[2026-03-16 17:30:08 UTC] DNS Server has started zone refresh for Secondary Catalog zone: cluster-catalog.lab
[2026-03-16 17:30:08 UTC] DNS Server successfully checked for 'cluster-catalog.lab' Secondary Catalog zone update from: dns-01.lab (10.10.10.2)
[2026-03-16 17:30:13 UTC] Saved zone file for domain: cluster-catalog.lab

[ShreyasZare]

Thanks for the details. Good to know that you found out that your router was interfering with DNS requests and now its fixed.

The invalid entries in SOA record are as per the Catalog zone standards [RFC 9432]. So there is no issue related to it.

The "Notify Failed" issue you see is most probably related to your router again. You need to make sure that any inbound DNS entries are forwarded by your router to the DNS server IP address on the local network. You need to make sure that there is port forwarding configured for this so that the NOTIFY requests from your primary server reaches the secondary server.