Handshake not working although nodes are reaching each other

[martinsson]

What version of nebula are you using? (nebula -version)

1.9.3

What operating system are you using?

Linux

Describe the Bug

I have two nodes, A and B

When A is initiating a handshake, node B starts logging about a handshake (that times out).
When B is initiating a handshake, node A starts logging about a handshake (that times out).

So the communication works both ways sort of, yet they don't succeed in establishing a handshake.

Node A is on aws with a public IP, node B is behind a LTE router with no public IP. The surprising thing is that at times it will work without me changing anything.

If I activate the relay on the node behind the LTE router it seems to work systematically. So I do have this work-around, but still I'd like to understand what's happening here. I cannot fathom that both nodes succeed in contacting each other without them being able to conclude the handshake

The lighthouse can always ping both nodes

Logs from affected hosts

Node A

Jan 07 16:52:37 ip-172-16-45-15 nebula[3884989]: time="2025-01-07T16:52:37Z" level=info msg="Handshake message sent" handshake="map[stage:1 style:ix_psk0]" initiatorIndex=606942997 localIndex=606942997 remoteIndex=0 udpAddrs="[49.93.23.52:17573 172.17.0.1:46695 172.18.0.1:46695 192.168.26.114:46695]" vpnIp=10.100.100.53
Jan 07 16:52:43 ip-172-16-45-15 nebula[3884989]: time="2025-01-07T16:52:43Z" level=info msg="Handshake timed out" durationNs=6126247464 handshake="map[stage:1 style:ix_psk0]" initiatorIndex=606942997 localIndex=606942997 remoteIndex=0 udpAddrs="[49.93.23.52:17573 172.17.0.1:46695 172.18.0.1:46695 192.168.26.114:46695]" vpnIp=10.100.100.53

Node B

Jan 07 16:51:51 nuc-08 nebula[697]: time="2025-01-07T16:51:51Z" level=info msg="Handshake message sent" handshake="map[stage:1 style:ix_psk0]" initiatorIndex=2092318468 localIndex=2092318468 remoteIndex=0 udpAddrs="[54.223.124.224:42446 172.16.45.15:42446 172.17.0.1:42446 172.19.0.1:42446]" vpnIp=10.100.100.56
Jan 07 16:51:57 nuc-08 nebula[697]: time="2025-01-07T16:51:57Z" level=info msg="Handshake timed out" durationNs=6162017980 handshake="map[stage:1 style:ix_psk0]" initiatorIndex=2092318468 localIndex=2092318468 remoteIndex=0 udpAddrs="[54.223.124.224:42446 172.16.45.15:42446 172.17.0.1:42446 172.19.0.1:42446]" vpnIp=10.100.100.56

Config files from affected hosts

Both nodes share the same config

...
static_host_map:
        "10.100.100.2": ["54.222.204.236:4242"]


lighthouse:
  am_lighthouse: false
  interval: 60
  hosts:
    - "10.100.100.2"

  remote_allow_list:
    '::/0': false # make sure that no ipv6 addresses are used for p2p communications

listen:
  host: 0.0.0.0
  port: 0

punchy:
  punch: true
  respond: true

firewall:
  conntrack:
    tcp_timeout: 12m
    udp_timeout: 3m
    default_timeout: 10m
    max_connections: 100000

outbound:
    # Allow all outbound traffic from this node
    - port: any
      proto: any
      host: any

inbound:
    # Allow icmp between any nebula hosts
    - port: any
      proto: icmp
      host: any

 ...

[nbrownus]

I suspect that you are in a double NAT situation and the LTE side is probably using a symmetric NAT, making this a really tricky hole punching problem.

The easiest way to escape without using a relay is to change the aws side to run nebula on a fixed port and specifically allow inbound traffic on that port. This will remove the NAT on the aws side and since you already have punchy.respond: true both sides should attempt to initiate a handshake, the LTE side will be the winner.

[martinsson]

Thanks for that quick reply.

I haven't been able to verify that it resolves the issue because it's working right now. As I mentioned it works from time to time. I'll keep retesting it to be sure.

Out of curiosity: how can a double nat be the problem given that packets are received and sent both ways already?

[nbrownus]

I didn't see evidence that both sides could communicate in the logs you provided. Unless your AWS network is allow all then a double NAT is the most likely scenario.