Keepalive fails after ipsec tunnel rekeying

[zyzzyxdonta]

Hi!

I've been experiencing some problems. Unfortunately, I don't know when this started (or if it ever worked correctly). I frequently loose connection. Sometimes it happens just a few seconds after connecting, sometimes it takes an hour.

I recorded a trace now of one such connection loss. It seems like keepalive doesn't work any more after rekeying the ipsec tunnel. I don't know if this is always what happens or if this was an exception.

My config is:

server-name=cp.hzdr.de
user-name=REDACTED
password=
password-factor=1
search-domains=
ignore-search-domains=
dns-servers=
ignore-dns-servers=
default-route=false
no-routing=false
add-routes=
ignore-routes=
no-dns=false
ignore-server-cert=false
tunnel-type=ipsec
ca-cert=
login-type=vpn_TwoFA_Token
cert-type=none
no-keychain=true
ike-lifetime=28800
ike-persist=false
log-level=info
client-mode=secure_connect
no-keepalive=false
icon-theme=auto
set-routing-domains=false
port-knock=false
auto-connect=false

And the trace is attached:

snx-rs-trace.txt

Thanks in advance!

[ancwrd1]

Hi, I am wondering if your local IP address changes frequently, e.g. 192.168.178.32 in this case. Could you perhaps check that? The ipsec xfrm state requires the IP address to stay the same.

The other thing to try is to select a transport type manually: try transport-type=udp and see if it helps.

[zyzzyxdonta]

No, my IP address stays the same. I set transport-type=udp and it didn't fix the problem, either.

Here is another trace. This time, the connection failed pretty much immediately:

snx-rs-trace-transport-udp.txt

[ancwrd1]

Could be something with the routing. You have very strange setup that I've never seen before: VPN server allocates a public IP address for your machine in the 149.x range, instead of using private IP address space, and then advertises a hell lot of routes. This is not very common.

Could you try enabling the default route option and see if it makes a difference? E.g. default-route=true.

Another thing to try is the SSL tunnel (tunnel-type=ssl), may be something is wrong with IPSec policies.

[zyzzyxdonta]

Sorry, it's not my setup, so I don't know why things are configured the way they are. I'm merely the user here. But I can contact the admin if need be.

With default-route=true; tunnel-type=ipsec: This seems to work. It ran successfully for more than 90 minutes, including successful rekeying, before I disconnected to test the other option. This options means all traffic goes via VPN, right?

With default-route=false; tunnel-type=ssl: This also seems to work (over an hour including rekeying).

[ancwrd1]

This options means all traffic goes via VPN, right?

Yes, all traffic goes via the tunnel in this case.

The reason it works with SSL is probably because it uses different sort of keepalive which does not require a special routing.

I think the issue here is with some route retrieved from VPN server, which breaks the routing rule configured for keepalive. I will have a look closer at the retrieved routes. You could also try to add them one by one to the list of ignored routes, to find which one breaks it.

Another workaround is to ignore all acquired routes via no-routing=true option and add manually only those you really need (usually DNS servers with /32 prefix, and the ones to access intranet sites).

[zyzzyxdonta]

OK, I'll try some stuff. Thanks so much for the help! 🙏🏻

[ancwrd1]

I think it could be also related to the interface MTU, see #130.
Try 1280 in the settings.

[zyzzyxdonta]

Setting MTU doesn't solve it.

I tried quite a lot of things with routing but I'm not getting consistent behaviour. E.g., I'm losing connection with default-route=true, which didn't happen earlier. I also looked through the list of routes and I couldn't see any overlaps. Also using only 149.220.4.0/22 and 149.220.16.0/20, or only 149.220.4.2/32 and 149.220.16.11/32 (the two nameservers) didn't keep the connection either.

Next, I tried setting up a notifier so that I could immediately try some stuff once the connection fails again:

while ip address show snx-xfrm > /dev/null; do sleep 5; done; notify-send --urgency critical "snx-rs" "snx-rs disconnected at $(date)"

And what I found is that if I immediately try to reconnect after losing the connection, it looks like I am connect, but the first keepalive (I guess after 20 seconds) already fails. I'm beginning to feel like this issue is at least partially due to temporary connection issues that last longer than the keepalive retry can bridge. So my original description is probably incorrect.

I think I'm going to collect timestamps of when I lost the connection and then ask our admin if he can correlate them with anything he sees in his monitoring or logs. I might also try to compile snx-rs myself with significantly more retries in the keepalive to see if this "solves" it.

Thanks again!

[ancwrd1]

You could perhaps also increase the keepalive response timeout from 2 secs to a higher value.

[ancwrd1]

Also there is an option to disable keepalive completely, if the issue is only with it.

[zyzzyxdonta]

Also there is an option to disable keepalive completely, if the issue is only with it.

OK, I tried that first as it is available in the settings. Unfortunately it didn't help. I still lost the connection, just snx-rs didn't notice. 🤷🏻‍♂️

[ancwrd1]

Does it only happen with IPSec tunnel or also with SSL?

[ancwrd1]

That looks like something on the server side. Does the Windows client work reliably?
There could be some subtle differences in the handshake but I haven't seen any other similar reports from the users.

The is also a parameter called "client-mode", which you can try to set to various values: endpoint_security, secure_connect or secure_remote. Default is secure_connect.

[zyzzyxdonta]

Does it only happen with IPSec tunnel or also with SSL?

SSL works. It ran on two different days for more than 5 hours each without disconnecting. (But I was on site. Can try again from remote, tomorrow.)

Does the Windows client work reliably?

I don't have a Windows system to test with and don't know any colleague who has one. The colleagues on Mac don't have problems. In the past, I used the old 32bit snx linux cli client from Checkpoint (sk90240?) and it worked, too. I don't know which settings it used, I only set server and username in ~/.snxrc. But I remember one had to concatenate password and OTP because it couldn't ask for them separately 🙈

The is also a parameter called "client-mode",

IPsec with endpoint_security or secure_connect loses connection on the first keepalive; with secure_remote, I get "Connection error - No IPv4 in reply!", immediately.