Jool fragmentation issues when NAT64 to IPv6 tunnel clients

[bkuebler]

Hi there,

want to use jool as SNAT64 with IPv6 tunnel clients (e.g., WireGuard). And i run into an issue, that packets got not correctly fragmented:

• IPv6 client connects via a tunnel interface to the SNAT64 Gateway (MTU 1420)
• Jool translates incoming IPv4 packets to IPv6 (expected max MTU 1280 due to lowest-ipv6-mtu configuration)

Expected behavior:

• Packet should be crafted/fragmented, that they will fit into the tunnel device MTU size or icmp replies has to be generated to the relying party.

Tested with the following steps:

• ICMP echo with default packet size works well
• HTTP or Small DNS requests/responses works < 1420 MTU
• HTTPS breaks
• ICMP echo via packet size of 1500 for testing

Discovered via tcpdump/wireshark the following on the last test via ICMP echo:

• Echo request from client to Gateway was fragmented correctly (size 1508) => 1368 + 140
• Echo request got translated to IPv4 from Gateway with packet size of 1508, 2 IPv4 fragments were sent 1480 + 28
• Echo reply received to Gateway with packet size of 1508, 2 IPv4 fragments (760, 748)
• Echo reply was generated from Gateway with payload size of 1508 and packet size of 1568
• Interface Response with ICMP6 package to big (of course)

Question:
• Do i miss something? I read the documentation of https://nicmx.github.io/Jool/en/mtu.html, but i expect, this setting should solve the discovered issue.

Diagram:

IPv6 Client (WireGuard)
        |
        | IPv6 packets (MTU 1420)
        v
   +-------------+
   |   Jool      |  <-- NAT64 (lowest-ipv6-mtu: 1280)
   +-------------+
        |
        | IPv4 packets (MTU: 1500)
        v
 Public Internet Server

OS: Debian 13.3
Package: 4.1.13-1.1 (jool-dkms, jool-tools)
Jool Version: 4.1.13.0
Debian Kernel: 6.12.63+deb13-cloud-amd64

[bkuebler]

I've updated the description, after more research to be more precise, about the current issue.

[ydahhrk]

Sorry it's been 2 weeks already...

Some questions:

• Who is the relying party in the diagram?
• Who is the Gateway in the diagram?

I'm going to assume that Gateway is the Jool box for the remainder of this message:

Echo request from client to Gateway was fragmented correctly (size 1508) => 1368 + 140

Ok, I'm a bit stuck with this. Are those payload sizes?

Since you went from 1500 to 1508, I'm guessing you added the ICMPv6 header? So 1508 is IPv6 payload, 1500 is ICMPv6 payload.

The actual initial packet is

• 1548 (total) = 40 (IPv6) + 8 (ICMPv6) + 1500 (data)

So the fragments are

• 1416 (total) = 40 (IPv6) + 8 (Frag) + 8 (ICMPv6) + 1360 (data)
• 188 (total) = 40 (IPv6) + 8 (Frag) + 140 (data)

1500 = 1360 + 140, so that makes sense.

Echo request got translated to IPv4 from Gateway with packet size of 1508, 2 IPv4 fragments were sent 1480 + 28

So the implied "complete" IPv4 packet would be

• 1528 (total) = 20 (IPv4) + 8 (ICMPv4) + 1500 (data)

Fragments:

• 1500 (total) = 20 (IPv4) + 8 (ICMPv4) + 1472 (data)
• 48 (total) = 20 (IPv4) + 28 (data)

Ok

Echo reply received to Gateway with packet size of 1508, 2 IPv4 fragments (760, 748)

• 780 = 20 + 8 + 752
• 768 = 20 + 748

It's weird that the data sizes are so changed. But okay...

Echo reply was generated from Gateway with payload size of 1508 and packet size of 1568

Hmm?

• 40 (IPv6) + 8 (ICMPv6) + 1500 (data) = 1548 (total)

Where did 1568 come from?

Ok... I guess it doesn't matter. The point is, Gateway seems to be mangling fragment sizes after Jool for some reason.

Questions:

• Is this a SIIT Jool or a NAT64 Jool?
• Can you enable debug logging? We need to know what Jool is thinking in the 4->6 direction.

[bkuebler]

Hi,

i am try to answer your questions:

• Client is any IPv6 only host connected via IPv6 tunnel via Wireguard to the Jool Box (in the diargram) also Known as Gateway. As endpoint in the public internet i used our famous github.com. Because it don't have IPv6 😂
• All the size numbers are from my wireguard capture of full package size. I used ping -6 -s 1500 it generates a package of 1508 ... and i used only icmp here for demonstration. It was easier to capture instead of capture a lot of packages while i am doing a curl against github.com via dns64 and nat64. No difference if icmp or tcp if package size is bigger than 1480.

It is in NAT64 mode not jool_siit

I can you provide the 2 pcap files from my capture if this helps. (From Client and Gateway / Jool Server).

I need some time to establish a new test, because i need to ramp up a new box for testing, due to the fact enable debug mode for jool in this environment will flood my logs.

[bkuebler]

Here are the output of one of those packages from jool debug

Jool NAT64/9f6ae240/default: ===============================================
Jool NAT64/9f6ae240/default: Packet: 140.82.x.x->217.160.x.x
Jool NAT64/9f6ae240/default: ICMPv4 type:0 code:0 id:64060
Jool NAT64/9f6ae240/default: Step 1: Determining the Incoming Tuple
Jool NAT64/9f6ae240/default: Tuple: 140.82.x.x#64060 -> 217.160.x.x#64060 [ICMP]
Jool NAT64/9f6ae240/default: Done step 1.
Jool NAT64/9f6ae240/default: Step 2: Filtering and Updating
Jool NAT64/9f6ae240/default: BIB entry: [2001:678:1180:X::X#4, 217.160.x.x#64060, ICMP]
Jool NAT64/9f6ae240/default: Session entry: [2001:678:1180:X::X#4, 2001:678:1180:X::X:7903#4, 217.160.x.x#64060, 140.82.x.x#64060, ICMP]
Jool NAT64/9f6ae240/default: Done: Step 2.
Jool NAT64/9f6ae240/default: Step 3: Computing the Outgoing Tuple
Jool NAT64/9f6ae240/default: Tuple: 2001:678:1180:X::X:7903#4 -> 2001:678:1180:X::X#4 [ICMP]
Jool NAT64/9f6ae240/default: Done step 3.
Jool NAT64/9f6ae240/default: Step 4: Translating the Packet
Jool NAT64/9f6ae240/default: Routing: 2001:678:1180::X:7903->2001:678:1180:X::X
Jool NAT64/9f6ae240/default: Packet routed via device 'wg0'.
Jool NAT64/9f6ae240/default: Done step 4.
Jool NAT64/9f6ae240/default: Sending packet.
Jool NAT64/9f6ae240/default: dst_output() returned errcode -90.
Jool: Dropping packet.
Jool NAT64/9f6ae240/default: ===============================================

[ydahhrk]

Here are the output of one of those packages from jool debug

This might be a different problem; it's a silent packet drop. The packet probably never arrived to wg0. It seems to contradict the notion that it's sending big fragments or returning ICMP errors.

Hmm....

• Echo reply was generated from Gateway with payload size of 1508 and packet size of 1568

How do you know this? It's what the Wireguard interface capture says?

Maybe there's some GRO going on between Jool and wg0...

I need some time to establish a new test, because i need to ramp up a new box for testing, due to the fact enable debug mode for jool in this environment will flood my logs.

So you have a test environment? I'd like to compile a custom version of Jool to print additional information. Would this be possible?

[bkuebler]

Yes would be possible to use a custom version here for debugging no problem.

[ydahhrk]

It's done; the code is in branch issue442. Let's see what exactly Jool is trying to send. Sample output:

...
[ 8959.253118] Jool NAT64/f10dae80/default: Sending packet.
[ 8959.253123] Jool NAT64/f10dae80/default:   len:104 data_len:0
[ 8959.253130] Jool NAT64/f10dae80/default:   nr_frags:0 gso_size:0 gso_segs:0 gso_type:0
[ 8959.253141] Jool NAT64/f10dae80/default: Success.