Memory leak on heavy read load.

[haraldnh]

I'm running latest release (linux-6.7.y-1) on a card here, and while it runs fine when idling along, once I start pushing more data towards the card it starts leaking memory. When copying in data to the machine at around 5Gbps (which is what the disk array can handle), it consumes kernel memory at about 25% of received volume. Stopping the transfer halts the leak, but nothing is reclaimed at any point. And once it runs all out of memory, the oom killer will of course take down everything.

It's been way too long since I worked with network drivers, so I don't even know where to start looking. I would think about a tail overwrite of a ring buffer or something similar, but that's just a wild guess.

[DatPat]

It would help to know if this leak also occurs in the send-path and which kernel version you're on.

[haraldnh]

It's in the receive path only.

$ uname -a
Linux vmh3 6.8.12-2-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-2 (2024-09-05T10:03Z) x86_64 GNU/Linux

$ dmesg | grep tn40xx
[ 17.471638] tn40xx: Tehuti Network Driver from https://github.com/acooks/tn40xx-driver, linux-6.7.y-1
[ 17.471649] tn40xx: Supported phys : QT2025 TLK10232 AQR105 MUSTANG
[ 17.472139] tn40xx 0000:0b:00.0: enabling device (0140 -> 0142)
[ 17.472468] tn40xx 0000:0b:00.0: srom 0x0 HWver 16 build 0 lane# 4 max_pl 0x1 mrrs 0x5
[ 17.603586] tn40xx 0000:0b:00.0: SVID PHY type 3; MDIO scan Found 2
[ 17.603594] tn40xx 0000:0b:00.0: PHY detected ID=43A400 - QT2025 10Gbps SFP+
[ 19.365221] tn40xx 0000:0b:00.0: QT2025 FW version 2.0.3.3 module type 0x2
[ 19.469121] tn40xx 0000:0b:00.0 eth0: fw 0xe
[ 19.469152] tn40xx 0000:0b:00.0 eth0: Port A
[ 19.469361] tn40xx 0000:0b:00.0: 1 1fc9:4024:1fc9:3015
[ 19.469706] tn40xx: detected 1 cards, 1 loaded

The machine is an old HPE DL380gen9 with dual E5-2680v4 and 320G ram, talking to a Mikrotik CRS 312 on a pair of Finisair SFPs.

[m7a]

I encountered the same issue yesterday and found that the following patch seems to much improve the situation here:

diff --git a/tn40.c b/tn40.c
index 02686bc..046c3c5 100644
--- a/tn40.c
+++ b/tn40.c
@@ -1604,8 +1604,10 @@ static int bdx_rx_alloc_page(struct bdx_priv *priv, struct bdx_page *bdx_page)
 	    alloc_pages(priv->rx_page_table.gfp_mask,
 			priv->rx_page_table.page_order);
 	if (likely(bdx_page->page != NULL)) {
-		get_page(bdx_page->page);
-		bdx_page->ref_count = 1;
+		if (bdx_page->ref_count == 0) {
+			get_page(bdx_page->page);
+			bdx_page->ref_count = 1;
+		}
 		bdx_page->dma =
 		    dma_map_page(&priv->pdev->dev, bdx_page->page, 0L,
 				 priv->rx_page_table.page_size,

I am not sure if this is correct yet. Some more testing is needed.

Edit: DO NOT USE THIS patch and see below

[haraldnh]

Looking through the code that fix feels quite wrong, it's allocating a new page which should call get_page and reset the ref_count. On the other hand, you got me started digging in a reasonable place, and I think a more likely fix is:

diff --git a/tn40.c b/tn40.c
index 719e25d..ae3ed6e 100644
--- a/tn40.c
+++ b/tn40.c
@@ -1760,7 +1760,7 @@ static struct bdx_page *bdx_rx_get_page(struct bdx_priv *priv)
                        netdev_dbg(priv->ndev,
                                   "Replacing - loops %d nFrees %d\n", nLoops,
                                   priv->rx_page_table.nFrees);
-                       bdx_rx_free_bdx_page(priv, firstPage);
+                       bdx_rx_free_page(priv, firstPage);
                        if (bdx_rx_alloc_page(priv, firstPage) == 0) {
                                rPage = firstPage;
                                list_del((struct list_head *)rPage);

I.e. when it re-allocates the page due to load, it currently forgets to handle the ref_count of the old page when it's freed.

I'm testing things in a slightly smaller box now, with a slower disk array, but things look sane so far. No visible memory leaks during the period I managed to push 6Gbit/s into the interface.

[m7a]

Looking through the code that fix feels quite wrong, it's allocating a new page which should call get_page and reset the ref_count.

OK. My initial intuition upon adding some printks around the allocating and freeing operations showed that this was called while ref_count was already 1 and I thought there might be some discrepancy between the logged ref_count = 1 and the actual refcount being incremented to 2 or such. However, since I then “experimentally” put the if to bypass this and this had a "positive impact" to my own surprise really, it was quite likely that this was not the best solution. Thank you very much for taking the time to fix it for real.

I'm testing things in a slightly smaller box now, with a slower disk array, but things look sane so far. No visible memory leaks during the period I managed to push 6Gbit/s into the interface.

I can confirm that your patch works here, too even with my “synthetic” high-throughput test case: pv /dev/zero | nc TARGET PORT on sender and nc -l PORT | pv > /dev/null on receiver side gives full 10G throughput for testing.

Unfortunately here, with either patch, the system still runs unstable wrt. the network driver: I still got lock-ups without any clue (its no longer an out of memory condition as far as I can tell and IIUC it is also independent of which of the two patches I use). My cursory attempts at debugging this have failed as the issue only occurs on the production system (which I don't want to crash really). Hence no further debugging is possible from my side right now. In any case it may be an entirely different issue and the memory leak may very well be resolved by your patch.

[haraldnh]

I've had the system running on this card now for a week, and it looks fairly solid so far. In the meantime I've lost the supercap for the raid controller, so that's become the bottleneck at the moment, but it looks reasonably robust.

With respect to the other lockup, it's hard to look without more info. I've seen switches that lock up when pushing the pps rate high enough, SFPs can crash and so on. If you have any description on what you think you do when it locks up, I can try to provoke my system. But just running traffic seem quite solid now, both on send and receive.

[m7a]

With respect to the other lockup, it's hard to look without more info. I've seen switches that lock up when pushing the pps rate high enough, SFPs can crash and so on. If you have any description on what you think you do when it locks up, I can try to provoke my system. But just running traffic seem quite solid now, both on send and receive.

Well thanks for the offer! Right now I think I need to make it reproducible on some sort of test system otherwise no debugging is possible at all 😄. In the problematic case it was the Edimax EN-9320TX-E directly connected (via Ethernet cable) to an Intel X550 card. I have since replaced the EN-9320TX-E with an Intel X540-T2 and this has improved the immediate situation. However, I do like come back to the issue and had started to play around with the “other” tn40xx driver (https://github.com/torvalds/linux/blob/master/drivers/net/ethernet/tehuti/tn40.c) but right now there are other tasks so it may take me a very long time to progress on it.

In any case I believe its unrelated to this issue here and I believe that your patch is definitely worth adding to the tn40xx as is even if it did not solve all of my particular issues 😄