Flush returning "Receive: netlink receive: recvmsg: no buffer space available"

[sontumzmdr]

I observed that when we combine multiple messages and call Flush, the Flush may fail (with error "Receive: netlink receive: recvmsg: no buffer space available") if the message payload crosses netlink buffer size.
Currently Flush() doesn't handle it? is application supposed to handle it? is there any API exposed from which application can know if the netlink message size is exceeded?

[stapelberg]

Which git commit version of google/nftables are you using?

I’m asking because @nickgarlis did some work in this area recently, and I’m wondering if you’re reporting behavior based on that, or based on an older version before that.

[sontumzmdr]

I tried using both v0.3 (github.com/google/nftables v0.3.0) and as well latest master (github.com/google/nftables v0.0.0-20220607152305-1f0380f5c76e), both has same behavior, Flush fails if message payload size is big.

[nickgarlis]

@sontumzmdr Could you share the exact message you're sending (ideally a code snippet) ? I could have a look later today.

[sontumzmdr]

@nickgarlis I wrote a test code to simulate the issue. (the rule content of the file is not relevant here; I just put some payloads for test)

code: test.go file

package main

import (
	"encoding/binary"
	"fmt"
	"log"
	"net"
	"os"
	"strconv"

	"github.com/google/nftables"
	"github.com/google/nftables/expr"
	"golang.org/x/sys/unix"
)

func main() {
	// Get rule count from command line argument
	ruleCount := 5 // Default value
	if len(os.Args) > 1 {
		count, err := strconv.Atoi(os.Args[1])
		if err != nil {
			log.Fatalf("Invalid rule count argument: %v. Usage: %s <rule_count>", err, os.Args[0])
		}
		if count < 1 {
			log.Fatalf("Rule count must be at least 1")
		}
		ruleCount = count
	}

	fmt.Printf("Creating %d rules...\n", ruleCount)

	conn, err := nftables.New()
	if err != nil {
		log.Fatalf("Failed to create nftables connection: %v", err)
	}

	// Create table
	table := conn.AddTable(&nftables.Table{
		Family: nftables.TableFamilyINet,
		Name:   "test_custom_chain",
	})

	// Create custom chain
	chain := conn.AddChain(&nftables.Chain{
		Name:     "my_rules",
		Table:    table,
		Type:     nftables.ChainTypeFilter,
		Hooknum:  nftables.ChainHookInput,
		Priority: nftables.ChainPriorityFilter,
	})

	// Create accept verdict chain
	acceptChain := conn.AddChain(&nftables.Chain{
		Name:  "accept",
		Table: table,
	})

	// Add accept rule to accept chain
	conn.AddRule(&nftables.Rule{
		Table: table,
		Chain: acceptChain,
		Exprs: []expr.Any{
			&expr.Verdict{
				Kind: expr.VerdictAccept,
			},
		},
	})

	// Define base IP ranges and ports (will be incremented for each rule)
	type ruleSpec struct {
		saddrStart net.IP
		saddrEnd   net.IP
		daddrStart net.IP
		daddrEnd   net.IP
		dportStart uint16
		dportEnd   uint16
	}

	// Add multiple rules based on count
	for i := 0; i < ruleCount; i++ {
		// Increment IP addresses and ports for each rule
		// Use simpler incremental values to avoid overflow
		spec := ruleSpec{
			saddrStart: net.ParseIP(fmt.Sprintf("1111:1234:5678:9abc:def0::%x", i+1)),
			saddrEnd:   net.ParseIP(fmt.Sprintf("1111:4321:8765:cba9:fed:ffff:%x:ffff", i)),
			daddrStart: net.ParseIP(fmt.Sprintf("1170:1234:5678:9abc:def0::%x", i+1)),
			daddrEnd:   net.ParseIP(fmt.Sprintf("1170:4321:8765:cba9:fed:ffff:%x:ffff", i)),
			dportStart: uint16(11309 + i*2),
			dportEnd:   uint16(11310 + i*2),
		}

		// Add rule: ip6 saddr <range> ip6 daddr <range> th dport <range> jump accept
		conn.AddRule(&nftables.Rule{
			Table: table,
			Chain: chain,
			Exprs: []expr.Any{
				// Load IPv6 source address
				&expr.Payload{
					DestRegister: 1,
					Base:         expr.PayloadBaseNetworkHeader,
					Offset:       8,  // IPv6 source address offset
					Len:          16, // IPv6 address length
				},
				// Check if saddr >= saddrStart
				&expr.Cmp{
					Op:       expr.CmpOpGte,
					Register: 1,
					Data:     spec.saddrStart.To16(),
				},
				// Check if saddr <= saddrEnd
				&expr.Cmp{
					Op:       expr.CmpOpLte,
					Register: 1,
					Data:     spec.saddrEnd.To16(),
				},

				// Load IPv6 destination address
				&expr.Payload{
					DestRegister: 1,
					Base:         expr.PayloadBaseNetworkHeader,
					Offset:       24, // IPv6 destination address offset
					Len:          16, // IPv6 address length
				},
				// Check if daddr >= daddrStart
				&expr.Cmp{
					Op:       expr.CmpOpGte,
					Register: 1,
					Data:     spec.daddrStart.To16(),
				},
				// Check if daddr <= daddrEnd
				&expr.Cmp{
					Op:       expr.CmpOpLte,
					Register: 1,
					Data:     spec.daddrEnd.To16(),
				},

				// Load layer 4 protocol
				&expr.Meta{
					Key:      expr.MetaKeyL4PROTO,
					Register: 1,
				},
				// Check if protocol is TCP or UDP (for port matching)
				&expr.Cmp{
					Op:       expr.CmpOpEq,
					Register: 1,
					Data:     []byte{unix.IPPROTO_TCP}, // Change to IPPROTO_UDP if needed
				},

				// Load destination port
				&expr.Payload{
					DestRegister: 1,
					Base:         expr.PayloadBaseTransportHeader,
					Offset:       2, // Destination port offset
					Len:          2, // Port length
				},
				// Check if dport >= dportStart
				&expr.Cmp{
					Op:       expr.CmpOpGte,
					Register: 1,
					Data:     binaryBigEndian(spec.dportStart),
				},
				// Check if dport <= dportEnd
				&expr.Cmp{
					Op:       expr.CmpOpLte,
					Register: 1,
					Data:     binaryBigEndian(spec.dportEnd),
				},

				// Jump to accept chain
				&expr.Verdict{
					Kind:  expr.VerdictJump,
					Chain: "accept",
				},
			},
		})

		fmt.Printf("Added rule %d: ip6 saddr %s-%s ip6 daddr %s-%s th dport %d-%d jump accept\n",
			i+1, spec.saddrStart, spec.saddrEnd, spec.daddrStart, spec.daddrEnd, spec.dportStart, spec.dportEnd)
	}

	// Flush the changes
	if err := conn.Flush(); err != nil {
		log.Fatalf("Failed to flush: %v", err)
	}

	fmt.Printf("\nSuccessfully created custom chain with %d range-based rules\n", ruleCount)
}

func binaryBigEndian(v uint16) []byte {
	b := make([]byte, 2)
	binary.BigEndian.PutUint16(b, v)
	return b
}

If I give count like 50 or 100 it works but when I try with 500 it fails.
root [ /home/sontu ]# ./test 500
....
....
Added rule 500: ip6 saddr 1111:1234:5678:9abc:def0::1f4-1111:4321:8765:cba9:fed:ffff:1f3:ffff ip6 daddr 1170:1234:5678:9abc:def0::1f4-1170:4321:8765:cba9:fed:ffff:1f3:ffff th dport 12307-12308 jump accept
2025/12/18 15:01:42 Failed to flush: SendMessages: netlink send-messages: sendmsg: message too long
root [ /home/sontu ]#

[nickgarlis]

@sontumzmdr I am not seeing an issue when I use the latest version (df1ed79) with 500 or even 2000 rules.

Perhaps it's something unique to your system ? What's the output of uname -a and getconf PAGESIZE ?

It also seems that the error you received from your test is different to the one in the issue title. Did the error message change after upgrading to the latest version ?

[sontumzmdr]

@nickgarlis I am using Mariner, below are the details

root [ /home/sontu ]# uname -a
Linux sontu-vm 5.15.186.1-1.cm2 #1 SMP Tue Jul 29 02:54:43 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
root [ /home/sontu ]#
root [ /home/sontu ]# getconf PAGESIZE
4096
root [ /home/sontu ]#

My application code is little complicated; hence I wrote the test code to simulate the issue, but as you mentioned the error is different here (behavior is same in latest master and as well in 0.3 version).
But the underneath issue is same, when the number of messages is big I hit the issue.
I have seen the Flush() code, I don't see we are doing any message size check and breaking the message. So eventually I am suspecting we will hit the issue (can vary upon system to system).

[nickgarlis]

It seems that you're using a relatively older kernel. I do intend to have a look once I have time but a more immediate fix for you might be to upgrade to a more recent version. If you find that upgrading fixes the problem, it'd be helpful to report back.

[nickgarlis]

@sontumzmdr I've now tested with 5.15.186 and I was not able to reproduce the issue.

Could you remove the print statements, test with strace and paste the output ?

# strace -e trace=setsockopt,getsockopt,sendmsg -s 0 ./test 500

Also, just to make sure... You're running the exact same test as the one pasted above right ? I am asking because the buffer logic is different if you initialize nftables using WithSockOptions as the netlink buffers will not be automatically enlarged.

[sontumzmdr]

@nickgarlis I have removed the print statements from the code. The code I am using is the exact same what I pasted.
Below is the output with strace option.

Working case with count 100:

root [ /home/sontu ]# strace -e trace=setsockopt,getsockopt,sendmsg -s 0 ./test 100
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=388970, si_uid=0} ---
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=388970, si_uid=0} ---
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=388970, si_uid=0} ---
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=388970, si_uid=0} ---
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=388970, si_uid=0} ---
getsockopt(3, SOL_SOCKET, SO_TYPE, [3], [4]) = 0
sendmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[...], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 67548
+++ exited with 0 +++
root [ /home/sontu ]#

Non-working case with count 300 (recvmsg: no buffer space available, what I reported in the heading) :

root [ /home/sontu ]# strace -e trace=setsockopt,getsockopt,sendmsg -s 0 ./test 300
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=388949, si_uid=0} ---
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=388949, si_uid=0} ---
getsockopt(3, SOL_SOCKET, SO_TYPE, [3], [4]) = 0
sendmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[...], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 201948
2025/12/20 05:51:09 Failed to flush: Receive: netlink receive: recvmsg: no buffer space available
+++ exited with 1 +++
root [ /home/sontu ]#

Non-working case with count 500 (sendmsg: message too long) :

root [ /home/sontu ]# strace -e trace=setsockopt,getsockopt,sendmsg -s 0 ./test 500
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=389030, si_uid=0} ---
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=389030, si_uid=0} ---
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=389030, si_uid=0} ---
getsockopt(3, SOL_SOCKET, SO_TYPE, [3], [4]) = 0
--- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=389030, si_uid=0} ---
2025/12/20 05:51:22 Failed to flush: SendMessages: netlink send-messages: sendmsg: message too long
+++ exited with 1 +++
root [ /home/sontu ]#

[luvnft]

Hi Velynne, Would my travel be covered as a speaker? Best, Hahz Terry TEAM LUV LUV NFT [Website: [www.luvnft.com](http://www.luvnft.com/)] [Email: ***@***.*** Sent from [Proton Mail](https://proton.me/mail/home) for iOS.

…

-------- Original Message --------

On Wednesday, 12/17/25 at 11:39 sontumzmdr ***@***.***> wrote: sontumzmdr created an issue [(google/nftables#350)](#350) I observed that when we combine multiple messages and call Flush, the Flush may fail (with error "Receive: netlink receive: recvmsg: no buffer space available") if the message payload crosses netlink buffer size. Currently Flush() doesn't handle it? is application supposed to handle it? is there any API exposed from which application can know if the netlink message size is exceeded? — Reply to this email directly, [view it on GitHub](#350), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/BBNOTBYXRQGJEC6LAEMWG534CGBLDAVCNFSM6AAAAACPKVN34KVHI2DSMVQWIX3LMV43ASLTON2WKOZTG4ZTSNRSHE2DKNA). You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[nickgarlis]

@sontumzmdr

I would have expected to see something like this in your strace output

# Get send buffer size
getsockopt(7, SOL_SOCKET, SO_SNDBUF, [212992], [4]) = 0
# Increase send buffer size
setsockopt(7, SOL_SOCKET, SO_SNDBUFFORCE, [458868], 4) = 0
# Get read buffer size
getsockopt(7, SOL_SOCKET, SO_RCVBUF, [212992], [4]) = 0
# Increase read buffer size
setsockopt(7, SOL_SOCKET, SO_RCVBUFFORCE, [8388608], 4) = 0

which is a result of enlargeWriteBuffer and enlargeReadBuffer.

For some reason the functions appear to not work properly for you ? Perhaps you could add some print statements in there to see what's happening.