Snabb multiple CPU

[norbertn503]

So when i start snabb with this snabb lwaftr run --name test --conf lwaftr.conf & and send significant amount of traffic through it. I observed that only one CPU core is used, and the rest is almost idle. From the top command:

%Cpu0 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu1 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu2 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu3 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu4 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu5 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu6 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu7 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu8 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu9 : 78.7 us, 21.3 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu10 : 0.0 us, 0.3 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu11 : 0.3 us, 0.0 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu12 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu13 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu14 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu15 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st

However when I start it with more CPUs like this snabb lwaftr run --name test --cpu 3,4 --conf lwaftr.conf & nothing really changes, performance not increase.

%Cpu0 : 0.0 us, 0.3 sy, 0.0 ni, 96.0 id, 0.0 wa, 0.0 hi, 3.6 si, 0.0 st
%Cpu1 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu2 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu3 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu4 : 25.0 us, 58.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 17.0 si, 0.0 st
%Cpu5 : 0.0 us, 0.0 sy, 0.0 ni, 90.1 id, 0.0 wa, 0.0 hi, 9.9 si, 0.0 st
%Cpu6 : 0.0 us, 0.0 sy, 0.0 ni, 86.1 id, 0.0 wa, 0.0 hi, 13.9 si, 0.0 st
%Cpu7 : 0.0 us, 0.0 sy, 0.0 ni, 87.5 id, 0.0 wa, 0.0 hi, 12.5 si, 0.0 st
%Cpu8 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu9 : 0.0 us, 0.0 sy, 0.0 ni, 86.6 id, 0.0 wa, 0.0 hi, 13.4 si, 0.0 st
%Cpu10 : 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu11 : 0.0 us, 0.0 sy, 0.0 ni, 90.4 id, 0.0 wa, 0.0 hi, 9.6 si, 0.0 st
%Cpu12 : 0.3 us, 0.0 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu13 : 0.0 us, 0.0 sy, 0.0 ni, 92.0 id, 0.0 wa, 0.0 hi, 8.0 si, 0.0 st
%Cpu14 : 0.0 us, 0.3 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu15 : 0.0 us, 0.0 sy, 0.0 ni, 89.4 id, 0.0 wa, 0.0 hi, 10.6 si, 0.0 st

Is there a way for snabb to use multiple CPU cores thus enhance performance ?

Also snabb process starts and an exe process as well

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20086 root 20 0 779040 24476 7196 R 100.0 0.0 10:33.73 exe
20084 root 20 0 213048 182232 41856 S 0.3 0.1 0:03.06 snabb

I didn't find anything regarding this, do you have any idea what is this?
I'm using debian 11 (bullseye), with two Intel Xeon E5-2630, 8 core per CPU

[eugeneia]

Hi!

Snabb lwAFTR scales via RSS so to use multiple cpu cores make sure you

• use network hardware that supports RSS (should be any NIC that Snabb supports, but IIRC it is possible to run Snabb lwAFTR in software mode without hardware, and I think in that case you are limited to a single CPU core)
• make sure the traffic is diverse enough for RSS to work (e.g., if all traffic has the same 4 tuple then RSS can not work)

That's all I can think of from the top of my head right now.

[lencsegabor]

Hi Max,
Thank you very much for your answer. We use Intel X540 NICs, which support RSS. With the current settings, only the IP addresses are included into the tuple for hashing, which can be enough when at least a few hundred lwB4-s are used, but we can set RSS so that it also includes the port numbers into the tuple for hashing.
Indeed, proper RSS setting can distribute the interrupts of packet arrivals among the CPU cores, and thus it can provide some help, but it does not solve the main issue of how to use multiple workers.
This document https://snabbco.github.io/#multiprocess-operation-core.worker says that Snabb can use multiple workers.
Could you help us how they can be started?

[eugeneia]

Could you share your lwaftr.conf? It might help to understand your setup.

[lencsegabor]

We tested several different configurations, because we do benchmarking with Norbi's new Lw4o6 AFTR tester: https://github.com/norbertn503/Lw-AFTR_tester
In more details, we plan to analyze the scalability of the Snabb AFTR implementation both regarding

• the number of CPU cores used

• the number of lwB4-s used
  At the latest, I used the following simple configuration file with 32 lwB4-s:
  root@dut:/home/norbi/snabb# cat lwaftr.conf
  softwire-config {
  instance {
  device "eno1";
  external-device "eno2";
  queue {
  id 0;
  // IPv6 interface
  internal-interface {
  ip "2001:db8:0:1::1"; // Snabb's internal IP
  mac "ec:f4:bb:dc:a6:b8"; // lwAFTR DUT eno1,
  next-hop {
  mac "ec:f4:bb:ef:98:a0"; //lwB4 mac, TESTER eno1
  }
  }

  // IPv4 interface
  external-interface {
    ip "176.0.0.1";          // Snabb's external IP
    mac "ec:f4:bb:dc:a6:ba"; // lwAFTR DUT eno2
    next-hop {
      //ip "192.0.2.2";        // IPv4 Server IP, TESTER eno2
      mac "ec:f4:bb:ef:98:a2";
    }
  }
  

  }
  }

  // IPv6 interface
  internal-interface {
  mtu 1540; // MTU of the IPv6 interface (note: IPv6 encapsulation overhead!)
  }

  // IPv4 interface.
  external-interface {
  mtu 1500; // MTU of the IPv4 interface
  }

  binding-table {
  softwire { ipv4 176.0.0.2; psid 1; b4-ipv6 2001:db8:0:1::2; br-address 2001:db8:2::2; port-set { psid-length 5; }}
  softwire { ipv4 176.0.0.2; psid 2; b4-ipv6 2001:db8:0:1::3; br-address 2001:db8:2::2; port-set { psid-length 5; }}
  softwire { ipv4 176.0.0.2; psid 3; b4-ipv6 2001:db8:0:1::4; br-address 2001:db8:2::2; port-set { psid-length 5; }}
  softwire { ipv4 176.0.0.2; psid 4; b4-ipv6 2001:db8:0:1::5; br-address 2001:db8:2::2; port-set { psid-length 5; }}
  softwire { ipv4 176.0.0.2; psid 5; b4-ipv6 2001:db8:0:1::6; br-address 2001:db8:2::2; port-set { psid-length 5; }}
  softwire { ipv4 176.0.0.2; psid 6; b4-ipv6 2001:db8:0:1::7; br-address 2001:db8:2::2; port-set { psid-length 5; }}
  softwire { ipv4 176.0.0.2; psid 7; b4-ipv6 2001:db8:0:1::8; br-address 2001:db8:2::2; port-set { psid-length 5; }}
  softwire { ipv4 176.0.0.2; psid 8; b4-ipv6 2001:db8:0:1::9; br-address 2001:db8:2::2; port-set { psid-length 5; }}
  softwire { ipv4 176.0.0.2; psid 9; b4-ipv6 2001:db8:0:1::a; br-address 2001:db8:2::2; port-set { psid-length 5; }}
  softwire { ipv4 176.0.0.2; psid 10; b4-ipv6 2001:db8:0:1::b; br-address 2001:db8:2::2; port-set { psid-length 5; }}
  softwire { ipv4 176.0.0.2; psid 11; b4-ipv6 2001:db8:0:1::c; br-address 2001:db8:2::2; port-set { psid-length 5; }}
  softwire { ipv4 176.0.0.2; psid 12; b4-ipv6 2001:db8:0:1::d; br-address 2001:db8:2::2; port-set { psid-length 5; }}
  softwire { ipv4 176.0.0.2; psid 13; b4-ipv6 2001:db8:0:1::e; br-address 2001:db8:2::2; port-set { psid-length 5; }}
  softwire { ipv4 176.0.0.2; psid 14; b4-ipv6 2001:db8:0:1::f; br-address 2001:db8:2::2; port-set { psid-length 5; }}
  softwire { ipv4 176.0.0.2; psid 15; b4-ipv6 2001:db8:0:1::10; br-address 2001:db8:2::2; port-set { psid-length 5; }}
  softwire { ipv4 176.0.0.2; psid 16; b4-ipv6 2001:db8:0:1::11; br-address 2001:db8:2::2; port-set { psid-length 5; }}
  softwire { ipv4 176.0.0.2; psid 17; b4-ipv6 2001:db8:0:1::12; br-address 2001:db8:2::2; port-set { psid-length 5; }}
  softwire { ipv4 176.0.0.2; psid 18; b4-ipv6 2001:db8:0:1::13; br-address 2001:db8:2::2; port-set { psid-length 5; }}
  softwire { ipv4 176.0.0.2; psid 19; b4-ipv6 2001:db8:0:1::14; br-address 2001:db8:2::2; port-set { psid-length 5; }}
  softwire { ipv4 176.0.0.2; psid 20; b4-ipv6 2001:db8:0:1::15; br-address 2001:db8:2::2; port-set { psid-length 5; }}
  softwire { ipv4 176.0.0.2; psid 21; b4-ipv6 2001:db8:0:1::16; br-address 2001:db8:2::2; port-set { psid-length 5; }}
  softwire { ipv4 176.0.0.2; psid 22; b4-ipv6 2001:db8:0:1::17; br-address 2001:db8:2::2; port-set { psid-length 5; }}
  softwire { ipv4 176.0.0.2; psid 23; b4-ipv6 2001:db8:0:1::18; br-address 2001:db8:2::2; port-set { psid-length 5; }}
  softwire { ipv4 176.0.0.2; psid 24; b4-ipv6 2001:db8:0:1::19; br-address 2001:db8:2::2; port-set { psid-length 5; }}
  softwire { ipv4 176.0.0.2; psid 25; b4-ipv6 2001:db8:0:1::1a; br-address 2001:db8:2::2; port-set { psid-length 5; }}
  softwire { ipv4 176.0.0.2; psid 26; b4-ipv6 2001:db8:0:1::1b; br-address 2001:db8:2::2; port-set { psid-length 5; }}
  softwire { ipv4 176.0.0.2; psid 27; b4-ipv6 2001:db8:0:1::1c; br-address 2001:db8:2::2; port-set { psid-length 5; }}
  softwire { ipv4 176.0.0.2; psid 28; b4-ipv6 2001:db8:0:1::1d; br-address 2001:db8:2::2; port-set { psid-length 5; }}
  softwire { ipv4 176.0.0.2; psid 29; b4-ipv6 2001:db8:0:1::1e; br-address 2001:db8:2::2; port-set { psid-length 5; }}
  softwire { ipv4 176.0.0.2; psid 30; b4-ipv6 2001:db8:0:1::1f; br-address 2001:db8:2::2; port-set { psid-length 5; }}
  softwire { ipv4 176.0.0.2; psid 31; b4-ipv6 2001:db8:0:1::20; br-address 2001:db8:2::2; port-set { psid-length 5; }}
  softwire { ipv4 176.0.0.2; psid 32; b4-ipv6 2001:db8:0:1::21; br-address 2001:db8:2::2; port-set { psid-length 5; }}
  }
  }

[eugeneia]

You are using the network card via linux interfaces:

device "eno1";
external-device "eno2";

I.e., you are using the Linux drivers, not Snabb's drivers. This works for testing purposes but is very slow and not feature complete compared to using Snabb's native drivers.
To really use Snabb lwAFTR you need to use:

device "01:00.0";
external-device "01:00.1";

where 01:00.0, 01:00.1 are PCI addresses for the ports of a supported network card.

Intel X540 should work as far as I am aware, so you can find the pci addresses using lspci | grep X540 and try to use the NIC natively.

[lencsegabor]

The PCI addresses are exactly the same as your guessed:
root@dut:~# lspci | grep X540
01:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 03)
01:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 03)

However, when I modified the lwaftr.conf file, killed Snabb, and strated it again, it crashed:

root@dut:/home/norbi/snabb# snabb lwaftr run --name test --cpu 2,4 --conf lwaftr.conf &
[1] 10531
root@dut:/home/norbi/snabb# lwaftr.conf: loading compiled configuration from lwaftr.o
lwaftr.conf: compiled configuration is out of date; recompiling.
lwaftr.conf: loading source configuration
lwaftr.conf: wrote compiled configuration lwaftr.o
Bound main process to NUMA node: 0 (CPU 0)
Binding data-plane PID 10537 to CPU 2.
file not found: core/main.lua: No such file or directory
file not found: core/app.lua: No such file or directory
file not found: core/app.lua: No such file or directory
file not found: core/main.lua: No such file or directory
line not found
core/main.lua:26: unsupported intel nic

Stack Traceback

(1) Lua function '?' at file 'core/main.lua:172' (best guess)
Local variables:
(*temporary) = string: "core/main.lua:26: unsupported intel nic"
(2) global C function 'error'
(3) Lua global 'assert' at file 'core/main.lua:26'
Local variables:
= nil
(4) Lua method 'new' at file 'apps/intel_mp/intel_mp.lua:421'
Local variables:
(*temporary) = table: 0x7f304e662fc8 {rss_tab:function: 0x7f304e6652b8, new:function: 0x7f304e664460, rss_tab_build:function: 0x7f304e6652e8 (more...)}
(*temporary) = table: 0x7f304e66cc50 {rate_limit:0, vmdq:true, pciaddr:01:00.0, mac_loopback:false, poolnum:0 (more...)}
(*temporary) = table: 0x7f304e597410 {shm_root:/intel-mp/01:00.0/, pciaddress:01:00.0, rate_limit:0, mtu:9014 (more...)}
(*temporary) = string: "0x8086"
(*temporary) = string: "0x1528"
(*temporary) = nil
(5) Lua function '?' at file 'core/app.lua:413' (best guess)
Local variables:
/ = string: "b4sideNic"
(*temporary) = table: 0x7f304e662fc8 {rss_tab:function: 0x7f304e6652b8, new:function: 0x7f304e664460, rss_tab_build:function: 0x7f304e6652e8 (more...)}
(*temporary) = table: 0x7f304e66cc50 {rate_limit:0, vmdq:true, pciaddr:01:00.0, mac_loopback:false, poolnum:0 (more...)}
(6) Lua field 'apply_config_actions' at file 'core/app.lua:454'
Local variables:
(*temporary) = table: 0x7f304e5e3e98 {1:table: 0x7f304ea13f78, 2:table: 0x7f304e6eb040, 3:table: 0x7f304e6f8b20 (more...)}
(*temporary) = table: 0x7f304e5e49d8 {unlink_output:function: 0x7f304e5e4a50, stop_app:function: 0x7f304e5e4c30 (more...)}
(*temporary) = Lua function '?' (defined at line 343 of chunk core/app.lua)
(*temporary) = C function: builtin#6
(*temporary) = table: 0x7f304e5e3e98 {1:table: 0x7f304ea13f78, 2:table: 0x7f304e6eb040, 3:table: 0x7f304e6f8b20 (more...)}
(*temporary) = number: 9
(*temporary) = number: 9
(*temporary) = table: 0x7f304e6b8c80 {1:start_app, 2:table: 0x7f304e66c9d0}
(*temporary) = string: "start_app"
(*temporary) = table: 0x7f304e66c9d0 {1:b4sideNic, 2:table: 0x7f304e662fc8, 3:table: 0x7f304e66cc50}
(7) Lua method 'commit_pending_actions' at file 'lib/ptree/worker.lua:82'
Local variables:
(*temporary) = table: 0x7f304e9cbcc0 {no_report:false, channel:table: 0x7f304e9cbd88, duration:inf, period:0.001 (more...)}
(*temporary) = table: 0x7f304e5e3e98 {1:table: 0x7f304ea13f78, 2:table: 0x7f304e6eb040, 3:table: 0x7f304e6f8b20 (more...)}
(*temporary) = boolean: true
(8) Lua method 'handle_actions_from_manager' at file 'lib/ptree/worker.lua:94'
Local variables:
(*temporary) = table: 0x7f304e9cbcc0 {no_report:false, channel:table: 0x7f304e9cbd88, duration:inf, period:0.001 (more...)}
(*temporary) = table: 0x7f304e9cbd88 {ring_buffer:cdata<struct 1763 *>: 0x7f304e8c8000}
(*temporary) = number: 3
(*temporary) = number: 4
(*temporary) = number: 1
(*temporary) = number: 3
(*temporary) = number: 4
(*temporary) = table: 0x7f304e5e3ac8 {1:commit, 2:table: 0x7f304e5e3e38}
(9) Lua field 'main' at file 'lib/ptree/worker.lua:118'
Local variables:
(*temporary) = table: 0x7f304e9cbcc0 {no_report:false, channel:table: 0x7f304e9cbd88, duration:inf, period:0.001 (more...)}
(*temporary) = number: inf
(*temporary) = number: 740662
(10) main chunk of [string "require('lib.scheduling').apply({["profile"]=..."] at line 2
(11) Lua function '?' at file 'core/main.lua:61' (best guess)
Local variables:
(*temporary) = string: "require('lib.scheduling').apply({["profile"]=false,["ingress_drop_monitor"]="flush",["cpu"]=2,["busywait"]=true,["jit_opt"]={["sizemcode"]=256,["maxmcode"]=2048}})
require('lib.ptree.worker').main()"
(*temporary) = Lua function '?' (defined at line 0 of chunk "require('lib.scheduling').apply({["profile"]=..."])
(12) global C function 'xpcall'
(13) main chunk of file 'core/main.lua' at line 257
(14) C function 'require'
(15) global C function 'pcall'
(16) main chunk of file 'core/startup.lua' at line 3
(17) global C function 'require'
(18) main chunk of [string "require "core.startup""] at line 1

2025-11-04 11:45:18: WARN: Worker 01:00.0/0 (pid 10537) crashed with status 1!
2025-11-04 11:45:18: WARN: Too many worker crashes, exiting!
Error while running fiber: core/shm.lua:37: shm open error (10537/apps/reassemblerv4/dtime.counter):No such file or directory

[1]+ Exit 1 snabb lwaftr run --name test --cpu 2,4 --conf lwaftr.conf

The first few lines of lwaftr.conf are:
softwire-config {
instance {
device "01:00.0";
external-device "01:00.1";
queue {

I read that it wrote "unsupported intel nic", but I doubt if the highly common X540 is really unsupported.

[eugeneia]

It might be unsupported, at the very least it is untested! :)

You can try adding the device id and see if it works (as far as I can tell the X540 should be a derivative of the 82599 chip which we support.)

Try adding a line here https://github.com/snabbco/snabb/blob/master/src/apps/intel_mp/intel_mp.lua#L334

[0x1528] = { registers = "82599ES", driver = Intel82599, max_q = 16 }

I got the device ID 0x1528 from https://github.com/snabbco/snabb/blob/master/src/lib/hardware/pci.lua#L82 which being listed there gives me some hope that it could work. (The intel driver was rewritten at some point, so the reason why—if it works—this device id is not enabled yet is because nobody has tested this NIC with the new driver since.

(These days everyone is using Mellanox Connect-X cards with Snabb, hence we never added Intel drivers for cards newer than the 82599 chip, but 82599, i210, and i350 are tested and should work still!)

[lencsegabor]

Your solution seems to work. I added the line, recompiled Snabb, and then it started without problem:
root@dut:/home/norbi/snabb# snabb lwaftr run --name test --cpu 2,4 --conf lwaftr.conf &
[1] 13913
root@dut:/home/norbi/snabb# lwaftr.conf: loading compiled configuration from lwaftr.o
lwaftr.conf: compiled configuration is up to date.
Bound main process to NUMA node: 0 (CPU 0)
Binding data-plane PID 13919 to CPU 2.

However, when I started the same test that I used yesterday, no frames arrive back to the Tester.

To check the situation, I wanted to start the test with the old configuration, but it failed:

root@dut:/home/norbi/snabb# lwaftr-with-kernel.conf: loading source configuration
lwaftr-with-kernel.conf: wrote compiled configuration lwaftr-with-kernel.o
Bound main process to NUMA node: 0 (CPU 0)
file not found: core/main.lua: No such file or directory
file not found: program/lwaftr/run/run.lua: No such file or directory
file not found: core/main.lua: No such file or directory
core/main.lua:26: No such device: eno2

Stack Traceback

(1) Lua function '?' at file 'core/main.lua:172' (best guess)
Local variables:
(*temporary) = string: "core/main.lua:26: No such device: eno2"
(2) global C function 'error'
(3) Lua global 'assert' at file 'core/main.lua:26'
Local variables:
= nil
(4) Lua field 'device_info' at file 'lib/hardware/pci.lua:41'
Local variables:
(*temporary) = string: "eno2"
(*temporary) = table: 0x7feff161df98 {}
(*temporary) = string: "/sys/bus/pci/devices/eno2"
(5) Lua upvalue '' at file 'program/lwaftr/setup.lua:380'
Local variables:
(*temporary) = table: 0x7feff17b7bd0 {links:table: 0x7feff1268810, apps:table: 0x7feff2638960}
(*temporary) = table: 0x7feff16a8300 {softwire_config:table: 0x7feff16c5fd8, alarm_notification:true}
(*temporary) = string: "inetNic"
(*temporary) = string: "b4sideNic"
(*temporary) = nil
(*temporary) = string: "eno1"
(*temporary) = number: 0
(*temporary) = table: 0x7feff1525c50 {internal_interface:table: 0x7feff14ea880, external_interface:table: 0x7feff16d7190 (more...)}
(*temporary) = string: "eno2"
(6) Lua field 'setup_fn' at file 'program/lwaftr/setup.lua:873'
Local variables:
K = table: 0x7feff1452a28 {softwire_config:table: 0x7feff1452ab8}
(*temporary) = table: 0x7feff151d030 {}
(*temporary) = C function: next
(*temporary) = table: 0x7feff151d078 {eno1/0:table: 0x7feff16a8300}
(*temporary) = userdata: NULL
(*temporary) = string: "eno1/0"
(*temporary) = table: 0x7feff16a8300 {softwire_config:table: 0x7feff16c5fd8, alarm_notification:true}
(*temporary) = table: 0x7feff17b7bd0 {links:table: 0x7feff1268810, apps:table: 0x7feff2638960}
(7) Lua method 'set_initial_configuration' at file 'lib/ptree/ptree.lua:171'
Local variables:
(*temporary) = table: 0x7feff1368520 {setup_fn:function: 0x7feff1367c40, peers:table: 0x7feff1374bb0, current_in_place_dependencies:table: 0x7feff14c8490 (more...)}
(*temporary) = table: 0x7feff1452a28 {softwire_config:table: 0x7feff1452ab8}
(8) Lua field 'ptree_manager' at file 'lib/ptree/ptree.lua:124'
Local variables:
b = table: 0x7feff1367d00 {setup_fn:function: 0x7feff1367c40, worker_default_scheduling:table: 0x7feff1b70078 (more...)}
(*temporary) = table: 0x7feff1367d48 {setup_fn:function: 0x7feff1367c40, worker_default_scheduling:table: 0x7feff1b70078 (more...)}
(*temporary) = table: 0x7feff1368520 {setup_fn:function: 0x7feff1367c40, peers:table: 0x7feff1374bb0, current_in_place_dependencies:table: 0x7feff14c8490 (more...)}
(9) Lua field 'run' at file 'program/lwaftr/run/run.lua:197'
Local variables:
(*temporary) = table: 0x7feff2602988 {1:--name, 2:test, 3:--cpu, 4:2,4, 5:--conf, 6:lwaftr-with-kernel.conf}
(*temporary) = table: 0x7feff1b6fff8 {name:test, verbosity:0}
(*temporary) = table: 0x7feff1b70078 {profile:false, ingress_drop_monitor:flush}
(*temporary) = string: "lwaftr-with-kernel.conf"
(*temporary) = nil
(*temporary) = nil
(*temporary) = table: 0x7feff1452a28 {softwire_config:table: 0x7feff1452ab8}
(*temporary) = Lua function '?' (defined at line 157 of chunk program/lwaftr/run/run.lua)
(*temporary) = table: 0x7feff1367b88 {worker_default_scheduling:table: 0x7feff1b70078}
(10) Lua field 'run' at file 'program/lwaftr/lwaftr.lua:25'
Local variables:
(*temporary) = table: 0x7feff2602988 {1:--name, 2:test, 3:--cpu, 4:2,4, 5:--conf, 6:lwaftr-with-kernel.conf}
(*temporary) = string: "run"
(*temporary) = string: "program.lwaftr.run.run"
(11) Lua function '?' at file 'core/main.lua:70' (best guess)
Local variables:
(*temporary) = string: "lwaftr"
(*temporary) = table: 0x7feff2602988 {1:--name, 2:test, 3:--cpu, 4:2,4, 5:--conf, 6:lwaftr-with-kernel.conf}
(12) global C function 'xpcall'
(13) main chunk of file 'core/main.lua' at line 257
(14) C function 'require'
(15) global C function 'pcall'
(16) main chunk of file 'core/startup.lua' at line 3
(17) global C function 'require'
(18) main chunk of [string "require "core.startup""] at line 1

[2]+ Exit 1 snabb lwaftr run --name test --cpu 2,4 --conf lwaftr-with-kernel.conf

And indeed, the eno1 and eno2 interfaces "disapperard" for the Linux kernel. We use DPDK, so I checked them as follows:
root@dut:/home/norbi/snabb# dpdk-devbind.py --status

Network devices using kernel driver

0000:07:00.0 'I350 Gigabit Network Connection 1521' if=eno3 drv=igb unused=
0000:07:00.1 'I350 Gigabit Network Connection 1521' if=eno4 drv=igb unused= Active

Other Network devices

0000:01:00.0 'Ethernet Controller 10-Gigabit X540-AT2 1528' unused=ixgbe
0000:01:00.1 'Ethernet Controller 10-Gigabit X540-AT2 1528' unused=ixgbe

After bounding the interfaces back to the ixgbe driver, the kernel can "see" them, and Snabb can also use them through the kernel:

root@dut:/home/norbi/snabb# snabb lwaftr run --name test --cpu 2,4 --conf lwaftr-with-kernel.conf &
[1] 14633
root@dut:/home/norbi/snabb# lwaftr-with-kernel.conf: loading compiled configuration from lwaftr-with-kernel.o
lwaftr-with-kernel.conf: compiled configuration is up to date.
Bound main process to NUMA node: 0 (CPU 0)
Binding data-plane PID 14635 to CPU 2.

Then the test works all right. It sends bidirectional traffic, 80,000 fps per direction, 60s duration.

root@tester:~/Lw-AFTR_tester# ./build/lw4o6_tester 84 80000 60 2000 2 2
Number of lwB4s read from lwB4Data.conf: 31
READ CMD STARTED
READ CMD ENDED
0,2,4,6,8
0,2,4,6,8
EAL: Detected CPU lcores: 16
EAL: Detected NUMA nodes: 2
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No free 2048 kB hugepages reported on node 0
EAL: No free 2048 kB hugepages reported on node 1
EAL: Probe PCI driver: net_ixgbe (8086:1528) device: 0000:01:00.0 (socket 0)
EAL: Probe PCI driver: net_ixgbe (8086:1528) device: 0000:01:00.1 (socket 0)
TELEMETRY: No legacy callbacks, legacy socket not created
Info: Left port and Left Sender CPU core belong to the same NUMA node: 0
Info: Right port and Right Receiver CPU core belong to the same NUMA node: 0
Info: Right port and Right Sender CPU core belong to the same NUMA node: 0
Info: Left port and Left Receiver CPU core belong to the same NUMA node: 0
Info: Testing started.
Info: forward sender's sending took 59.9999875700 seconds.
forward frames sent: 4800000
Info: reverse sender's sending took 59.9999875667 seconds.
reverse frames sent: 4800000
reverse frames received: 4800000
forward frames received: 4800000
Info: Test finished.

All frames arrived back. :-)

Now I only change the running mode of Snabb:

root@dut:/home/norbi/snabb# lwaftr.conf: loading compiled configuration from lwaftr.o
lwaftr.conf: compiled configuration is up to date.
Bound main process to NUMA node: 0 (CPU 0)
Binding data-plane PID 14649 to CPU 2.

Then I run exactly the same test, with the very same command line parameters and configuration files:

root@tester:~/Lw-AFTR_tester# ./build/lw4o6_tester 84 80000 60 2000 2 2
Number of lwB4s read from lwB4Data.conf: 31
READ CMD STARTED
READ CMD ENDED
0,2,4,6,8
0,2,4,6,8
EAL: Detected CPU lcores: 16
EAL: Detected NUMA nodes: 2
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No free 2048 kB hugepages reported on node 0
EAL: No free 2048 kB hugepages reported on node 1
EAL: Probe PCI driver: net_ixgbe (8086:1528) device: 0000:01:00.0 (socket 0)
EAL: Probe PCI driver: net_ixgbe (8086:1528) device: 0000:01:00.1 (socket 0)
TELEMETRY: No legacy callbacks, legacy socket not created
Info: Left port and Left Sender CPU core belong to the same NUMA node: 0
Info: Right port and Right Receiver CPU core belong to the same NUMA node: 0
Info: Right port and Right Sender CPU core belong to the same NUMA node: 0
Info: Left port and Left Receiver CPU core belong to the same NUMA node: 0
Info: Testing started.
Info: forward sender's sending took 59.9999875733 seconds.
forward frames sent: 4800000
Info: reverse sender's sending took 59.9999875733 seconds.
reverse frames sent: 4800000
forward frames received: 0
reverse frames received: 0
Info: Test finished.

But no frames arrives back.

What else do I need to change in the lwaftr.conf file?

[eugeneia]

To debug this is might be useful to inspect the output of

./snabb config get-state test /

while snabb lwaftr is running and dropping (or forwarding) packets.

It should count dropped packets in descriptive statistics.

[lencsegabor]

Unfortunately, it did not report anything:
root@dut:/home/norbi/snabb# snabb config get-state test /
br-instances {
binding {
bind-instance {
name test;
traffic-stat {
active-softwire-num 0;
discontinuity-time 2025-11-04T13:55:28Z+01:00;
dropped-icmpv4-bytes 0;
dropped-icmpv4-packets 0;
dropped-ipv4-bytes 0;
dropped-ipv4-fragments 0;
dropped-ipv4-packets 0;
dropped-ipv6-bytes 0;
dropped-ipv6-packets 0;
hairpin-ipv4-bytes 0;
hairpin-ipv4-packets 0;
ipv6-fragments-bytes-reassembled 0;
ipv6-fragments-reassembled 0;
rcvd-ipv4-bytes 0;
rcvd-ipv4-packets 0;
rcvd-ipv6-bytes 0;
rcvd-ipv6-packets 0;
sent-ipv4-bytes 0;
sent-ipv4-packets 0;
sent-ipv6-bytes 0;
sent-ipv6-packets 0;
}
}
}
}

It is possible that packets did not even reach Snabb? If so, then why?
What do you think?

[eugeneia]

The driver might just not work properly for this card. I can only speculate without having access. There is also ./snabb shm /var/run/snabb/... for querying lower level statistics counter. The PCI driver should expose the hardware counters of the NIC there.

I remembered that I also at some point added XDP support to snabb lwatr, so if you run lwaftr with --xdp and and your kernel+drivers support it you should be able to use multiple RSS queues even with using linux network interfaces.

By the way I just noticed that your config only specifies a single queue {}, if I recall correctly you need to configure one queue for each core. See https://github.com/snabbco/snabb/blob/master/src/lib/yang/snabb-softwire-v3.yang#L694

[lencsegabor]

Hi Max,
I added another queue by simply copying the first one and increasing its id as follows:
root@dut:/home/norbi/snabb# head lwaftr-with-kernel-2q.conf -n 48

softwire-config {
  instance {
    device "eno1";
    external-device "eno2";
    queue {
      id 0;
      // IPv6 interface
      internal-interface {
        ip "2001:db8:0:1::1";     // Snabb's internal IP
        mac "ec:f4:bb:dc:a6:b8";  // lwAFTR DUT eno1,
        next-hop {
          mac "ec:f4:bb:ef:98:a0";  //lwB4 mac, TESTER eno1
        }
      }

      // IPv4 interface
      external-interface {
        ip "176.0.0.1";          // Snabb's external IP
        mac "ec:f4:bb:dc:a6:ba"; // lwAFTR DUT eno2
        next-hop {
          //ip "192.0.2.2";        // IPv4 Server IP, TESTER eno2
          mac "ec:f4:bb:ef:98:a2";
        }
      }
    }
    queue {
      id 1;
      // IPv6 interface
      internal-interface {
        ip "2001:db8:0:1::1";     // Snabb's internal IP
        mac "ec:f4:bb:dc:a6:b8";  // lwAFTR DUT eno1,
        next-hop {
          mac "ec:f4:bb:ef:98:a0";  //lwB4 mac, TESTER eno1
        }
      }

      // IPv4 interface
      external-interface {
        ip "176.0.0.1";          // Snabb's external IP
        mac "ec:f4:bb:dc:a6:ba"; // lwAFTR DUT eno2
        next-hop {
          //ip "192.0.2.2";        // IPv4 Server IP, TESTER eno2
          mac "ec:f4:bb:ef:98:a2";
        }
      }
    }

  }

Is it the right way to do so?

And I gave one more CPU core to Snabb:
root@dut:/home/norbi/snabb# snabb lwaftr run --name test --cpu 2,4,6 --conf lwaftr-with-kernel-2q.conf &
[1] 19361
root@dut:/home/norbi/snabb# lwaftr-with-kernel-2q.conf: loading compiled configuration from lwaftr-with-kernel-2q.o
lwaftr-with-kernel-2q.conf: compiled configuration is up to date.
Bound main process to NUMA node: 0 (CPU 0)
Binding data-plane PID 19363 to CPU 2.
Binding data-plane PID 19365 to CPU 4.

In 'top', now I can see two 'exe'-s running.

However, when I did a test, I had a surprise. It seems that more packets arrive than sent:

root@tester:~/Lw-AFTR_tester# ./build/lw4o6_tester 84 80000 1 2000 2 2
Number of lwB4s read from lwB4Data.conf: 31
READ CMD STARTED
READ CMD ENDED
0,2,4,6,8
0,2,4,6,8
EAL: Detected CPU lcores: 16
EAL: Detected NUMA nodes: 2
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No free 2048 kB hugepages reported on node 0
EAL: No free 2048 kB hugepages reported on node 1
EAL: Probe PCI driver: net_ixgbe (8086:1528) device: 0000:01:00.0 (socket 0)
EAL: Probe PCI driver: net_ixgbe (8086:1528) device: 0000:01:00.1 (socket 0)
TELEMETRY: No legacy callbacks, legacy socket not created
Info: Left port and Left Sender CPU core belong to the same NUMA node: 0
Info: Right port and Right Receiver CPU core belong to the same NUMA node: 0
Info: Right port and Right Sender CPU core belong to the same NUMA node: 0
Info: Left port and Left Receiver CPU core belong to the same NUMA node: 0
Info: Testing started.
Info: reverse sender's sending took 0.9999875733 seconds.
reverse frames sent: 80000
Info: forward sender's sending took 0.9999875683 seconds.
forward frames sent: 80000
forward frames received: 121306
reverse frames received: 121376
Info: Test finished.

Only 80,000 frames were sent in each directions. But more than 120,000 frames arrived back to the tester.

How could it happen?

[lencsegabor]

Hi Max,

I tried to debug the situation. This is what I did:

If I use a single queue, then everything looks OK.

root@dut:/home/norbi/snabb# snabb lwaftr run --name test --cpu 2 --conf lwaftr-with-kernel.conf &
[1] 3097
root@dut:/home/norbi/snabb# lwaftr-with-kernel.conf: loading compiled configuration from lwaftr-with-kernel.o
lwaftr-with-kernel.conf: compiled configuration is up to date.
Bound main process to NUMA node: 0 (CPU 0)
Binding data-plane PID 3099 to CPU 2.

Below I send a single packet per direction and then both packets arrive all right.

root@tester:~/Lw-AFTR_tester# ./build/lw4o6_tester 84 1 1 2000 2 2
Number of lwB4s read from lwB4Data.conf: 31
READ CMD STARTED
READ CMD ENDED
0,2,4,6,8
0,2,4,6,8
EAL: Detected CPU lcores: 16
EAL: Detected NUMA nodes: 2
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No free 2048 kB hugepages reported on node 0
EAL: No free 2048 kB hugepages reported on node 1
EAL: Probe PCI driver: net_ixgbe (8086:1528) device: 0000:01:00.0 (socket 0)
EAL: Probe PCI driver: net_ixgbe (8086:1528) device: 0000:01:00.1 (socket 0)
TELEMETRY: No legacy callbacks, legacy socket not created
Info: Left port and Left Sender CPU core belong to the same NUMA node: 0
Info: Right port and Right Receiver CPU core belong to the same NUMA node: 0
Info: Right port and Right Sender CPU core belong to the same NUMA node: 0
Info: Left port and Left Receiver CPU core belong to the same NUMA node: 0
Info: Testing started.
Info: reverse sender's sending took 0.0000040350 seconds.
reverse frames sent: 1
Info: forward sender's sending took 0.0000040250 seconds.
forward frames sent: 1
forward frames received: 1
reverse frames received: 1
Info: Test finished.

tcpdump shows correct behavior: both packets can be seen at both interfaces:

root@dut:~# tcpdump -i eno1 -n
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eno1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
13:41:31.977605 IP6 fe80::eef4:bbff:fedc:a6b8 > ff02::16: HBH ICMP6, multicast listener report v2, 4 group record(s), length 88
13:41:31.989598 IP6 fe80::eef4:bbff:fedc:a6b8 > ff02::16: HBH ICMP6, multicast listener report v2, 4 group record(s), length 88
13:41:36.135550 IP6 2001:db8:0:1::15 > 2001:db8:2::2: IP 176.0.0.2.42416 > 203.0.113.2.42085: UDP, length 18
13:41:36.138029 IP6 2001:db8:2::2 > 2001:db8:0:1::15: IP 203.0.113.2.42506 > 176.0.0.2.42396: UDP, length 18
^C
4 packets captured
4 packets received by filter
0 packets dropped by kernel

root@dut:~# tcpdump -i eno2 -n
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eno2, link-type EN10MB (Ethernet), snapshot length 262144 bytes
13:41:32.209601 IP6 fe80::eef4:bbff:fedc:a6ba > ff02::16: HBH ICMP6, multicast listener report v2, 4 group record(s), length 88
13:41:32.645599 IP6 fe80::eef4:bbff:fedc:a6ba > ff02::16: HBH ICMP6, multicast listener report v2, 4 group record(s), length 88
13:41:36.135550 IP 203.0.113.2.42506 > 176.0.0.2.42396: UDP, length 18
13:41:36.138960 IP 176.0.0.2.42416 > 203.0.113.2.42085: UDP, length 18
^C
4 packets captured
4 packets received by filter
0 packets dropped by kernel

This is what I expected to happen. :-)

However, when I use two queues, something strange happens:

root@dut:/home/norbi/snabb# snabb lwaftr run --name test --cpu 2,4 --conf lwaftr-with-kernel-2q.conf &
[1] 3115
root@dut:/home/norbi/snabb# lwaftr-with-kernel-2q.conf: loading compiled configuration from lwaftr-with-kernel-2q.o
lwaftr-with-kernel-2q.conf: compiled configuration is up to date.
Bound main process to NUMA node: 0 (CPU 0)
Binding data-plane PID 3117 to CPU 2.
Binding data-plane PID 3119 to CPU 4.

I execute the very same test:

root@tester:~/Lw-AFTR_tester# ./build/lw4o6_tester 84 1 1 2000 2 2
Number of lwB4s read from lwB4Data.conf: 31
READ CMD STARTED
READ CMD ENDED
0,2,4,6,8
0,2,4,6,8
EAL: Detected CPU lcores: 16
EAL: Detected NUMA nodes: 2
EAL: Detected shared linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No free 2048 kB hugepages reported on node 0
EAL: No free 2048 kB hugepages reported on node 1
EAL: Probe PCI driver: net_ixgbe (8086:1528) device: 0000:01:00.0 (socket 0)
EAL: Probe PCI driver: net_ixgbe (8086:1528) device: 0000:01:00.1 (socket 0)
TELEMETRY: No legacy callbacks, legacy socket not created
Info: Left port and Left Sender CPU core belong to the same NUMA node: 0
Info: Right port and Right Receiver CPU core belong to the same NUMA node: 0
Info: Right port and Right Sender CPU core belong to the same NUMA node: 0
Info: Left port and Left Receiver CPU core belong to the same NUMA node: 0
Info: Testing started.
Info: reverse sender's sending took 0.0000041500 seconds.
reverse frames sent: 1
Info: forward sender's sending took 0.0000041050 seconds.
forward frames sent: 1
reverse frames received: 2
forward frames received: 2
Info: Test finished.

It seems that two packets arrived per direction.

root@dut:~# tcpdump -i eno1 -n
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eno1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
13:54:29.881614 IP6 fe80::eef4:bbff:fedc:a6b8 > ff02::16: HBH ICMP6, multicast listener report v2, 4 group record(s), length 88
13:54:29.941598 IP6 fe80::eef4:bbff:fedc:a6b8 > ff02::16: HBH ICMP6, multicast listener report v2, 4 group record(s), length 88
13:54:34.056322 IP6 2001:db8:0:1::1a > 2001:db8:2::2: IP 176.0.0.2.52734 > 203.0.113.2.18861: UDP, length 18
13:54:34.056692 IP6 2001:db8:2::2 > 2001:db8:0:1::1a: IP 203.0.113.2.38917 > 176.0.0.2.51983: UDP, length 18
13:54:34.059032 IP6 2001:db8:2::2 > 2001:db8:0:1::1a: IP 203.0.113.2.38917 > 176.0.0.2.51983: UDP, length 18
13:54:34.063577 IP6 2001:db8:0:1::1a > 2001:db8:2::2: ICMP6, destination unreachable, unknown unreach code (5)
0x0000: 0105 8e6b 0000 0000 6000 0000 002e 04ff
0x0010: 2001 0db8 0002 0000 0000 0000 0000 0002
0x0020: 2001 0db8 0000 0001 0000 0000 0000 001a
0x0030: 4500 002e 0000 0000 0911 c5ba cb00 7102
0x0040: b000 0002 9805 cb0f 001a b329 4944 454e
0x0050: 5449 4659 0001 0203 0405 0607 0809
13:54:34.063791 IP6 2001:db8:2::2 > 2001:db8:0:1::1a: IP 176.0.0.1 > 176.0.0.2: ICMP host 203.0.113.2 unreachable, length 54
13:54:34.064348 IP6 2001:db8:2::2 > 2001:db8:0:1::1a: IP 176.0.0.1 > 176.0.0.2: ICMP host 203.0.113.2 unreachable, length 54
13:54:34.065684 IP6 2001:db8:0:1::1a > 2001:db8:2::2: ICMP6, destination unreachable, unknown unreach code (5)
0x0000: 0105 e1da 0000 0000 6000 0000 004a 04ff
0x0010: 2001 0db8 0002 0000 0000 0000 0000 0002
0x0020: 2001 0db8 0000 0001 0000 0000 0000 001a
0x0030: 4500 004a 0000 0000 fe01 5caf b000 0001
0x0040: b000 0002 0301 a957 0000 0000 4500 002e
0x0050: 0000 0000 0a11 c4ba b000 0002 cb00 7102
0x0060: cdfe 49ad 001a fe92 4944 454e 5449 4659
0x0070: 0001 0203 0405 0607 0809
13:54:34.071295 IP6 2001:db8:0:1::1a > 2001:db8:2::2: ICMP6, destination unreachable, unknown unreach code (5)
0x0000: 0105 8e6b 0000 0000 6000 0000 002e 04ff
0x0010: 2001 0db8 0002 0000 0000 0000 0000 0002
0x0020: 2001 0db8 0000 0001 0000 0000 0000 001a
0x0030: 4500 002e 0000 0000 0911 c5ba cb00 7102
0x0040: b000 0002 9805 cb0f 001a b329 4944 454e
0x0050: 5449 4659 0001 0203 0405 0607 0809
13:54:34.076339 IP6 2001:db8:0:1::1a > 2001:db8:2::2: ICMP6, destination unreachable, unknown unreach code (5)
0x0000: 0105 e1da 0000 0000 6000 0000 004a 04ff
0x0010: 2001 0db8 0002 0000 0000 0000 0000 0002
0x0020: 2001 0db8 0000 0001 0000 0000 0000 001a
0x0030: 4500 004a 0000 0000 fe01 5caf b000 0001
0x0040: b000 0002 0301 a957 0000 0000 4500 002e
0x0050: 0000 0000 0a11 c4ba b000 0002 cb00 7102
0x0060: cdfe 49ad 001a fe92 4944 454e 5449 4659
0x0070: 0001 0203 0405 0607 0809

Above I marked the packet with bold that was duplicated.

root@dut:~# tcpdump -i eno2 -n
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eno2, link-type EN10MB (Ethernet), snapshot length 262144 bytes
13:54:30.113601 IP6 fe80::eef4:bbff:fedc:a6ba > ff02::16: HBH ICMP6, multicast listener report v2, 4 group record(s), length 88
13:54:30.317599 IP6 fe80::eef4:bbff:fedc:a6ba > ff02::16: HBH ICMP6, multicast listener report v2, 4 group record(s), length 88
13:54:34.056322 IP 203.0.113.2.38917 > 176.0.0.2.51983: UDP, length 18
13:54:34.058442 IP 176.0.0.2.52734 > 203.0.113.2.18861: UDP, length 18
13:54:34.059959 IP 176.0.0.2.52734 > 203.0.113.2.18861: UDP, length 18
13:54:34.071439 IP 176.0.0.1 > 203.0.113.2: ICMP host 176.0.0.2 unreachable, length 54
13:54:34.072998 IP 176.0.0.1 > 203.0.113.2: ICMP host 176.0.0.2 unreachable, length 54
13:54:34.073498 IP 176.0.0.1 > 176.0.0.1: ICMP host 176.0.0.2 unreachable, length 82
13:54:34.076750 IP 176.0.0.1 > 176.0.0.1: ICMP host 176.0.0.2 unreachable, length 82

Above I marked the packet with bold that was duplicated.

Thus, it seems to me that the packets do not happen to arrive just into one of the queues, but they are put intto both queues, and perhaps this is why they are duplicated. What is worse, there are several ICMP error messages. (They are not reported by the Tester, because it recognizes the test frames by a pattern located at the required position.)

Therefore, I suspect that the problem is caused by the way I tried to use two queues.

What did I wrong?

How should I use two (or more) queues?

[eugeneia]

Try starting snabb lwaftr with the --xdp flag.

[lencsegabor]

I though that it would work without xdp. Now I added --xdp as follows:

root@dut:/home/norbi/snabb# snabb lwaftr run --name test --cpu 2,4 --xdp --conf lwaftr-with-kernel-2q.conf &
[1] 5227
root@dut:/home/norbi/snabb# lwaftr-with-kernel-2q.conf: loading compiled configuration from lwaftr-with-kernel-2q.o
lwaftr-with-kernel-2q.conf: compiled configuration is up to date.
Configuring internal interface for XDP...
shell: ip link set down dev eno1
file not found: core/main.lua: No such file or directory
file not found: core/main.lua: No such file or directory
file not found: program/lwaftr/setup.lua: No such file or directory
file not found: program/lwaftr/setup.lua: No such file or directory
file not found: lib/yang/list.lua: No such file or directory
file not found: program/lwaftr/setup.lua: No such file or directory
file not found: core/main.lua: No such file or directory
program/lwaftr/setup.lua:236: bad argument #1 to 'format' (number expected, got boolean)

Stack Traceback

(1) Lua function '?' at file 'core/main.lua:172' (best guess)
Local variables:
(*temporary) = string: "program/lwaftr/setup.lua:236: bad argument #1 to 'format' (number expected, got boolean)"
(2) method C function 'format'
(3) Lua upvalue '▒' at file 'program/lwaftr/setup.lua:236'
Local variables:
cmd = string: "ip link set down dev eno1"
status = boolean: true
(*temporary) = Lua function '?' (defined at line 24 of chunk core/main.lua)
(*temporary) = number: 0
(*temporary) = boolean: false
(4) Lua function '?' at file 'program/lwaftr/setup.lua:463' (best guess)
Local variables:
(*temporary) = string: "eno1"
(*temporary) = table: 0x7fbdde616878 {next_hop:table: 0x7fbdde616960, mac:cdata<unsigned char [6]>: 0x7fbdde616e90 (more...)}
(*temporary) = table: 0x7fbdde9dab10 {error_rate_limiting:table: 0x7fbdde9ed9a8, mru:1460, allow_incoming_icmp:true (more...)}
(*temporary) = Lua function '?' (defined at line 476 of chunk program/lwaftr/setup.lua)
(5) Lua field 'xdp_ifsetup' at file 'program/lwaftr/setup.lua:475'
Local variables:
(*temporary) = table: 0x7fbdde9eb348 {softwire_config:table: 0x7fbdde9eb870}
(*temporary) = Lua function '?' (defined at line 1217 of chunk lib/yang/list.lua)
(*temporary) = nil
(*temporary) = string: "eno1"
(*temporary) = string: "eno1"
(*temporary) = table: 0x7fbdde29d1a8 {queue:table: 0x7fbdde61cd08, device:eno1, external_device:eno2}
(*temporary) = string: "eno2"
(*temporary) = table: 0x7fbdde616878 {next_hop:table: 0x7fbdde616960, mac:cdata<unsigned char [6]>: 0x7fbdde616e90 (more...)}
(*temporary) = table: 0x7fbdde617a18 {next_hop:table: 0x7fbdde617b00, mac:cdata<unsigned char [6]>: 0x7fbdde617dc8 (more...)}
(*temporary) = number: 2
(*temporary) = Lua function '?' (defined at line 462 of chunk program/lwaftr/setup.lua)
(6) Lua field 'run' at file 'program/lwaftr/run/run.lua:154'
Local variables:
(*temporary) = table: 0x7fbddf470980 {1:--name, 2:test, 3:--cpu, 4:2,4, 5:--xdp, 6:--conf, 7:lwaftr-with-kernel-2q.conf (more...)}
(*temporary) = table: 0x7fbdde9ddc00 {verbosity:0, name:test, xdp:true}
(*temporary) = table: 0x7fbdde9ddc80 {enable_xdp:table: 0x7fbdde9e2cc8, profile:false, ingress_drop_monitor:flush (more...)}
(*temporary) = string: "lwaftr-with-kernel-2q.conf"
(*temporary) = nil
(*temporary) = nil
(*temporary) = table: 0x7fbdde9eb348 {softwire_config:table: 0x7fbdde9eb870}
(7) Lua field 'run' at file 'program/lwaftr/lwaftr.lua:25'
Local variables:
(*temporary) = table: 0x7fbddf470980 {1:--name, 2:test, 3:--cpu, 4:2,4, 5:--xdp, 6:--conf, 7:lwaftr-with-kernel-2q.conf (more...)}
(*temporary) = string: "run"
(*temporary) = string: "program.lwaftr.run.run"
(8) Lua function '?' at file 'core/main.lua:70' (best guess)
Local variables:
(*temporary) = string: "lwaftr"
(*temporary) = table: 0x7fbddf470980 {1:--name, 2:test, 3:--cpu, 4:2,4, 5:--xdp, 6:--conf, 7:lwaftr-with-kernel-2q.conf (more...)}
(9) global C function 'xpcall'
(10) main chunk of file 'core/main.lua' at line 257
(11) C function 'require'
(12) global C function 'pcall'
(13) main chunk of file 'core/startup.lua' at line 3
(14) global C function 'require'
(15) main chunk of [string "require "core.startup""] at line 1

[1]+ Exit 1 snabb lwaftr run --name test --cpu 2,4 --xdp --conf lwaftr-with-kernel-2q.conf
root@dut:/home/norbi/snabb#

I have no idea why it did not start.

If I remove "--xdp" from the command line, then Snabb starts:

root@dut:/home/norbi/snabb# lwaftr-with-kernel-2q.conf: loading compiled configuration from lwaftr-with-kernel-2q.o
lwaftr-with-kernel-2q.conf: compiled configuration is up to date.
Bound main process to NUMA node: 0 (CPU 0)
Binding data-plane PID 5236 to CPU 2.
Binding data-plane PID 5238 to CPU 4.

What can be the problem?