Metron branch is broken

[gkatsikas]

Even the simplest FastClick app is broken in the Metron branch.
Issues occur with conf/metron/metron-dispatcher-flow.click when launching secondary processes.

sudo gdb --args bin/click --dpdk -w 0000:03:00.0 -- conf/dpdk/dpdk-bounce.click

GNU gdb (Ubuntu 8.1-0ubuntu3.2) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from bin/click...done.
(gdb) r

Starting program: /home/katsikas/nfv/projects/fastclick/bin/click --dpdk -w 0000:03:00.0 -- conf/dpdk/dpdk-bounce.click
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
EAL: Detected 16 lcore(s)
EAL: Detected 2 NUMA nodes
[New Thread 0x7ffff0199700 (LWP 10006)]
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
[New Thread 0x7fffef998700 (LWP 10007)]
EAL: Selected IOVA mode 'PA'
EAL: Probing VFIO support...
EAL: VFIO support initialized
[New Thread 0x7fffef197700 (LWP 10008)]
[New Thread 0x7fffee996700 (LWP 10009)]
[New Thread 0x7fffee195700 (LWP 10010)]
[New Thread 0x7fffed994700 (LWP 10011)]
[New Thread 0x7fffed193700 (LWP 10012)]
[New Thread 0x7fffec992700 (LWP 10013)]
[New Thread 0x7fffec191700 (LWP 10014)]
[New Thread 0x7fffeb990700 (LWP 10015)]
[New Thread 0x7fffeb18f700 (LWP 10016)]
[New Thread 0x7fffea98e700 (LWP 10017)]
[New Thread 0x7fffea18d700 (LWP 10018)]
[New Thread 0x7fffe998c700 (LWP 10019)]
[New Thread 0x7fffe918b700 (LWP 10020)]
[New Thread 0x7fffe898a700 (LWP 10021)]
[New Thread 0x7fffe8189700 (LWP 10022)]
EAL: PCI device 0000:03:00.0 on NUMA socket 0
EAL: probe driver: 15b3:1017 net_mlx5
Initializing flow parser...
Initializing DPDK
Ingress traffic on port 0 is not restricted anymore to the defined flow rules
deleted virtual method called
terminate called without an active exception

Thread 1 "click" received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1 0x00007ffff5ced801 in __GI_abort () at abort.c:79
#2 0x00007ffff66e0957 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3 0x00007ffff66e6ae6 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4 0x00007ffff66e6b21 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5 0x00007ffff66e791f in __cxa_deleted_virtual () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6 0x00005555560aa48a in Router::initialize (this=, errh=0x555556de0f90) at ../lib/router.cc:1451
#7 0x00005555560444db in parse_configuration (text=..., text_is_expr=, hotswap=, errh=0x555556de0f90) at click.cc:404
#8 0x00005555556fb58e in main (argc=, argv=) at click.cc:739

[tbarbette]

Which binutils? Which GCC? NSLab racks? I would bet on binutils, those developers are cow boys^^

[gkatsikas]

nslrack06-07-08 (Ubuntu 18.04.4, kernel 4.15.0-91-generic)
gcc: 7.5
binutils: 2.30
DPDK: 20.02 (also failed with older versions)

Is there a known issue with binutils?
I will try on other racks with different versions

[tbarbette]

Yes, with 2.30 the code will crash with Xeon Skylake and higher, because they did an incorrect AVX512 optimization. With 2.34 we observed a problem similar to this one where some code was optimized out but actually still called... Reverting to 2.32 worked in that case :p

But then maybe it's different here. Will take a look.

[gkatsikas]

Racks 06-08 are Haswell-based; probably this problems holds for this architecture too.

[gkatsikas]

Also, I noticed something which is not related to the bug but caught my attention.
The configuration flag --enable-cpu-load is not recognized by configure.in anymore. Did you change anything that slept my attention or is it a problematic merge that we should roll-back?

[gkatsikas]

Interestingly, on rack14 (Skylake) the dpdk-bounce works fine, but metron is still problematic when spawning secondary processes (try_slave() method):

EAL: PCI device 0000:17:00.0 on NUMA socket 0
EAL: probe driver: 15b3:1017 net_mlx5
Device 0000:17:00.0 is not driven by the primary process
net_mlx5: can not attach rte ethdev
net_mlx5: probe of PCI device 0000:17:00.0 aborted after encountering an error: Cannot allocate memory
EAL: Requested device 0000:17:00.0 cannot be used
Continuing initialization...
Successful initialization!

[tbarbette]

--enable-cpu-load suffered a bad merge for sure.

I'm finishing something and then will look at it.

[tbarbette]

For me it works. Maybe you should recompile both DPDK and Click, cleaning before from the same machine?

[gkatsikas]

Did you also try Metron with a Mellanox NIC? Which machine did you use?

[tbarbette]

I just tried to launch (Mellanox yes) and did not get the messages you had. Rack 05

[gkatsikas]

Problem found:

When passing the following configuration to the Metron element:
SLAVE_DPDK_ARGS "-w0000:03:00.0"
one should be careful to omit any space between -w and the PCI ID of the NIC (i.e., 0000:03:00.0)

[gkatsikas]

RSS and VMDq-based service chain deployments crash in run_service_chain() method (Child part, just before or during DPDK initialization). See the output below (RSS-based deployment):

Writing configuration: elementclass MetronSlave {
input[0] -> MarkIPHeader(OFFSET 14) -> filter0 :: IPFilter(allow ((ip ttl >= 2 && ip ttl <= 255)), deny all); filter0 -> IPRewriter(pattern 10.0.0.4 1000-65535 - - 0 0) -> DecIPTTL() -> EtherRewrite(SRC 50:6B:4B:43:88:CA, DST 50:6B:4B:43:8A:DA) -> [0]output; filter0[1] -> Discard;
};

slave :: MetronSlave();

slaveFD0C0 :: FromDPDKDevice(0, QUEUE 0, N_QUEUES 1, MAXTHREADS 1, BURST 32, NUMA false, VERBOSE 99, ACTIVE 1);
StaticThreadSched(slaveFD0C0 0);
slaveFD0C0 -> [0]slave;
slaveFD0C1 :: FromDPDKDevice(0, QUEUE 1, N_QUEUES 1, MAXTHREADS 1, BURST 32, NUMA false, VERBOSE 99, ACTIVE 0);
StaticThreadSched(slaveFD0C1 1);
slaveFD0C1 -> [0]slave;
slaveFD0C2 :: FromDPDKDevice(0, QUEUE 2, N_QUEUES 1, MAXTHREADS 1, BURST 32, NUMA false, VERBOSE 99, ACTIVE 0);
StaticThreadSched(slaveFD0C2 2);
slaveFD0C2 -> [0]slave;
slaveFD0C3 :: FromDPDKDevice(0, QUEUE 3, N_QUEUES 1, MAXTHREADS 1, BURST 32, NUMA false, VERBOSE 99, ACTIVE 0);
StaticThreadSched(slaveFD0C3 3);
slaveFD0C3 -> [0]slave;
slaveFD0C4 :: FromDPDKDevice(0, QUEUE 4, N_QUEUES 1, MAXTHREADS 1, BURST 32, NUMA false, VERBOSE 99, ACTIVE 0);
StaticThreadSched(slaveFD0C4 4);
slaveFD0C4 -> [0]slave;
slaveFD0C5 :: FromDPDKDevice(0, QUEUE 5, N_QUEUES 1, MAXTHREADS 1, BURST 32, NUMA false, VERBOSE 99, ACTIVE 0);
StaticThreadSched(slaveFD0C5 5);
slaveFD0C5 -> [0]slave;
slaveFD0C6 :: FromDPDKDevice(0, QUEUE 6, N_QUEUES 1, MAXTHREADS 1, BURST 32, NUMA false, VERBOSE 99, ACTIVE 0);
StaticThreadSched(slaveFD0C6 6);
slaveFD0C6 -> [0]slave;
slaveFD0C7 :: FromDPDKDevice(0, QUEUE 7, N_QUEUES 1, MAXTHREADS 1, BURST 32, NUMA false, VERBOSE 99, ACTIVE 0);
StaticThreadSched(slaveFD0C7 7);
slaveFD0C7 -> [0]slave;

slaveTD0 :: ExactCPUSwitch();
slaveTD0C0 :: ToDPDKDevice(0, QUEUE 0, VERBOSE 99, MAXQUEUES 1);slaveTD0[0] -> slaveTD0C0;
slaveTD0C1 :: ToDPDKDevice(0, QUEUE 1, VERBOSE 99, MAXQUEUES 1);slaveTD0[1] -> slaveTD0C1;
slaveTD0C2 :: ToDPDKDevice(0, QUEUE 2, VERBOSE 99, MAXQUEUES 1);slaveTD0[2] -> slaveTD0C2;
slaveTD0C3 :: ToDPDKDevice(0, QUEUE 3, VERBOSE 99, MAXQUEUES 1);slaveTD0[3] -> slaveTD0C3;
slaveTD0C4 :: ToDPDKDevice(0, QUEUE 4, VERBOSE 99, MAXQUEUES 1);slaveTD0[4] -> slaveTD0C4;
slaveTD0C5 :: ToDPDKDevice(0, QUEUE 5, VERBOSE 99, MAXQUEUES 1);slaveTD0[5] -> slaveTD0C5;
slaveTD0C6 :: ToDPDKDevice(0, QUEUE 6, VERBOSE 99, MAXQUEUES 1);slaveTD0[6] -> slaveTD0C6;
slaveTD0C7 :: ToDPDKDevice(0, QUEUE 7, VERBOSE 99, MAXQUEUES 1);slaveTD0[7] -> slaveTD0C7;
slave[0] -> slaveTD0;

Initializing flow parser...
:2: While configuring ‘slave/filter0 :: IPFilter’:
pattern 0: warning: relation ‘<= 255’ is always true (range 0-255)
FromDPDKDevice : remove StaticThreadSched to use FastClick's auto-thread assignment
slaveFD0C1: using queues from 1 to 1
slaveFD0C1: Queue 1 handled by th 1
click: ../include/click/vector.hh:291: T& Vector<T, ALIGNMENT>::operator[](Vector<T, ALIGNMENT>::size_type) [with T = QueueDevice::QueueInfo; long unsigned int ALIGNMENT = 64; Vector<T, ALIGNMENT>::size_type = int]: Assertion `(unsigned) i < (unsigned) vm_.n_' failed.
Could not read from control socket: Error 0
Could not launch service chain...
Cannot instantiate service chain with ID e82807f5-b89e-438a-b22d-583448a1542c