Unable to run Packetmill + devirtualize code

[vaastav]

Hi,

I am trying to run/follow-along with the packetmill/click-devirtualize example from the README.packetmill file.
However, when I run the configuration with embedclick, I see the following error:

requirement 'clickdv_IJJ0j6MOZMboBjnR0fNtla' not available

When I look into the userlevel directory, I also see that the clickdv*.cc and clickdv*.hh files are absent.

Could you point out the right way of running packetmill and click-devirtualize.

[tbarbette]

Hi Vaastav!

I guess the first step is to get it working without X-Change and without the LLVM passes. So if you just install clang and do this :

./configure --disable-linuxmodule --enable-userlevel --enable-user-multithread --enable-etherswitch --disable-dynamic-linking --enable-local --enable-dpdk --enable-research --enable-flow --disable-task-stats --enable-cpu-load --prefix $(pwd)/build/ --enable-intel-cpu CXX="clang++ -fno-access-control" CC="clang" CXXFLAGS="-std=gnu++14 -O3" --disable-bound-port-transfer --enable-dpdk-pool --disable-dpdk-packet
make
sudo make uninstall
sudo make install

Then create a sample Click config file, let's name it "test.click":

FromDPDKDevice(0) -> EtherMirror -> ToDPDKDevice(0)

and run sudo bin/packetmill --dpdk -- test.click ? What do you get?

[vaastav]

That seems to run and compile without any issues.
This also seems to work with the click configuration I had.

I was running into issues with the following set of commands (from the README.packetmill.md file) instead:

sudo bin/click-devirtualize CONFIG  > package.uo
ar x package.uo config
cd userlevel
sudo ../bin/click-mkmindriver -V -C .. -p embed --ship -u ../package.uo
make embedclick MINDRIVER=embed STATIC=0

sudo userlevel/embedclick --dpdk -l 0-35 -n 6 -w 0000:17:00.0 -v -- config

If I wanted to look at the modified/devirtualised version of the source code from Packetmill, how would I do that?

[vaastav]

While I have gotten it to successfully run, the IR optimizations do not seem to be applied.
I see the following as part of the output:

Couldn't find embedclick.0.5.precodegen.bc
Skipping IR optimization

[tbarbette]

As part of the packetmill pipeline, the files click_dv* are left hanging in the userlevel folder. The click_dv* is the transformed code, before LLVM passes though.

Did you compile FastClick with the LLVM headers available? Maybe you need LLVM 10. The output at the end of ./configure will tell you.

[tbarbette]

I updated https://github.com/tbarbette/fastclick/blob/main/README.packetmill.md and just run successfully the passes with LLVM 14 right now. Using the packetmill automatic toolchain. Note to enable LLVM you need to use the second configure line given in the file. The one above does not extract the LLVM IR.

[vaastav]

Hi @tbarbette , thanks for the response. I am able to generate the .bc files as well as the click_dv* files.
However, when I try running packetmill, I see the error saying "No LLVM Module found". THis is coming from the place where the packetmill binary tries to load the llvm IR module from the .bc file.

For context, I am using llvm-16 with -std=c++17 and not llvm-14.

[tbarbette]

My system does not allow installing 16 but with 17 with theses command :

./configure --disable-linuxmodule --enable-userlevel --enable-user-multithread --enable-etherswitch --disable-dynamic-linking --enable-local --enable-research --disable-task-stats --enable-flow --enable-cpu-load --prefix $(pwd)/build/ --enable-intel-cpu --enable-dpdk-pool --enable-rand-align CXX="clang++ -flto -fno-access-control" CC="clang -flto" CXXFLAGS="-std=gnu++14 -O3" LDFLAGS="-flto -fuse-ld=lld -Wl,-plugin-opt=save-temps" RANLIB="/bin/true" LD="ld.lld" READELF="llvm-readelf" AR="llvm-ar" --disable-bound-port-transfer --enable-dpdk --enable-dpdk-pool --disable-dpdk-packet --disable-dpdk-softqueue
make clean
make -j16

It actually gives me a lot of error because of API change. Did you try to re-configure (and it worked), then clean to ensure everything has been rebuilt and make again without any error?

[vaastav]

I made a fresh clone of the repo to try this from scratch
The configure command I ran was the following:

./configure --disable-linuxmodule --enable-userlevel --enable-user-multithread --enable-etherswitch --disable-dynamic-linking --enable-local --enable-research --disable-task-stats --enable-flow --enable-cpu-load --prefix $(pwd)/build/ --enable-intel-cpu --enable-dpdk-pool --enable-rand-align CXX="clang++-16 -flto -fno-access-control" CC="clang-16 -flto" CXXFLAGS="-std=c++17 -O3" LDFLAGS="-flto -fuse-ld=lld-16 -Wl,-plugin-opt=save-temps" RANLIB="/bin/true" LD="ld.lld-16" READELF="llvm-readelf-16" AR="llvm-ar-16" --disable-bound-port-transfer --enable-dpdk --enable-dpdk-pool --disable-dpdk-packet
make clean
make -j16

The fastclick was configured correctly and I was able to build everything without any errors.
However, when I ran packetmill, I still get the following error:

The rest of the execution works as expected and the unoptimized version of the config is launche.

[vaastav]

For context, the error comes from this specific line: https://github.com/tbarbette/fastclick/blob/main/lib/llvmutils.cc#L792

I have tried running the llvm-dis tool on embedclick.0.5.precodegen.bc file and that seems to work and produce a valid .ll file which means that the generated .bc file does infact have a valid LLVM module.

[vaastav]

Ok I realized that the No LLVM module found line was being triggered because of mismatch in the llvm versions (include directive was picking up llvm-14 but linker was using llvm-16). After fixing the inconsistency, I think I am at the point where I am now seeing compilation errors in llvmutils.cc because of the API change.

So, am I correct to assume that I must either switch to llvm-14 or upgrade the API myself to use llvm-16? (I am happy to submit a PR if you think this would be useful in the future)

[vaastav]

I was able to fix all compilation errors with the following set of changes and configure commands

./configure --disable-linuxmodule --enable-userlevel --enable-user-multithread --enable-etherswitch --disable-dynamic-linking --enable-local --enable-research --disable-task-stats --enable-flow --enable-cpu-load --prefix $(pwd)/build/ --enable-intel-cpu --enable-dpdk-pool --enable-rand-align CXX="clang++-16 -flto -fno-access-control" CC="clang-16 -flto" CXXFLAGS="-std=c++17 -O3 -I/usr/local/include -I/usr/include/llvm-16" LDFLAGS="-flto -fuse-ld=lld-16 -Wl,-plugin-opt=save-temps -lLLVM-16" RANLIB="/bin/true" LD="ld.lld-16" READELF="llvm-readelf-16" AR="llvm-ar-16" LLVMCONFIG="llvm-config-16" --disable-bound-port-transfer --enable-dpdk --enable-dpdk-pool --disable-dpdk-packet

To lib/llvmutils.cc, I needed to make the following changes:

• Add #include "llvm/IR/Instructions.h" to resolve llvm::GetElementPtrInst

To packetmill.cc, I needed to make the following changes:

• Change from opt to opt-16 when optimizing the binary.

However, now I run into a different issue when I try to run packetmill:

For context, when I run regular click, this is the output I see:

[tbarbette]

Somehow the drivers are not loaded with PacketMill. Probably LTO has removed the mlx5 driver. However we did use mlx5 so I'm not sure what happens here. You can see no NICs are detected by the EAL. Can you try to make with V=1 to check for the command line and maybe force some of the drivers and PCI stuff to be taken in with -l,--whole-archive ?

[vaastav]

Using the --whole-archive fixed the issue and packetmill was able to execute.

However, now I am getting a segfault whenever the optimized embedclick-opt gets a workload.
The segfault is happening in the LinearIPLookup::PushBatch function and it only gets triggered with embedclick-opt but not with the non-optimized embedclick

I tried to triage it to which reordering was causing the issue and this is what I found:

1. If I remove the reordering of AllAnno and Packet structs from the handPick function in lib/llvmutils.cc then the segfault does not happen and all things work normally.
2. If I remove the reordering of AllAnno struct but keep the reordering of Packet struct then there is no segfault but the router stops processing packets after ~2 seconds.
3. If I remove the reordering of the Packet struct but keep the reordering of AllAnno struct then there is a segfault as soon as it gets a workload.

Do you have any idea what could be causing these segfaults? Is this potentially a packetmill implementation issue?

[tbarbette]

We tried those passes with a few configurations, but I don't remember exactly which elements were tested and worked.

Fundamentally, this optimization is a bit of a hack. There might be a missed pointer rearrangement. Definitely, if you dereference a pointer and try to access some part of the structure with a manual offset, the reordering pass will not work.

Can you check where the segfault happened with GDB (and compiled with -g). Maybe the missed access will be obvious...

That's a limit of the technique. The goal of PacketMill passes is more to show that proper language constructs could be beneficial for NFV. That's one reason we published at ASPLOS too. Doing a complete and clean version of this, language-based, is another story. And language is not our core business. One could imagine some kind of "virtual struct" that behaves like a struct but is, on the standard, re-arranged at compile time according to some PGO, for instance.

We're thinking about creating a Rust wrapper for FastClick here at UCLouvain, which would allow us to do this reordering using a safer construct. But that's far from done.

[vaastav]

The bug seems to be happening here