ATC24: Fast (Trapless) Kernel Probes Everywhere

Environment setup

This artifact uses Docker to setup the environment. Please make sure Docker is installed before move on. The QEMU script also requires KVM to be available on the host machine.

First, clone the repo:

git clone [email protected]:hardos-ebpf-fuzzing/atc24-uno-kprobe.git

Then, build the Docker image:

cd atc24-uno-kprobe
docker build . -t 'kprobe'

This step may take a while, as it builds our modified llvm together with clang and lld. Also note that LLVM toolchain compilation generally takes large amount of resources, especially memory, so the build process may get killed on system with low memory. This Docker build is tested on a Linux system with 32GB memory.

Build and benchmark the unoptimized kernel

Enter the Docker container first:

# working dir should be /atc24/ upon entry
docker run -it --device=/dev/kvm kprobe bash

Switch to the kernel directory:

cd linux

Build the kernel with the unmodified distro LLVM toolchain:

make LLVM=1 olddefconfig
make LLVM=1 -j`nproc`

Build the userspace benchmark program:

make -C ..

Boot the kernel with QEMU:

../q

The QEMU script uses overlayfs to create a mirroring of host file system, so upon entering the VM, the current directory is still the kernel directory.

Register the kprobe onto a test system call we created:

insmod ./samples/kprobes/kprobe_example.ko

This step registers a kprobe onto the instruction we crafted in the kprobe_bench_test_func function. This instruction is not optimizable by current Linux. The kprobe_bench_test_func measures the time it takes to execute the instruction (together with all the kprobe operations) using the RDTSC and RDTSCP instructions, which returns the time in terms of processor cycles. The elaped time is then sent to userspace via the return value of our kprobe_bench system call.

Run the userspace benchmark program to benchmark the unoptimized kprobe performance:

../test

When done, use Control-D to exit the VM

Build and benchmark the optimized kernel

The implementation details of the LLVM pass can be found under llvm-project/llvm/lib/Target/X86/X86KprobeOpt.cpp; the kernel side support code is under linux/arch/x86/kernel/kprobes/nop-opt.

Again, enter the Docker container

docker run -it --device=/dev/kvm kprobe bash

Add our modified LLVM to PATH:

export PATH=`realpath ./llvm-project/build/bin`:$PATH

Configure the kernel to enable uno-kprobe optimization:

cd linux
./scripts/config -e KPROBES_NOP_OPT
make LLVM=1 olddefconfig

Build the kernel again

make LLVM=1 -j`nproc`

Build the userspace benchmark program:

make -C ..

Boot the kernel with QEMU:

../q

Register the kprobe onto a test system call we created:

insmod ./samples/kprobes/kprobe_example.ko

Run the userspace benchmark program to benchmark the optimized kprobe performance:

../test

The unoptimized and optimized result from QEMU will likely be different from the results reported in the paper on baremetal machines, but the performance improvement should still be significant.

Modified files in detail

llvm-project

llvm/lib/Target/X86/CMakeLists.txt
llvm/lib/Target/X86/X86.h
llvm/lib/Target/X86/X86AsmPrinter.h
llvm/lib/Target/X86/X86InstrCompiler.td
llvm/lib/Target/X86/X86KprobeOpt.cpp
llvm/lib/Target/X86/X86MCInstLower.cpp
llvm/lib/Target/X86/X86TargetMachine.cpp

linux

arch/Kconfig
arch/x86/Kconfig
arch/x86/include/asm/kprobes.h
arch/x86/kernel/alternative.c
arch/x86/kernel/kprobe_bench.c
arch/x86/kernel/kprobes/Makefile
arch/x86/kernel/kprobes/common.h
arch/x86/kernel/kprobes/core.c
arch/x86/kernel/kprobes/nop-opt/Makefile
arch/x86/kernel/kprobes/nop-opt/nop-opt.c
arch/x86/kernel/kprobes/nop-opt/nop-opt_64.S
arch/x86/kernel/kprobes/opt.c
include/linux/kprobes.h
kernel/kprobes.c