KCFI-EBPF

Getting Started

Build

This repo contains submodules and needs to be initiated:

git submodule update --init --recursive --progress

Build trace proecess tool:

cd parse_trace
cargo build -r
cd -

The tool requires rust toolchain to build, you can install it with rustup:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Next build LLVM:

mkdir -p llvm-project/build && cd llvm-project/build
cmake -GNinja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DLLVM_ENABLE_PROJECTS="clang;lld" -DLLVM_TARGETS_TO_BUILD=X86 -DLLVM_ENABLE_ASSERTIONS=ON -DCMAKE_C_FLAGS=-pipe -DCMAKE_CXX_FLAGS=-pipe ../llvm
ninja
cd -

The built LLVM binaries (clang, llvm-ar, lld, etc) are under llvm-project/build/bin.

Build kernel and libbpf:

cd linux
export PATH=`realpath ../llvm-project/build/bin`:$PATH
make LLVM=1 oldconfig
make LLVM=1 -j`nproc`
make -C tools/lib/bpf -j`nproc`

Then build the userspace control tools of eKCFI:

make trace trace_kern.o -C ebpf
cd ekcfi_test
cargo build -r
cd -

Finally build our kernel module

cd mod
make LLVM=1
cd -

Obtaining indirect call trace

Boot into the kernel:

# In kernel directory
`realpath ../yifei-q`

The VM console will open if everything works correctly:

root@q:~/linux#

Prepare for trace

insmod ../mod/ekcfi.ko # Inserts our module
../ebpf/trace # Installs the tracing eBPF program
../ekcfi_test/target/release/ekcfi_test -mtrace vmlinux # Patches all nops generated by clang

We will use uname as an example:

uname -a

The result can be read from /sys/kernel/debug/tracing/trace:

cat /sys/kernel/debug/tracing/trace | grep uname > trace_data.txt

Note

1. Due to the indeterministic nature of the scheduler, it is a good idea to run uname a few times to better capture indirect calls in the scheduler
2. To apply on another program, several files needs to be changed because uname is hard-coded at this point. Namely these includes:
   • ebpf/trace_kern.c:9
   • ebpf/policy_kern.c:11
   • ebpf/Makefile:36

Once finished, the VM can be shutdown by pressing control-D. Due to how the overlayfs is mounted in the VM, any changes to the kernel directory will reflect on the host (but changes to other directories will not).

Also, because this research prototype does currently does not cleanup module resource when unloading, VM actually needs to be shutdown after the previous step to avoid problems.

Enforce KCFI based on obtained trace

Build the trace parsing tool

cd parse_trace
cargo build -r
cd -

Build the policy eBPF program

mv linux/trace_data.txt ebpf/
make -C ebpf

The policy_kern.c program implements an example policy: for each indirect call from a particular program (in this case uname) it check whether the call is valid (i.e. whether the caller-callee pair has appeared in the trace). It triggers a kernel panic upon check failure.

Boot the kernel again:

# In kernel directory
`realpath ../yifei-q`

Then run the following to set up KCFI:

insmod ../mod/ekcfi.ko # Inserts our module
../ebpf/trace # Installs the policy eBPF program
../ekcfi_test/target/release/ekcfi_test -mtrace vmlinux # Patches all nops generated by clang

Now execute uname

uname -a

If nothing shows up it is probably correct (since a panic will happen if check fails).

Other notes

• The userspace tools should be able to be easily rewritten in C (now it's Rust).
• There is an ugly hack in the module which relies on privilegged userspace to read out the address of symbols not exported by the kernel. This is because of our design principle to not modify the kernel source code. If we were to implement the framework in-tree, then this hack is not needed at all.