Skip to content

Commit c69999c

Browse files
committed
Add a chapter about logging custom structs with PerfEventArray
Signed-off-by: Michal Rostecki <vadorovsky@gmail.com>
1 parent 54f1609 commit c69999c

22 files changed

Lines changed: 656 additions & 0 deletions

File tree

Lines changed: 161 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,161 @@
1+
# Getting data to user-space
2+
3+
In previous chapters we were logging packets with aya-log. However, what if we
4+
need to send additional data about the packet or any other type of information
5+
for the user-space program to utilize? In this chapter, we will explore how to
6+
leverage perf buffers and define data structures in the *-common* crate to
7+
transfer data from eBPF program to user-space applications. By doing so,
8+
user-space programs can access and utilize the transferred data effectively.
9+
10+
!!! example "Source code"
11+
12+
Full code for the example in this chapter is available
13+
[here](https://github.com/aya-rs/book/tree/main/examples/xdp-perfbuf-custom-data)
14+
15+
## Sharing data
16+
17+
In this chapter, we'll be sending data from the kernel-space to the user-space
18+
by writing it into a struct and outputting it to user-space. We can achieve this
19+
by using an eBPF map. There are different types of maps available, but in this
20+
case, we'll use `PerfEventArray`.
21+
22+
`PerfEventArray` is a collection of per-CPU circular buffers that enable the
23+
kernel to emit events (defined as custom structs) to user-space. Each CPU has
24+
its own buffer, and the eBPF program emits an event to the buffer of the CPU
25+
it's currently running on. The events are out of order, meaning that they arrive
26+
in the user-space in a different order than they were created and sent from the
27+
eBPF program.
28+
29+
To gather events from all CPUs, a user-space program needs to spawn a thread for
30+
each CPU to poll for the events and then iterate over them.
31+
32+
The data structure we'll be using needs to hold an IPv4 address and a port.
33+
34+
```rust linenums="1" title="xdp-perfbuf-custom-data-common/src/lib.rs"
35+
--8<-- "examples/xdp-perfbuf-custom-data/xdp-perfbuf-custom-data-common/src/lib.rs"
36+
```
37+
38+
1. We implement the `aya::Pod` trait for our struct since it is Plain Old Data
39+
as can be safely converted to a byte-slice and back.
40+
41+
!!! tip "Alignment, padding and verifier errors"
42+
43+
At program load time, the eBPF verifier checks that all the memory used is
44+
properly initialized. This can be a problem if - to ensure alignment - the
45+
compiler inserts padding bytes between fields in your types.
46+
47+
**Example:**
48+
49+
```rust
50+
#[repr(C)]
51+
struct SourceInfo {
52+
source_port: u16,
53+
source_ip: u32,
54+
}
55+
56+
let source_port = ...;
57+
let source_ip = ...;
58+
let si = SourceInfo { source_port, source_ip };
59+
```
60+
61+
In the example above, the compiler will insert two extra bytes between the
62+
struct fields `source_port` and `source_ip` to make sure that `source_ip` is
63+
correctly aligned to a 4 bytes address (assuming `mem::align_of::<u32>() ==
64+
4`). Since padding bytes are typically not initialized by the compiler,
65+
this will result in the infamous `invalid indirect read from stack` verifier
66+
error.
67+
68+
To avoid the error, you can either manually ensure that all the fields in
69+
your types are correctly aligned (e.g. by explicitly adding padding or by
70+
making field types larger to enforce alignment) or use `#[repr(packed)]`.
71+
Since the latter comes with its own foot-guns and can perform less
72+
efficiently, explicitly adding padding or tweaking alignment is recommended.
73+
74+
**Solution ensuring alignment using larger types:**
75+
76+
```rust
77+
#[repr(C)]
78+
struct SourceInfo {
79+
source_port: u32,
80+
source_ip: u32,
81+
}
82+
83+
let source_port = ...;
84+
let source_ip = ...;
85+
let si = SourceInfo { source_port, source_ip };
86+
```
87+
88+
**Solution with explicit padding:**
89+
90+
```rust
91+
#[repr(C)]
92+
struct SourceInfo {
93+
source_port: u16,
94+
padding: u16,
95+
source_ip: u32,
96+
}
97+
98+
let source_port = ...;
99+
let source_ip = ...;
100+
let si = SourceInfo { source_port, padding: 0, source_ip };
101+
```
102+
103+
## Getting packet data from the context and into the map
104+
105+
Our eBPF program code is going to be similar to the one in the previous chapters.
106+
It's going to get the information about source IP address and port from packet
107+
headers.
108+
109+
The difference is that once we get the data from headers, instead of logging
110+
them, we create a `PacketLog` struct and output it to our `PerfEventArray`
111+
112+
The resulting code looks like this:
113+
114+
```rust linenums="1" title="xdp-perfbuf-custom-data-ebpf/src/main.rs"
115+
--8<-- "examples/xdp-perfbuf-custom-data/xdp-perfbuf-custom-data-ebpf/src/main.rs"
116+
```
117+
118+
1. Create our map.
119+
2. Output the event to the map.
120+
121+
Don't forget to rebuild your eBPF program!
122+
123+
## Reading data
124+
125+
In order to read from the `AsyncPerfEventArray`, we have to call
126+
`AsyncPerfEventArray::open()` for each online CPU, then we have to poll the file
127+
descriptor for events. While this is do-able using `PerfEventArray` and `mio` or
128+
`epoll`, the code is much less easy to follow. Instead, we'll use `tokio`, which
129+
was added to our template for us.
130+
131+
We'll need to add a dependency on `bytes` to `xdp-log/Cargo.toml` since
132+
this will make it easier to deal with the chunks of bytes yielded by the
133+
`AsyncPerfEventArray`.
134+
135+
Here's the code:
136+
137+
```rust linenums="1" title="xdp-perfbuf-custom-data/src/main.rs"
138+
--8<-- "examples/xdp-perfbuf-custom-data/xdp-perfbuf-custom-data/src/main.rs"
139+
```
140+
141+
1. Define our map.
142+
2. Call `open()` for each online CPU.
143+
3. Spawn a `tokio::task`.
144+
4. Create buffers.
145+
5. Read events in to buffers.
146+
6. Use `read_unaligned` to read the event data into a `PacketLog`.
147+
7. Log the packet data to the console.
148+
149+
## Running the program
150+
151+
As before, the interface can be overwritten by providing the interface name as a
152+
parameter, for example, `RUST_LOG=info cargo xtask run -- --iface wlp2s0`.
153+
154+
```console
155+
$ RUST_LOG=info cargo xtask run
156+
[2023-01-25T08:57:41Z INFO xdp_perfbuf_custom_data] SRC IP: 60.235.240.157, SRC_PORT: 443
157+
[2023-01-25T08:57:41Z INFO xdp_perfbuf_custom_data] SRC IP: 98.21.76.76, SRC_PORT: 443
158+
[2023-01-25T08:57:41Z INFO xdp_perfbuf_custom_data] SRC IP: 95.194.217.172, SRC_PORT: 443
159+
[2023-01-25T08:57:41Z INFO xdp_perfbuf_custom_data] SRC IP: 95.194.217.172, SRC_PORT: 443
160+
[2023-01-25T08:57:41Z INFO xdp_perfbuf_custom_data] SRC IP: 95.10.251.142, SRC_PORT: 443
161+
```
Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
[alias]
2+
xtask = "run --package xtask --"
Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
### https://raw.github.com/github/gitignore/master/Rust.gitignore
2+
3+
# Generated by Cargo
4+
# will have compiled files and executables
5+
debug/
6+
target/
7+
8+
# Remove Cargo.lock from gitignore if creating an executable, leave it for libraries
9+
# More information here https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html
10+
Cargo.lock
11+
12+
# These are backup files generated by rustfmt
13+
**/*.rs.bk
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
{
2+
"rust-analyzer.linkedProjects": ["Cargo.toml", "xdp-perfbuf-custom-data-ebpf/Cargo.toml"]
3+
}
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
{
2+
"rust-analyzer.linkedProjects": ["Cargo.toml", "xdp-perfbuf-custom-data-ebpf/Cargo.toml"]
3+
}
Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
[workspace]
2+
members = ["xdp-perfbuf-custom-data", "xdp-perfbuf-custom-data-common", "xtask"]
Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# xdp-perfbuf-custom-data
2+
3+
## Prerequisites
4+
5+
1. Install a rust stable toolchain: `rustup install stable`
6+
1. Install a rust nightly toolchain: `rustup install nightly`
7+
1. Install bpf-linker: `cargo install bpf-linker`
8+
9+
## Build eBPF
10+
11+
```bash
12+
cargo xtask build-ebpf
13+
```
14+
15+
To perform a release build you can use the `--release` flag.
16+
You may also change the target architecture with the `--target` flag
17+
18+
## Build Userspace
19+
20+
```bash
21+
cargo build
22+
```
23+
24+
## Run
25+
26+
```bash
27+
RUST_LOG=info cargo xtask run
28+
```
Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
[package]
2+
name = "xdp-perfbuf-custom-data-common"
3+
version = "0.1.0"
4+
edition = "2021"
5+
6+
[features]
7+
default = []
8+
user = [ "aya" ]
9+
10+
[dependencies]
11+
aya = { version = ">=0.11", optional=true }
12+
13+
[lib]
14+
path = "src/lib.rs"
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
#![no_std]
2+
3+
#[repr(C)]
4+
#[derive(Clone, Copy)]
5+
pub struct PacketLog {
6+
pub ipv4_address: u32,
7+
pub port: u32,
8+
}
9+
10+
#[cfg(feature = "user")]
11+
unsafe impl aya::Pod for PacketLog {} // (1)
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
[build]
2+
target-dir = "../target"
3+
target = "bpfel-unknown-none"
4+
5+
[unstable]
6+
build-std = ["core"]

0 commit comments

Comments
 (0)