|
| 1 | +# Getting data to user-space |
| 2 | + |
| 3 | +In previous chapters we were logging packets with aya-log. However, what if we |
| 4 | +need to send additional data about the packet or any other type of information |
| 5 | +for the user-space program to utilize? In this chapter, we will explore how to |
| 6 | +leverage perf buffers and define data structures in the *-common* crate to |
| 7 | +transfer data from eBPF program to user-space applications. By doing so, |
| 8 | +user-space programs can access and utilize the transferred data effectively. |
| 9 | + |
| 10 | +!!! example "Source code" |
| 11 | + |
| 12 | + Full code for the example in this chapter is available |
| 13 | + [here](https://github.com/aya-rs/book/tree/main/examples/xdp-perfbuf-custom-data) |
| 14 | + |
| 15 | +## Sharing data |
| 16 | + |
| 17 | +In this chapter, we'll be sending data from the kernel-space to the user-space |
| 18 | +by writing it into a struct and outputting it to user-space. We can achieve this |
| 19 | +by using an eBPF map. There are different types of maps available, but in this |
| 20 | +case, we'll use `PerfEventArray`. |
| 21 | + |
| 22 | +`PerfEventArray` is a collection of per-CPU circular buffers that enable the |
| 23 | +kernel to emit events (defined as custom structs) to user-space. Each CPU has |
| 24 | +its own buffer, and the eBPF program emits an event to the buffer of the CPU |
| 25 | +it's currently running on. The events are out of order, meaning that they arrive |
| 26 | +in the user-space in a different order than they were created and sent from the |
| 27 | +eBPF program. |
| 28 | + |
| 29 | +To gather events from all CPUs, a user-space program needs to spawn a thread for |
| 30 | +each CPU to poll for the events and then iterate over them. |
| 31 | + |
| 32 | +The data structure we'll be using needs to hold an IPv4 address and a port. |
| 33 | + |
| 34 | +```rust linenums="1" title="xdp-perfbuf-custom-data-common/src/lib.rs" |
| 35 | +--8<-- "examples/xdp-perfbuf-custom-data/xdp-perfbuf-custom-data-common/src/lib.rs" |
| 36 | +``` |
| 37 | + |
| 38 | +1. We implement the `aya::Pod` trait for our struct since it is Plain Old Data |
| 39 | +as can be safely converted to a byte-slice and back. |
| 40 | + |
| 41 | +!!! tip "Alignment, padding and verifier errors" |
| 42 | + |
| 43 | + At program load time, the eBPF verifier checks that all the memory used is |
| 44 | + properly initialized. This can be a problem if - to ensure alignment - the |
| 45 | + compiler inserts padding bytes between fields in your types. |
| 46 | + |
| 47 | + **Example:** |
| 48 | + |
| 49 | + ```rust |
| 50 | + #[repr(C)] |
| 51 | + struct SourceInfo { |
| 52 | + source_port: u16, |
| 53 | + source_ip: u32, |
| 54 | + } |
| 55 | + |
| 56 | + let source_port = ...; |
| 57 | + let source_ip = ...; |
| 58 | + let si = SourceInfo { source_port, source_ip }; |
| 59 | + ``` |
| 60 | + |
| 61 | + In the example above, the compiler will insert two extra bytes between the |
| 62 | + struct fields `source_port` and `source_ip` to make sure that `source_ip` is |
| 63 | + correctly aligned to a 4 bytes address (assuming `mem::align_of::<u32>() == |
| 64 | + 4`). Since padding bytes are typically not initialized by the compiler, |
| 65 | + this will result in the infamous `invalid indirect read from stack` verifier |
| 66 | + error. |
| 67 | + |
| 68 | + To avoid the error, you can either manually ensure that all the fields in |
| 69 | + your types are correctly aligned (e.g. by explicitly adding padding or by |
| 70 | + making field types larger to enforce alignment) or use `#[repr(packed)]`. |
| 71 | + Since the latter comes with its own foot-guns and can perform less |
| 72 | + efficiently, explicitly adding padding or tweaking alignment is recommended. |
| 73 | + |
| 74 | + **Solution ensuring alignment using larger types:** |
| 75 | + |
| 76 | + ```rust |
| 77 | + #[repr(C)] |
| 78 | + struct SourceInfo { |
| 79 | + source_port: u32, |
| 80 | + source_ip: u32, |
| 81 | + } |
| 82 | + |
| 83 | + let source_port = ...; |
| 84 | + let source_ip = ...; |
| 85 | + let si = SourceInfo { source_port, source_ip }; |
| 86 | + ``` |
| 87 | + |
| 88 | + **Solution with explicit padding:** |
| 89 | + |
| 90 | + ```rust |
| 91 | + #[repr(C)] |
| 92 | + struct SourceInfo { |
| 93 | + source_port: u16, |
| 94 | + padding: u16, |
| 95 | + source_ip: u32, |
| 96 | + } |
| 97 | + |
| 98 | + let source_port = ...; |
| 99 | + let source_ip = ...; |
| 100 | + let si = SourceInfo { source_port, padding: 0, source_ip }; |
| 101 | + ``` |
| 102 | + |
| 103 | +## Getting packet data from the context and into the map |
| 104 | + |
| 105 | +Our eBPF program code is going to be similar to the one in the previous chapters. |
| 106 | +It's going to get the information about source IP address and port from packet |
| 107 | +headers. |
| 108 | + |
| 109 | +The difference is that once we get the data from headers, instead of logging |
| 110 | +them, we create a `PacketLog` struct and output it to our `PerfEventArray` |
| 111 | + |
| 112 | +The resulting code looks like this: |
| 113 | + |
| 114 | +```rust linenums="1" title="xdp-perfbuf-custom-data-ebpf/src/main.rs" |
| 115 | +--8<-- "examples/xdp-perfbuf-custom-data/xdp-perfbuf-custom-data-ebpf/src/main.rs" |
| 116 | +``` |
| 117 | + |
| 118 | +1. Create our map. |
| 119 | +2. Output the event to the map. |
| 120 | + |
| 121 | +Don't forget to rebuild your eBPF program! |
| 122 | + |
| 123 | +## Reading data |
| 124 | + |
| 125 | +In order to read from the `AsyncPerfEventArray`, we have to call |
| 126 | +`AsyncPerfEventArray::open()` for each online CPU, then we have to poll the file |
| 127 | +descriptor for events. While this is do-able using `PerfEventArray` and `mio` or |
| 128 | +`epoll`, the code is much less easy to follow. Instead, we'll use `tokio`, which |
| 129 | +was added to our template for us. |
| 130 | + |
| 131 | +We'll need to add a dependency on `bytes` to `xdp-log/Cargo.toml` since |
| 132 | +this will make it easier to deal with the chunks of bytes yielded by the |
| 133 | +`AsyncPerfEventArray`. |
| 134 | + |
| 135 | +Here's the code: |
| 136 | + |
| 137 | +```rust linenums="1" title="xdp-perfbuf-custom-data/src/main.rs" |
| 138 | +--8<-- "examples/xdp-perfbuf-custom-data/xdp-perfbuf-custom-data/src/main.rs" |
| 139 | +``` |
| 140 | + |
| 141 | +1. Define our map. |
| 142 | +2. Call `open()` for each online CPU. |
| 143 | +3. Spawn a `tokio::task`. |
| 144 | +4. Create buffers. |
| 145 | +5. Read events in to buffers. |
| 146 | +6. Use `read_unaligned` to read the event data into a `PacketLog`. |
| 147 | +7. Log the packet data to the console. |
| 148 | + |
| 149 | +## Running the program |
| 150 | + |
| 151 | +As before, the interface can be overwritten by providing the interface name as a |
| 152 | +parameter, for example, `RUST_LOG=info cargo xtask run -- --iface wlp2s0`. |
| 153 | + |
| 154 | +```console |
| 155 | +$ RUST_LOG=info cargo xtask run |
| 156 | +[2023-01-25T08:57:41Z INFO xdp_perfbuf_custom_data] SRC IP: 60.235.240.157, SRC_PORT: 443 |
| 157 | +[2023-01-25T08:57:41Z INFO xdp_perfbuf_custom_data] SRC IP: 98.21.76.76, SRC_PORT: 443 |
| 158 | +[2023-01-25T08:57:41Z INFO xdp_perfbuf_custom_data] SRC IP: 95.194.217.172, SRC_PORT: 443 |
| 159 | +[2023-01-25T08:57:41Z INFO xdp_perfbuf_custom_data] SRC IP: 95.194.217.172, SRC_PORT: 443 |
| 160 | +[2023-01-25T08:57:41Z INFO xdp_perfbuf_custom_data] SRC IP: 95.10.251.142, SRC_PORT: 443 |
| 161 | +``` |
0 commit comments