Skip to content

research.md

Latest commit

 

History

History
179 lines (151 loc) · 11.1 KB

research.md

File metadata and controls

179 lines (151 loc) · 11.1 KB
Raw

Research Notes: Aya + eBPF for Outbound TCP Observation

Date: 2026-07-23 Source: web research (general-purpose agent), consolidated and verified against docs.rs / GitHub.

1. Aya framework current state

  • Current crates: aya (userspace) and aya-ebpf (kernel-side, formerly aya-bpf), plus aya-log / aya-log-ebpf for logging.
  • As of Aya 0.13, Bpf was renamed to Ebpf and BpfLoader to EbpfLoader. This is now the canonical API (aya::Ebpf, aya::EbpfLoader, confirmed live on docs.rs). Old names are kept as type aliases for a transition period but are deprecated; new code should use Ebpf/EbpfLoader directly.
  • Related 0.13 breaking change: ProbeContext (kprobe/uprobe) split from RetProbeContext (kretprobe/uretprobe). Return-value access on a kretprobe must go through RetProbeContext, not a unified context, relevant since our design uses kretprobes to read the connect() result.
  • Other 0.13 changes to be aware of if consulting older tutorials: Maps API rework, removal of aya::ProgramFd, changed EbpfLoader::set_global signature, BTF types moved into a separate aya-obj crate. Any pre-0.13 example code needs re-verification against current docs.rs.

2. Kernel hook selection

Compared three options for observing outbound TCP connects:

Option Verdict
kprobe+kretprobe on tcp_v4_connect / tcp_v6_connect Chosen
tracepoint syscalls:sys_enter_connect / sys_exit_connect Rejected
tracepoint sock:inet_sock_set_state Rejected

Reasoning:

  • The syscall tracepoint only exposes the raw sockaddr argument passed by userspace: awkward to parse for IPv6/mapped-address cases, and gives no access to the kernel struct sock internals we want (state, established addresses).
  • sock:inet_sock_set_state requires filtering for the TCP_SYN_SENT transition and separately correlating process context; less direct than hooking the connect call itself.
  • The kprobe/kretprobe pair on tcp_v4_connect/tcp_v6_connect is the de facto standard used by bcc/libbpf-tools tcpconnect.bpf.c, bpftrace's tcpconnect, and Aya-based prior art (e.g. mfontanini/sockwho). Pattern: entry kprobe stashes the struct sock * in a hash map keyed by pid_tgid; kretprobe checks the return value (0 == success), reads connection details from the cached sock*, then deletes the map entry.
  • IPv4 and IPv6 are handled by separate kernel functions; both need their own probe attachments; no single unified hook covers both address families.

Explicit tradeoff (documented for the design doc and README): kernel-internal functions like tcp_v4_connect are not part of a stable kernel ABI: names/signatures can change across kernel versions or config. The C/libbpf ecosystem mitigates the field-layout half of this risk with BTF/CO-RE relocations; rustc/bpf-linker emit no CO-RE field relocations, so this project instead performs the equivalent resolution in userspace at startup, parsing the running kernel's BTF and injecting the resolved sock_common offsets into eBPF globals before load (see design.md §5, addendum §7 below, and the README's Known Limitations). This is a deliberate deviation from the contract's general "prefer stable tracepoints" guidance, justified because it's the only practical way to get correct PID + real destination IP/port + correct v4/v6 disambiguation at the connect() call site.

3. Process identification (PID / comm)

  • aya_ebpf::helpers::bpf_get_current_pid_tgid() returns a u64: TGID (process id, matches ps//proc semantics) in the high 32 bits, thread id in the low 32 bits. Use the high bits as "pid" for display.
  • aya_ebpf::helpers::bpf_get_current_comm() returns [u8; 16]: kernel's TASK_COMM_LEN is 16 bytes (including NUL), unchanged historically. All target browser process names fit: firefox (7), chrome (6), chromium (8), brave (5), brave-browser (13), all ≤ 15 chars + NUL. Still worth a code comment since TASK_COMM_LEN silently truncates anything ≥ 16 bytes (relevant if the browser list is ever extended).

4. Event transport: kernel → userspace

  • aya::maps::RingBuf (requires kernel ≥ 5.8) is current best practice over PerfEventArray: single shared ring buffer across CPUs (vs per-CPU), strict event ordering, event-driven wakeup, reserve/commit direct writes, in-kernel drop reporting.
  • Userspace polling pattern: either wrap the ring buffer fd in tokio::io::unix::AsyncFd for async polling, or use a blocking loop with epoll. For this minimal, non-concurrent CLI, a blocking poll loop is sufficient and keeps the dependency tree smaller (no tokio needed).

5. IPv4 / IPv6 field access and byte order

  • Address/port fields read from struct sock.__sk_common:
    • v4: skc_daddr (dest addr), skc_rcv_saddr (src addr)
    • v6: skc_v6_daddr.in6_u.u6_addr32 / skc_v6_rcv_saddr.in6_u.u6_addr32
    • ports: skc_dport (dest port), skc_num (local/src port)
  • Byte order gotcha: skc_dport is in network byte order and must be byte-swapped (u16::from_be / bpf_ntohs equivalent) before display. skc_num (local port) is typically already host byte order in modern kernels: worth a runtime sanity check/comment rather than a blind assumption, since this asymmetry is a common source of bugs in tcpconnect-style tools.

6. Addendum (2026-07-23): comm capture point, found via live testing

Not covered by the original web research, discovered only through empirical testing against the running kernel: bpf_get_current_comm() reports the calling thread's name, and for Chromium-based browsers (Brave, Chrome) the connect() syscall is issued from a dedicated IO thread whose kernel comm is "Chrome_ChildIOThread" (truncated to "Chrome_ChildIOT"), never the browser's own process name. An in-eBPF comm-based filter therefore silently drops every real Chromium event.

This was diagnosed with a standalone bpftrace one-liner run directly against the kernel, independent of this project's code:

sudo bpftrace -e 'kprobe:tcp_v4_connect { printf("%s pid=%d\n", comm, pid); }'

which printed Chrome_ChildIOT pid=6464 while visiting example.com in Brave, confirming the kprobe fires correctly and the bug was isolated to the userspace/eBPF filtering assumption, not probe attachment or the kernel hook choice. Fix: match on /proc/<pid>/comm (the thread-group leader's name) using the already-captured pid, instead of the in-kernel thread comm. See design.md section 3 for the applied fix.

7. Addendum (2026-07-24): runtime BTF offset resolution

Follow-up to the tradeoff note in §2. Since rustc/bpf-linker cannot emit CO-RE field relocations, the fixed-offset fragility was addressed in userspace instead:

  • At startup the CLI loads /sys/kernel/btf/vmlinux via aya::Btf::from_sys_fs() and resolves the byte offsets of skc_daddr, skc_dport, skc_num, and skc_v6_daddr in struct sock_common, then injects them into #[unsafe(no_mangle)] globals in the eBPF program with EbpfLoader::set_global(..., must_exist = true) before load. The globals are read with core::ptr::read_volatile in the probe so LLVM cannot constant-fold the fallback initializers into the instruction stream (verified: the compiled object retains R_BPF_64_64 relocations against all four symbols).
  • aya-obj 0.3 parses BTF but keeps member fields of its types private, so the resolver walks the raw type section of Btf::to_bytes() directly. Two layout subtleties matter: sock_common promotes several fields out of C anonymous structs/unions, which must be followed recursively while accumulating bit offsets, and structs containing bitfields use the kind_flag member encoding (bit offset in the low 24 bits, bitfield width in the high 8), which must be decoded before converting to byte offsets.
  • Failure policy: no BTF on the kernel → warn and fall back to the built-in offsets (the pre-existing behavior); BTF present but sock_common unresolvable → hard error, since the built-in offsets are then equally suspect.
  • A host-side unit test (no root needed, /sys/kernel/btf/vmlinux is world-readable) runs the resolver against the build machine's kernel and asserts skc_daddr == 0, skc_dport == 12, skc_num == 14, skc_v6_daddr ∈ {48, 56} (56 observed on the Fedora x86_64 development machine, matching the previously hand-computed values).

8. Addendum (2026-07-24): QUIC/UDP coverage, verified live

All findings below came from bpftrace runs against the real kernel (Fedora, Linux 7.1.3) with real browser traffic, not from documentation:

  • The unprefixed ip4_datagram_connect wrapper never fires on this kernel. It exists in kallsyms, but tracing it during heavy UDP activity (curl DNS lookups, systemd-resolved, browsers) produced zero hits, while __ip4_datagram_connect caught everything. The original probe attempt on the wrapper looked like "browsers don't use UDP connect": the same class of false negative as the earlier Chrome_ChildIOThread comm bug (§6): always verify which symbol actually fires before concluding the event doesn't happen. __ip6_datagram_connect likewise covers v6, including the ip6_datagram_connect_v6_only wrapper's path.
  • Chromium/Brave QUIC uses connected UDP sockets. With the correct inner symbols traced, Chrome_ChildIOT threads showed steady v4 and v6 UDP connects to port 443 (QUIC) while browsing, plus UDP connects to port 53 from the browser's async DNS resolver.
  • Two distinct classes of dport=0 UDP connect() exist, needing two filters. First: connect(AF_UNSPEC) disconnects (Chromium ThreadPoolForeg/libuv threads). They return 0 ("success") and are filtered at the entry probe by reading uaddr->sa_family. Second, discovered only during the live smoke test because it survives the first filter: Chromium's address sorter connects UDP sockets with sin_port = 0 to each DNS candidate IP (observed as batches of four GitHub Pages addresses) purely so the kernel's route lookup reveals source-address selection, RFC 6724 style. These carry a real destination address, so they pass the AF_UNSPEC check and briefly printed as bogus UDP -> IP:0 events. Since no packet can ever flow to port 0, the exit probe drops UDP events whose resolved skc_dport is 0.
  • http3.is's on-page verdict is unreliable for this purpose: it reported "HTTP/3 was not used" (it checks for stale h3-29/h3-27 drafts, and first visits negotiate over TCP before Alt-Svc is learned) during the same session in which the kernel trace showed QUIC connects. The kernel-side evidence is authoritative.

9. Sources