skbx

skbx shows where a packet went inside Linux—and keeps the receipts.

Follow a packet through the flight recorder →

Field guides:

• Trace a website request across your Linux host, ISP, and target
• Debug a Linux packet drop with replayable eBPF evidence

It observes kernel networking functions, TC/XDP programs, packet transformations, tunnels, drops, and selected BPF helper activity with Rust and CO-RE eBPF. Every observation lands in a bounded, replayable evidence stream with stable handles and explicit loss telemetry.

Use it when “the packet disappeared” is not a sufficient incident report.

capture → event:8c6f… → replay → route:21b4… → explain
            evidence          pattern          context

Inspired by pwru, rebuilt around an agent-first contract: deterministic observations, machine-readable capabilities, bounded state, and no invented evidence.

The short version

You need to…	skbx gives you…
See the kernel path of an SKB	BTF-discovered kprobes with exact function evidence
Follow clones, copies, COW, and XDP-to-SKB transitions	Capture-local lineage IDs and explicit match origin
Inspect TC or XDP behavior	Exact program identity, entry/exit pairing, and decoded XDP actions
Filter without guessing field layouts	Target-BTF-checked packet and metadata expressions
Hand evidence to a human or agent	Versioned JSONL, stable handles, deterministic replay, and explain
Know whether the trace is trustworthy	Reserve, recursion, read, decode, enrichment, and output telemetry

First packet

Build prerequisites are listed below. Once skbx is installed:

# 1. Check the host.
skbx doctor --json

# 2. See exactly what would attach—without attaching it.
skbx plan --filter-func 'ip.*' --json

# 3. Capture ten bounded seconds of ICMP evidence.
sudo skbx capture --probe ip_rcv \
  --duration 10 \
  --output trace.jsonl \
  icmp

# 4. Rebuild packet routes without root.
skbx replay trace.jsonl --format json

# 5. Follow any event handle back to its surrounding evidence.
skbx explain trace.jsonl event:<handle>

Want a fast human view instead?

sudo skbx capture --probe tcp_v4_do_rcv \
  --format text \
  --timestamp relative \
  --output-caller \
  --output-tcp-flags \
  --output -

Built for operators and agents

The CLI explains itself before an agent touches the host:

skbx describe --format json  # commands, capabilities, limits, invariants
skbx schema                  # the versioned traceq JSON Schema
skbx doctor --json           # host prerequisites with actionable evidence
skbx plan --json             # deterministic attachment plan

The native engine remains the source of truth. An AI system may explain a captured event, but it cannot manufacture one. Machine output stays on stdout; diagnostics stay on stderr; missing footers and observation loss are explicit.

Install the agent workflow

The repository is also a plugin marketplace for Codex and Claude Code. Both clients install the same use-skbx skill; the plugin contains instructions, not the native binary, credentials, an MCP server, or a background hook.

For Codex:

codex plugin marketplace add https://github.com/copyleftdev/skbx.git
codex plugin add skbx@skbx-tools

Start a new Codex session after installation.

For Claude Code:

claude plugin marketplace add https://github.com/copyleftdev/skbx.git
claude plugin install skbx@skbx-tools

Run /reload-plugins or restart Claude Code after installation. See the agent onboarding guide for the trust boundary, first prompt, manifests, and validation details.

Arc: multi-host mission control

skbx-arc is an experimental, local-first control plane for bounded captures across several Linux sensors. Its Mission Constellation keeps three claims visually distinct: what a sensor observed, what Arc correlated across hosts, and what remains unknown.

cargo run -p skbx-arc -- serve --demo

Then open http://127.0.0.1:7878. The seeded website-timeout mission can be explored without root, BPF attachment, or network access. skbx-agent fixture-once exercises the real polling and artifact-upload protocol with an existing traceq file.

Arc is a lab vertical slice today: state is in memory, the listener is loopback-only by default, and there is no authentication or live remote capture backend. See the Arc architecture and rollout guide for the evidence contract, correlation algorithm, trust boundary, and production gates.

Route dossiers: beyond a line of hops

skbx-route is an experimental sibling CLI that performs bounded, rootless IPv4 UDP route observation and records every TTL as a replayable routeq evidence dossier. It keeps one five-tuple across the run to reduce per-flow load-balancing artifacts, and it never treats a silent router as proof of a drop. Explore the interactive route-dossier field guide for the evidence levels, visibility boundaries, and packet-budget model.

# Install the sibling CLI from the repository.
cargo install --git https://github.com/copyleftdev/skbx --locked skbx-route

# Inspect the exact packet and time budget without resolving or probing.
skbx-route plan example.com --json

# Exercise replay and explanation without touching the network.
skbx-route demo --output demo.routeq.jsonl
skbx-route replay demo.routeq.jsonl --format json

# Run a bounded live observation on Linux without root.
skbx-route trace example.com \
  --max-hops 20 \
  --probes 2 \
  --timeout-ms 500 \
  --max-duration-ms 30000 \
  --output route.routeq.jsonl

# Opt in to passive prefix, origin-ASN, and RPKI context from RIPEstat.
skbx-route enrich route.routeq.jsonl \
  --max-lookups 64 \
  --max-duration-ms 30000 \
  --output route.enriched.routeq.jsonl

The initial vertical slice performs standards-based UDP path observation only. It does not port-scan intermediate routers, enumerate their services, or claim to observe remote forwarding decisions. Passive RIPEstat enrichment is a separate opt-in command with independent HTTPS request and duration budgets. It skips private and special-purpose addresses and sends each remaining public responder address to the RIPEstat Data API; RIPEstat's service terms apply. The dossier contract distinguishes observed, enriched, correlated, candidate, and unknown facts so public routing metadata and future cooperative skbx-sensor evidence cannot blur that boundary.

What it can observe

• BTF-discovered struct sk_buff * arguments in positions 1–5;
• individual kprobes and signature-grouped kprobe-multi attachment;
• base and split BTF from named or all loaded kernel modules;
• IPv4/IPv6, bounded extension chains, TCP/UDP/ICMP, fragments, and TCP flags;
• mark, interface, namespace, MTU, socket fallback, and caller evidence;
• independent outer and inner-L2/inner-L3 libpcap predicates;
• SKB clone/copy/COW lineage and XDP frame-to-SKB correlation;
• stack-associated non-SKB teardown paths through logical free;
• JIT-discovered BPF helper calls and bounded map key/value evidence;
• TC entry observations and paired XDP entry/exit actions;
• BTF-checked SKB/XDP metadata, bitfields, byte-order-aware filters, and bounded boolean expressions;
• atomic sk_buff and skb_shared_info BTF renderings;
• named SKB drop reasons, kernel stacks, and the SKB control buffer;
• exact capture limits, rotation boundaries, and reliability footers.

The full, evidence-backed comparison with pwru lives in the parity matrix. Design invariants and the hot-path model live in the architecture guide.

Useful captures

Follow a marked packet through transformations

sudo skbx capture \
  --filter-mark 0x2a \
  --filter-track-skb \
  --output trace.jsonl

Filter on target-kernel fields

sudo skbx capture --probe ip_rcv \
  --filter-skb-expr \
    'skb->mark = 0b101010 && skb->protocol = 0x0800' \
  --output-skb-metadata 'skb->dev->ifindex' \
  --output trace.jsonl

Observe loaded XDP programs

sudo skbx capture \
  --filter-trace-xdp \
  --filter-xdp-expr \
    '(xdp->frame_sz = 0 || xdp->frame_sz >= 0o1)' \
  --output-xdp-metadata 'xdp->rxq->dev->ifindex' \
  --output trace.jsonl \
  icmp

Inspect a tunnel

sudo skbx capture --probe ip_local_out \
  --output-tunnel \
  --filter-tunnel-pcap-l2 'ether proto 0x0800' \
  --filter-tunnel-pcap-l3 'icmp' \
  --output trace.jsonl \
  udp port 4789

Run skbx capture --help for the complete surface.

Evidence model

The native stream is traceq/0.1.0: append-only JSONL bounded by a mandatory header and footer.

capture_start
event
event
…
capture_end

Each event carries a stable event: handle. Replay groups ordered events into bounded packet routes, reports consensus and outliers, and emits route: handles. explain retrieves the target event plus nearby evidence sharing the same packet identity.

If the footer is absent—or reports reserve failures, tracer recursion misses, decode failures, enrichment failures, or output failures—the capture is not silently presented as complete.

Install from source

Requirements

• Linux on x86_64 or arm64;
• Rust 1.85 or newer;
• Clang/LLVM with the BPF backend;
• bpftool;
• libelf and libpcap development packages;
• a kernel exposing /sys/kernel/btf/vmlinux;
• root or appropriate BPF/perf capabilities for live capture.

On Ubuntu:

sudo apt-get install "linux-tools-$(uname -r)" clang llvm libelf-dev libpcap-dev pkg-config

On Debian:

sudo apt-get install bpftool clang llvm libelf-dev libpcap-dev pkg-config

If a vendor-specific Ubuntu kernel-tools package omits bpftool, install linux-tools-generic and put the directory containing its bpftool binary first on PATH.

Then install directly from GitHub:

cargo install --git https://github.com/copyleftdev/skbx --locked skbx-cli
skbx doctor

Cargo installs the skbx binary under ~/.cargo/bin by default. To build from a checkout instead, run cargo build --release --locked.

Replay, schema inspection, and evidence lookup do not require root.

Development

make check      # fmt + tests + clippy -D warnings
make build      # optimized release build
make benchmark  # deterministic 100k-event replay benchmark

The build generates vmlinux.h in Cargo’s output directory and embeds the compiled CO-RE object. Generated kernel headers never dirty the source tree.

Root-required integration gates use disposable network namespaces and clean up after themselves. See the validation guide before running them.

Community

Network debugging gets better when evidence is easy to share.

• Found a bug? Use the structured bug report.
• Have a packet path skbx cannot explain yet? Open an observation request.
• Want to contribute? Start with CONTRIBUTING.md.
• Found a security issue? Please follow SECURITY.md, not a public issue.

Bring a kernel version, the exact command, doctor --json, and the reliability footer. Packet folklore is welcome; packet evidence is better.

License

Userspace is licensed under AGPL-3.0-or-later. The eBPF program under crates/skbx-sensor/bpf/ is GPL-2.0-only.