Close vfork-bomb TOCTOU residual via BPF-LSM exec filter

[drewmchugh]

Context

PR kipz/nono#20 ships an exec filter that closes the original full-path mediation bypass and (after comment 4337852261) also closes the AllowShim TOCTOU bypass via a multi-threaded execve check (commit 05b26c8, validated 0/600 against kipz's POC).

A residual remains: vfork-bomb. A multi-threaded parent vforks a child whose execve traps in our filter; the child has Threads=1 (passes the check), but vfork shares the parent's MM, and the parent's sibling threads can swap the args[0] buffer between our supervisor's classification and the kernel's post-CONTINUE re-read.

Empirically confirmed exploitable: 22/300 bypasses on x86_64 against the shipped multi-threaded check using a vfork-bomb POC (/tmp/exec-filter-poc/vfork_attacker.c, structure parallel to kipz's). Higher rate than the original POC (8/300) because the swap thread runs unimpeded in the parent's tgid — we have no awareness of it.

Why neither userspace path closes it cleanly

Two userspace closure options were evaluated and rejected:

• Path A (block vfork-style cloning at seccomp). Block vfork, clone(CLONE_VM, !CLONE_THREAD), and clone3 with the same flag pattern. clone3 is the hard part — flags come via a struct pointer that BPF can't deref, so it'd need USER_NOTIF or blanket-block (collateral on Go runtime, glibc 2.34+ posix_spawn).
• Path B (deny execve when MM is shared with a multi-threaded process). kcmp(KCMP_VM) detects shared MM precisely. Implementation is small (~60 lines, no BPF changes). But the populations it identifies are mostly Go's syscall.ForkExec and glibc posix_spawn legitimate use, not just attackers — neither has a fork() fallback when its vfork-style clone returns EACCES. Same userspace error breaks bazel build, gh, kubectl, terraform, etc.

Both paths break Go-based tooling end-to-end. The collateral isn't acceptable for the workspace fleet's typical workload.

Why BPF-LSM is the right primitive

The race exists because seccomp-notify on execve is fundamentally pre-syscall — the kernel re-reads user memory after our CONTINUE. We can't substitute the validated path; the kernel has to honor what the user passed.

BPF-LSM hooks fire inside the kernel's exec path, after the kernel has resolved the binary (bprm_check_security runs in the kernel's do_execveat_common after path lookup, after binfmt resolution, before exec is committed). Whatever the BPF program sees there is what the kernel will actually exec — there's no further user-memory re-read. Closes both the original kipz race and vfork-bomb structurally.

Available since Linux 5.7 (BPF_PROG_TYPE_LSM). No kernel patches needed; just attach a program at runtime.

Caveat: requires bpf in the active LSM list, which means it needs to be in the kernel boot cmdline (lsm=...). Not enabled by default on most distros.

Workspaces architecture as it relates to this

Investigated ~/dd/dd-source/domains/devex/workspaces/ to understand deployment.

• Layered as: EC2 host (Ubuntu kernel from AWS AMI) → ECS-managed Docker container per host → workspace user environment.
• AMI build: Packer config at domains/devex/workspaces/config/cloud/ami/. Currently does NOT customize kernel boot cmdline. Inherits Ubuntu cloud kernel default (lockdown,capability,landlock,yama,apparmor — no bpf).
• Container caps (verified via capsh): bounding set includes CAP_SYS_ADMIN, CAP_BPF, CAP_PERFMON, CAP_SYS_PTRACE. So a container process with the right effective caps CAN load BPF programs.
• Default workspace user (bits) has empty effective caps but sudo NOPASSWD: ALL.
• /usr/bin/nono is a regular root-owned executable, no setuid, no file caps. Installed by something outside the workspaces AMI (probably shadowfax).
• Host kernel cmdline (verified): BOOT_IMAGE=/boot/vmlinuz-6.8.0-1051-aws root=PARTUUID=... ro console=tty1 console=ttyS0 nvme_core.io_timeout=4294967295 panic=-1 — no lsm= parameter.
• `/sys/kernel/security/lsm` (verified): lockdown,capability,landlock,yama,apparmor (no bpf). Confirms our problem.

Two-front change required

Front 1: Workspaces AMI

Add bpf to the active LSM list via grub cmdline. Two implementation options:

Option A (declarative, preferred): drop a file at config/cloud/ami/files/initroot/etc/default/grub.d/99-bpf-lsm.cfg:

```
GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT lsm=lockdown,capability,landlock,yama,apparmor,bpf"
```

Plus a packer post-step to run `update-grub`. Order in lsm= matters: `bpf` last by convention since hooks run in list order and bpf is typically additive monitoring.

Option B (imperative): a packer shell provisioner that `sed`'s `/etc/default/grub` and runs `update-grub`. Simpler but less version-controlled.

Prerequisite to verify: `sudo zgrep CONFIG_BPF_LSM /boot/config-$(uname -r)` on a workspace VM. If `=y`, no kernel rebuild needed (Ubuntu 22.04+ cloud kernels almost certainly enable it).

Front 2: Nono BPF-LSM exec filter

Replace (or augment) the current seccomp-notify exec filter with a BPF-LSM program that:

1. Attaches to `bprm_check_security` (kernel hook fired during exec, post-resolution, pre-commit).
2. Reads the resolved binary's inode/path from `linux_binprm`.
3. Compares against a deny-set populated by the supervisor at session start (BPF map, e.g., `BPF_MAP_TYPE_HASH` keyed by inode, or `BPF_MAP_TYPE_LPM_TRIE` keyed by path).
4. Returns `-EACCES` to abort the exec atomically.
5. Emits audit events via BPF ringbuf or perf events; supervisor reads and writes JSONL.

Privilege model (separate concern from workspaces AMI): nono runs as user `bits`, has no CAP_BPF effective. Three options:

• Sudo at invocation: works (NOPASSWD), changes UX for users.
• File caps on `/usr/bin/nono`: `setcap cap_bpf+eip`. Whoever installs nono (shadowfax?) adds this step. Cleanest for users.
• Privileged loader daemon: separate systemd service runs as root, loads the LSM program once, exposes a Unix socket the unprivileged broker uses to populate the deny-set map. More infrastructure.

Fallback: not all hosts will have BPF-LSM. nono needs to detect availability at runtime (`/sys/kernel/security/lsm` contains `bpf` AND we can load a probe program) and fall back to the existing seccomp-notify path with documented vfork residual on hosts that don't have it.

Open questions / things to validate

1. CONFIG_BPF_LSM=y on workspace kernel? Almost certainly yes for Ubuntu 22.04+ but verify with `sudo zgrep CONFIG_BPF_LSM /boot/config-$(uname -r)`.
2. Datadog Agent CWS interaction. CWS supports BPF-LSM as a hook backend in modern versions. Does enabling `lsm=...,bpf` cause CWS to switch backends silently? Likely benign or beneficial (better detection telemetry) but worth a sync with the agent team.
3. Other tooling assumptions. Anything on the workspace fleet that depends on `bpf` NOT being in the LSM list. None expected, but worth a thought.
4. One-workspace-per-host mitigates blast radius — a workspace's BPF-LSM programs only affect its own EC2 instance. No cross-tenant concern.
5. Performance: BPF LSM hooks are no-ops when no programs are loaded; bare cost of adding bpf to lsm= is single-digit ns per syscall. Negligible.
6. Where does nono get installed in workspaces? Need to find the shadowfax (or other) provisioning step that lays down `/usr/bin/nono` and bake `setcap cap_bpf+eip` (or equivalent) there.
7. Compatibility floor: kernel ≥ 5.7 for BPF-LSM. Workspace kernel is 6.8 — fine.
8. deny-set map design: inode-keyed vs path-keyed. inode-keyed is fast and stable but requires re-resolving paths to inodes at session start; path-keyed via LPM_TRIE is more flexible but slower.
9. Shebang chain handling: `bprm_check_security` fires on each exec including the binfmt_script-resolved interpreter, so shebang chains are caught at every level naturally — could simplify or replace the userspace shebang walker.

Step-by-step action plan

1. [Workspaces team] Verify `CONFIG_BPF_LSM=y` on workspace host kernel (one `zgrep`).
2. [Workspaces team] Single-VM smoke test: `sudo` edit grub on one workspace, reboot, confirm `/sys/kernel/security/lsm` contains `bpf`, run a couple workloads (Bazel build, Bun-based agent, Datadog Agent stays healthy). ~30 min.
3. [Workspaces team] PR to add `config/cloud/ami/files/initroot/etc/default/grub.d/99-bpf-lsm.cfg` (Option A above) + packer post-step for `update-grub`. Bake test AMI via `pipeline_id = "yourname_local_build"`. Test via `scripts/test-ami.sh`.
4. [Workspaces team] Stage AMI rollout via `staging.auto.tfvars` → conductor flow. Canary, then prod.
5. [Nono side, can start in parallel after step 2] New branch `am/exec-filter-bpf-lsm` off this PR. Implement the BPF-LSM program and loader (likely using the `aya` Rust crate for ergonomics; alternative: libbpf-rs or hand-rolled libbpf-sys). Detect availability at runtime, fall back to seccomp-notify when unavailable.
6. [Nono side] Decide privilege model. My lean: file caps on `/usr/bin/nono` (`setcap cap_bpf+eip`). Coordinate with shadowfax install step.
7. [Nono side] Validate against vfork POC at `/tmp/exec-filter-poc/vfork_attacker.c` (or equivalent). Target: 0/N bypasses for N ≥ 600.
8. [Nono side] Validate against original kipz POC (`/tmp/exec-filter-poc/attacker.c`). Target: 0/N bypasses.
9. [Nono side] PR. Probably retargets PR feat(linux): seccomp exec filter to close the full-path mediation bypass kipz/nono#20 base or follows it as a sequential PR.

Reference / context for picking this up later

POCs at `/tmp/exec-filter-poc/` on this workspace:

• `attacker.c` — kipz's pthread_create + execve POC (closed by shipped multi-threaded check).
• `vfork_attacker.c` — the vfork-bomb variant, exploitable at 22/300 against shipped code.
• `run_test.sh` — runner that takes `NONO=path/to/nono SHIM=path/to/nono-shim ATTEMPTS=N LABEL=tag ATTACKER_SRC=attacker.c|vfork_attacker.c`.

Files in nono codebase touched by current exec filter (PR kipz#20):

• `crates/nono-cli/src/exec_strategy/supervisor_linux.rs` — `handle_exec_notification`, `count_threads`, etc.
• `crates/nono-cli/src/mediation/filter_audit.rs` — `FilterAuditEvent`, reasons.
• `crates/nono-cli/src/mediation/shebang.rs` — chain walker.
• `crates/nono/src/sandbox/linux.rs` — `build_seccomp_exec_filter`, `install_seccomp_exec_filter`.
• `docs/linux-exec-filter-plan.md` — current design doc; vfork-bomb section is the residual we're closing here.

Workspaces files relevant to AMI change:

• `~/dd/dd-source/domains/devex/workspaces/config/cloud/ami/amazon-ubuntu.pkr.hcl` — packer config, where you'd add a provisioner.
• `~/dd/dd-source/domains/devex/workspaces/config/cloud/ami/files/initroot/` — files baked into the AMI; new grub.d file goes here.
• `~/dd/dd-source/domains/devex/workspaces/config/cloud/AGENTS.md` — AMI build/deploy procedures.

Empirical state of this workspace (captured during planning):

• Hostname: `am-nono`
• Kernel: `6.8.0-1051-aws`
• LSM stack: `lockdown,capability,landlock,yama,apparmor` (no bpf)
• Boot cmdline: no `lsm=` parameter
• Container bounding set: includes `cap_bpf`, `cap_sys_admin`, `cap_perfmon`
• User `bits` sudo: NOPASSWD ALL
• `/usr/bin/nono`: regular executable, no caps