docs: document virtio-net data plane tuning

Author: kvinwang

docs/bridge-networking.md

@@ -4,7 +4,7 @@ By default, dstack-vmm uses **user** networking (QEMU's built-in SLIRP stack, no
 
 ## When to use bridge networking
 
-- High connection concurrency (passt becomes CPU-bound at ~25K+ concurrent connections)
+- High connection concurrency (user-mode networking becomes CPU-bound at ~25K+ concurrent connections)
 - Workloads that need full L2 network access
 - Environments where VMs need to be directly reachable on the LAN
 
@@ -21,11 +21,11 @@ bridge = "virbr0"
 ### Per-VM override
 
 Individual VMs can override the global networking mode via:
-- **CLI**: `vmm-cli.py deploy --net bridge` or `--net passt`
+- **CLI**: `vmm-cli.py deploy --net bridge`, `--net user`, or `--net macvtap`
 - **Web UI**: Networking dropdown in the deploy dialog
 - **API**: `networking: { mode: "bridge" }` in `VmConfiguration`
 
-Only the mode is per-VM; the bridge interface name always comes from the global config.
+The bridge interface name comes from the global config unless the node lists it in `cvm.allowed_bridges`. VMs may also override the vhost and queue settings — see [network-data-plane.md](network-data-plane.md).
 
 ## Host setup
 
@@ -143,9 +143,15 @@ mode = "bridge"
 bridge = "dstack-br0"
 ```
 
-### QEMU bridge helper setup (required for both options)
+### QEMU bridge helper setup (needed unless every bridge NIC goes through netd)
 
 The bridge helper allows QEMU to create and attach TAP devices without VMM needing root privileges.
+It is used only on the single-queue bridge paths; a NIC that `netd` builds never touches it, so a
+node that runs `netd` for all of its bridge VMs does not need it at all.
+
+The VMM probes `/usr/lib/qemu/qemu-bridge-helper`, `/usr/libexec/qemu-bridge-helper` and
+`/usr/local/libexec/qemu-bridge-helper`. Set `cvm.qemu_bridge_helper` in `vmm.toml` for a path
+outside that list.
 
 ```bash
 # Allow QEMU to use the bridge
@@ -159,12 +165,13 @@ sudo chmod u+s /usr/lib/qemu/qemu-bridge-helper
 
 ## How it works
 
-- VMM passes `-netdev bridge,id=net0,br=<bridge>` to QEMU
-- QEMU's bridge helper (setuid) creates a TAP device and attaches it to the bridge
+- With more than one queue pair, or with libvirt filtering on, `netd` creates the TAP and the VMM passes `-netdev tap,id=net0,ifname=<tap>,...` — this is the usual case on a node running `netd` with multi-vCPU VMs, since queue pairs default to the VM's vCPU count. Without `netd`, a bridge NIC that took that default drops back to one queue pair and takes a helper path below
+- Otherwise the VMM passes `-netdev tap,id=net0,br=<bridge>,helper=<qemu-bridge-helper>,vhost=on`, or `-netdev bridge,id=net0,br=<bridge>` when vhost is off or no helper is found
+- QEMU's bridge helper (setuid) creates a TAP device and attaches it to the bridge on the two helper paths
 - Guest MAC address is derived from SHA256 of the VM ID, with an optional configurable prefix (stable across restarts for DHCP IP consistency)
 - The host DHCP server (dnsmasq) assigns an IP to the VM
-- When QEMU exits, the TAP device is automatically destroyed
-- VMM does not need root or `CAP_NET_ADMIN`
+- On the two bridge-helper paths the TAP disappears when QEMU exits; a `netd`-created TAP is persistent and is deleted when the VMM tears the VM's networking down
+- The VMM process itself needs neither root nor `CAP_NET_ADMIN` on any path; the `netd` path moves that privilege into a separate root service instead
 
 ### MAC address prefix
 
@@ -194,13 +201,15 @@ The remaining bytes are derived from the VM ID hash. The prefix applies to all n
 
 ### Mixing networking modes
 
-Bridge and passt VMs can coexist. Set the global default in `vmm.toml` and override per-VM as needed:
+Bridge and user-mode VMs can coexist. Set the global default in `vmm.toml` and override per-VM as needed:
 
 ```bash
-# Global default is bridge, but deploy this VM with passt
-vmm-cli.py deploy --name my-vm --image dstack-0.5.6 --compose app.yaml --net passt
+# Global default is bridge, but deploy this VM with user networking
+vmm-cli.py deploy --name my-vm --image dstack-0.5.6 --compose app.yaml --net user
 ```
 
-### vhost-net and TDX
+### vhost-net and multiqueue
+
+Bridge NICs use the host kernel's vhost-net data plane by default, and can expose several virtio-net queue pairs. Both are configurable per node and per VM — see [network-data-plane.md](network-data-plane.md) for the knobs, the mode support matrix, and how to pick a queue count.
 
-vhost-net (kernel data plane offload for virtio-net) is **not enabled** for bridge mode. TDX encrypts guest memory, which prevents the host kernel from performing DMA-based packet offload. The default QEMU userspace virtio backend is used instead.
+vhost-net works in a TDX guest: the virtio rings and buffers live in shared, unencrypted memory so that a host-side backend can reach them, which is the same mechanism `vhost-vsock-pci` has always relied on.

docs/libvirt-network-filter.md

@@ -14,8 +14,11 @@ host mechanism.
 
 The measurable acceptance criteria are:
 
-- `network_filter = "none"` preserves the existing QEMU `-netdev bridge`
-  behavior and does not require `netd` or libvirt.
+- `network_filter = "none"` installs no nwfilter binding. It still uses `netd`
+  for any NIC with more than one queue pair, and a `tap` netdev behind
+  `qemu-bridge-helper` whenever vhost is on; only a single-queue, non-vhost
+  bridge NIC keeps the historical `-netdev bridge` path with no `netd` or
+  libvirt dependency.
 - `network_filter = "libvirt"` creates the TAP and filter binding before QEMU
   is submitted to Supervisor, and uses QEMU `-netdev tap`.
 - A failed TAP or filter setup prevents QEMU from starting and rolls back all
@@ -53,8 +56,11 @@ allowed_macvtap_parents = []
 
 Macvtap is excluded from `allowed_network_modes` by default. Empty bridge and
 macvtap-parent allowlists prevent RPC callers from overriding the respective
-node defaults. If macvtap is explicitly enabled, callers may select only a
-parent in `allowed_macvtap_parents`; the macvtap forwarding mode always comes
+node defaults. If macvtap is explicitly enabled, callers may select a
+parent listed in `allowed_macvtap_parents`, the node's own configured parent, or
+one this VM already holds — restating a value the node would have supplied
+anyway grants nothing new. The same applies to `bridge_name` and
+`allowed_bridges`. The macvtap forwarding mode always comes
 from `[cvm.networking].macvtap_mode` and cannot be selected through deployment
 RPCs. These allowlists authorize attachment targets; an nwfilter is not a
 substitute for that authorization.
@@ -82,7 +88,8 @@ For libvirt mode, startup is:
 2. Create the TAP for the configured QEMU UID and attach it to the bridge.
 3. Create a libvirt nwfilter binding for the TAP.
 4. Bring the TAP up and return success.
-5. Start QEMU directly with `-netdev tap,script=no,downscript=no`.
+5. Start QEMU directly with `-netdev tap,script=no,downscript=no`, carrying
+   `vhost=on|off` and, above one queue pair, `queues=N`.
 
 Teardown stops QEMU first, removes the binding, and deletes the TAP. Operations
 are serialized by `netd`. The design intentionally does not add ownership
@@ -100,18 +107,28 @@ validated by libvirt.
 
 ## Deployment modes
 
-Production should run one shared service. `netd` reads only the `[netd]`
-section, so its root-owned configuration can be small and independent of every
-VMM instance:
+Production should run one shared service. `netd` reads the `[netd]` section,
+plus `cvm.network_filter.mode` if the file has one, so its root-owned
+configuration can be small and independent of every VMM instance:
 
 ```toml
 # /etc/dstack/netd.toml
 [netd]
 socket = "/run/dstack/netd.sock"
 socket_mode = 0o660
 libvirt_uri = "qemu:///system"
+# Refuse to build a bridge TAP with no nwfilter binding. Required here because
+# this file has no [cvm] section for netd to read the node's policy from.
+require_filter = true
 ```
 
+`require_filter` is netd's own copy of the invariant, not a convenience. netd is
+the privileged side of the socket, and anything that can reach the socket can
+ask for an unfiltered TAP on a host bridge — a request a filtering node has to
+refuse in the daemon rather than in its caller. When netd and the VMM share one
+`vmm.toml`, leaving it unset derives it from `cvm.network_filter.mode` so the
+two cannot drift apart.
+
 Production deployments can use systemd socket activation. The socket unit
 owns the filesystem mode and ownership; `netd.socket_mode` applies only to the
 standalone bind path.
@@ -162,10 +179,37 @@ sudo dstack-vmm --config ./vmm.toml \
   --netd-socket /run/dstack-dev/netd.sock
 ```
 
-User networking and bridge networking with `mode = "none"` never connect to
-`netd`. Libvirt mode fails closed if `netd` is unavailable.
-
-Filtered TAP netdevs currently set `vhost=off`. This keeps the initial backend
-on the directly bound TAP path and avoids adding `/dev/vhost-net` permissions
-to the QEMU user. It is a deliberate security-first throughput tradeoff; a
-future configurable vhost mode requires equivalent filter integration tests.
+User networking never asks `netd` to build an interface; the VMM still opens a
+short liveness-probe connection to the netd socket on every launch and when
+describing a stopped VM. Libvirt mode fails closed if `netd`
+is unavailable. Bridge networking with `mode = "none"` connects only when it
+needs more than one queue pair, as described below.
+
+Filtered TAP netdevs follow the node's `vhost` and `queues` settings like any
+other TAP-backed NIC (see [network-data-plane.md](network-data-plane.md)). The
+nwfilter binding is installed on the host TAP interface, so packets traverse it
+whether they were written by QEMU or by a vhost worker; filtering is unaffected
+by the data plane choice. Enabling vhost does require the QEMU user to be able
+to open `/dev/vhost-net`.
+
+`netd` also creates the TAP for unfiltered bridge NICs that ask for more than
+one queue pair, because `qemu-bridge-helper` returns a single descriptor and
+cannot create a `multi_queue` device. Those TAPs carry no nwfilter binding, so
+a multiqueue bridge node needs `netd` even when `network_filter.mode = "none"`.
+
+An empty filter name is what selects that unfiltered TAP, so `mode = "libvirt"`
+with an empty `filter` is rejected at config load rather than quietly producing
+an unbound TAP.
+
+Removal carries the same distinction: the VMM tells `netd` whether the interface
+it is asking about was created with a binding, from a record made when it was
+built rather than from configuration that may have changed since. A binding it
+was told about must be gone before `netd` returns; otherwise `netd` still asks
+libvirt to clear one — an interface name is reused by the same VM, and a
+leftover binding's rules would be inherited — but a `libvirtd` it cannot reach
+is a warning rather than a failure. So a node with `virsh` installed and no
+running `libvirtd` can create and destroy multiqueue TAPs. The flag defaults to
+true on the wire, so an older VMM's removals still drop their bindings.
+
+`netd` requires the `virsh` binary to be present whatever the filter mode; it is
+`libvirtd` that unfiltered work does not need.

docs/macvtap-networking.md

@@ -19,6 +19,9 @@ Configure a NIC through node configuration or an authorized VMM RPC request:
 
 `parent` must name an existing host interface. `macvtap_mode` may be
 `private`, `bridge`, `vepa`, or `passthru`; an empty value selects `private`.
+Macvtap NICs also honour the `vhost` and `queues` settings described in
+[network-data-plane.md](network-data-plane.md); netd creates the interface with
+matching hardware queues and the launcher opens `/dev/tapN` once per queue.
 The configured netd socket permissions apply in the same way as for
 libvirt-filtered bridge networking.
 
@@ -49,8 +52,9 @@ and the same deterministic MAC address passed to QEMU. Netd then:
 4. reads its kernel-assigned ifindex and waits for `/dev/tap<ifindex>`; and
 5. returns that runtime device path to the VMM.
 
-The per-VM launcher opens the character device, places it at the fd referenced
-by QEMU's `-netdev tap,fd=...` argument, and then execs QEMU. This keeps device
+The per-VM launcher opens the character device once per queue pair, places the
+descriptors at the fds referenced by QEMU's `-netdev tap,fd=...` (or `fds=...`)
+argument, and then execs QEMU. This keeps device
 paths out of persistent VM
 configuration, works with both Supervisor and systemd process managers, and
 does not pass network fds through `sudo`.