docs: document virtio-net data plane tuning

Author: kvinwang

docs/bridge-networking.md

@@ -25,7 +25,7 @@ Individual VMs can override the global networking mode via:
 - **Web UI**: Networking dropdown in the deploy dialog
 - **API**: `networking: { mode: "bridge" }` in `VmConfiguration`
 
-Only the mode is per-VM; the bridge interface name always comes from the global config.
+The bridge interface name comes from the global config unless the node lists it in `cvm.allowed_bridges`. VMs may also override the vhost and queue settings — see [network-data-plane.md](network-data-plane.md).
 
 ## Host setup
 
@@ -159,7 +159,7 @@ sudo chmod u+s /usr/lib/qemu/qemu-bridge-helper
 
 ## How it works
 
-- VMM passes `-netdev bridge,id=net0,br=<bridge>` to QEMU
+- VMM passes `-netdev tap,id=net0,br=<bridge>,helper=<qemu-bridge-helper>,vhost=on` to QEMU, or `-netdev bridge,id=net0,br=<bridge>` when vhost is disabled
 - QEMU's bridge helper (setuid) creates a TAP device and attaches it to the bridge
 - Guest MAC address is derived from SHA256 of the VM ID, with an optional configurable prefix (stable across restarts for DHCP IP consistency)
 - The host DHCP server (dnsmasq) assigns an IP to the VM
@@ -201,6 +201,8 @@ Bridge and passt VMs can coexist. Set the global default in `vmm.toml` and overr
 vmm-cli.py deploy --name my-vm --image dstack-0.5.6 --compose app.yaml --net passt
 ```
 
-### vhost-net and TDX
+### vhost-net and multiqueue
 
-vhost-net (kernel data plane offload for virtio-net) is **not enabled** for bridge mode. TDX encrypts guest memory, which prevents the host kernel from performing DMA-based packet offload. The default QEMU userspace virtio backend is used instead.
+Bridge NICs use the host kernel's vhost-net data plane by default, and can expose several virtio-net queue pairs. Both are configurable per node and per VM — see [network-data-plane.md](network-data-plane.md) for the knobs, the mode support matrix, and how to pick a queue count.
+
+vhost-net works in a TDX guest: the virtio rings and buffers live in shared, unencrypted memory so that a host-side backend can reach them, which is the same mechanism `vhost-vsock-pci` has always relied on.

docs/libvirt-network-filter.md

@@ -165,7 +165,24 @@ sudo dstack-vmm --config ./vmm.toml \
 User networking and bridge networking with `mode = "none"` never connect to
 `netd`. Libvirt mode fails closed if `netd` is unavailable.
 
-Filtered TAP netdevs currently set `vhost=off`. This keeps the initial backend
-on the directly bound TAP path and avoids adding `/dev/vhost-net` permissions
-to the QEMU user. It is a deliberate security-first throughput tradeoff; a
-future configurable vhost mode requires equivalent filter integration tests.
+Filtered TAP netdevs follow the node's `vhost` and `queues` settings like any
+other TAP-backed NIC (see [network-data-plane.md](network-data-plane.md)). The
+nwfilter binding is installed on the host TAP interface, so packets traverse it
+whether they were written by QEMU or by a vhost worker; filtering is unaffected
+by the data plane choice. Enabling vhost does require the QEMU user to be able
+to open `/dev/vhost-net`.
+
+`netd` also creates the TAP for unfiltered bridge NICs that ask for more than
+one queue pair, because `qemu-bridge-helper` returns a single descriptor and
+cannot create a `multi_queue` device. Those TAPs carry no nwfilter binding, so
+a multiqueue bridge node needs `netd` even when `network_filter.mode = "none"`.
+
+An empty filter name is what selects that unfiltered TAP, so `mode = "libvirt"`
+with an empty `filter` is rejected at config load rather than quietly producing
+an unbound TAP.
+
+Removal and `Check` carry the same distinction: the VMM tells `netd` whether the
+interface it is asking about was created with a binding. Unfiltered TAPs are
+never looked up in libvirt, so a node that has `virsh` installed but no reachable
+`libvirtd` can still create and destroy multiqueue TAPs. The flag defaults to
+true on the wire, so an older VMM's removals still drop their bindings.

docs/macvtap-networking.md

@@ -19,6 +19,9 @@ Configure a NIC through node configuration or an authorized VMM RPC request:
 
 `parent` must name an existing host interface. `macvtap_mode` may be
 `private`, `bridge`, `vepa`, or `passthru`; an empty value selects `private`.
+Macvtap NICs also honour the `vhost` and `queues` settings described in
+[network-data-plane.md](network-data-plane.md); netd creates the interface with
+matching hardware queues and the launcher opens `/dev/tapN` once per queue.
 The configured netd socket permissions apply in the same way as for
 libvirt-filtered bridge networking.
 
@@ -49,8 +52,9 @@ and the same deterministic MAC address passed to QEMU. Netd then:
 4. reads its kernel-assigned ifindex and waits for `/dev/tap<ifindex>`; and
 5. returns that runtime device path to the VMM.
 
-The per-VM launcher opens the character device, places it at the fd referenced
-by QEMU's `-netdev tap,fd=...` argument, and then execs QEMU. This keeps device
+The per-VM launcher opens the character device once per queue pair, places the
+descriptors at the fds referenced by QEMU's `-netdev tap,fd=...` (or `fds=...`)
+argument, and then execs QEMU. This keeps device
 paths out of persistent VM
 configuration, works with both Supervisor and systemd process managers, and
 does not pass network fds through `sudo`.

docs/network-data-plane.md

@@ -0,0 +1,159 @@
+# virtio-net data plane tuning
+
+Every CVM NIC has two knobs that decide how many packets it can move: whether
+the host kernel's vhost-net data plane is used, and how many virtio-net queue
+pairs the device exposes. Both are set per node and overridable per VM.
+
+## Why it matters
+
+Without vhost-net, QEMU drains every received packet on its single main-loop
+thread. That thread is the ceiling, and it does not grow with vCPUs:
+
+```
+maximum packets per second ≈ 1 core ÷ per-packet main-loop cost
+```
+
+The per-packet cost varies with traffic shape — a few microseconds for uniform
+synthetic streams, tens of microseconds for bidirectional short-connection
+traffic — so the ceiling is a property of the workload, not a fixed number.
+What is fixed is the shape of the failure: throughput climbs normally until the
+main loop saturates at 100% of one core, then packets are dropped at the TAP
+before they ever reach the guest. Guest-side counters stay clean, which makes
+the cliff easy to misdiagnose as a network problem.
+
+`vhost=on` moves that work into the host kernel. Guest-side outbound traffic
+uses the same thread, so a busy guest pays the cost twice over.
+
+That returns a whole core, but it relocates the ceiling rather than removing
+it: packets now arrive faster than a single guest receive queue can drain, and
+the drops reappear at a higher rate. Raising `queues` is what removes them.
+Enable vhost everywhere; raise the queue count for VMs that are actually
+throughput-bound.
+
+## Configuration
+
+```toml
+[cvm]
+# Largest queue pair count a deployment RPC caller may request.
+max_net_queues = 8
+
+[cvm.networking]
+mode = "bridge"
+bridge = "dstack-br0"
+vhost = true
+queues = 1
+```
+
+A VM overrides either value at deploy time:
+
+```bash
+vmm-cli.py deploy --name my-vm --image dstack-0.5.9 --compose app.yaml \
+  --net bridge --net-queues 4
+vmm-cli.py deploy --name latency-vm --image dstack-0.5.9 --compose app.yaml \
+  --net bridge --net-no-vhost
+```
+
+Both fields are also on `NetworkingConfig` in the deployment RPC. A request that
+sets only `vhost`/`queues` keeps the node's own networking mode, so tuning does
+not force a caller to restate — or be allowed to choose — a backend. `queues` is
+rejected above the node's `max_net_queues`; `vhost` is not otherwise restricted,
+since it only affects the requesting VM. `GetMeta` reports
+`networking.max_queues` so a client can present the real bound.
+
+Only what a deployment explicitly asks for is recorded against the VM. Values it
+inherits stay owned by the node, so changing `[cvm.networking]` later — including
+setting `vhost = false` to roll the whole node back — still reaches VMs that were
+deployed with some other networking override. A request that names a backend
+does pin that backend's bridge or parent for the life of the VM, as before; one
+that only tunes pins nothing.
+
+Neither field changes the CVM's measurement. `mr_config_id` covers the compose
+hash and instance info, so retuning a NIC does not change app identity or
+require an on-chain update.
+
+## What each mode supports
+
+| Mode | netdev | vhost | queues > 1 |
+|---|---|---|---|
+| `user` | `user,...` | no backend, ignored | not supported, ignored |
+| `bridge` | `bridge,br=` or `tap,br=,helper=` | yes | yes, through netd |
+| `bridge` with libvirt filtering | `tap,ifname=` | yes | yes, through netd |
+| `macvtap` | `tap,fd=` / `tap,fds=` | yes | yes |
+| `custom` | operator's own string | operator's own string | device line only |
+
+QEMU's `bridge` netdev accepts neither `vhost=` nor `queues=`, so enabling
+vhost switches bridge mode to a `tap` netdev driven by the same setuid
+`qemu-bridge-helper`. The VMM still needs no network privileges. The helper has
+no compiled-in default path for the `tap` netdev, so the VMM probes the known
+distribution locations; set `cvm.qemu_bridge_helper` if yours is elsewhere. If
+no helper is found the NIC falls back to the non-vhost `bridge` netdev with a
+warning, because vhost is a default and a default must not stop a node from
+booting VMs.
+
+The helper returns exactly one descriptor, which is why more than one queue
+pair in bridge mode is created by `netd` instead: it adds a persistent
+`multi_queue` TAP that QEMU then opens once per queue. That applies whether or
+not libvirt filtering is on, so a multiqueue bridge node must run `netd` (see
+[libvirt-network-filter.md](libvirt-network-filter.md)). `netd` reports back the
+queue count it created, and the VMM refuses to launch on a mismatch — a `netd`
+deployed separately as a root service can be older than the VMM asking it for
+multiqueue, and QEMU would otherwise reject the interface from inside the
+per-VM launcher.
+
+For macvtap, the per-VM launcher opens the `/dev/tapN` character device once
+per queue pair and hands QEMU the descriptors as `fds=`. `netd` creates the
+interface with matching `numtxqueues`/`numrxqueues`.
+
+Custom mode owns its whole netdev string, including any `vhost=`/`queues=`
+options. The `queues` field still applies to the generated device line, which
+is what makes a hand-written netdev usable with multiqueue.
+
+## Choosing a queue count
+
+More queues raise the packet ceiling for bandwidth-heavy workloads and lower it
+for latency-sensitive ones. Receive processing is spread over more vCPUs, and
+under TDX a cross-vCPU wakeup costs an IPI and a VM exit. Measured on one
+8-vCPU TDX CVM, changing only the guest's channel count:
+
+| Queue pairs | Short-connection throughput |
+|---|---|
+| 1 | 22.3k conn/s |
+| 2 | ~20k conn/s |
+| 4 | 15–21k conn/s |
+| 8 | 6.2–7.7k conn/s |
+
+The same CVM with 8 queues moved 3.0 Mpps of 64-byte UDP with no loss, against
+roughly 600k with one queue. Start at 1, raise it for throughput-bound
+workloads, and measure.
+
+The guest driver uses at most one queue pair per vCPU, so a VM with fewer vCPUs
+than queues simply leaves the extra pairs idle — `ethtool -l eth0` reports the
+smaller number. That is not rejected at deployment, because `vmm-cli.py resize`
+can raise the vCPU count later.
+
+`vectors` is derived, never configured: `2N + 2`, one vector per queue
+direction plus config and control. One queue pair emits no `mq=on` or
+`vectors=` at all, leaving the historical command line byte for byte identical.
+
+## Requirements
+
+The account running QEMU must be able to open `/dev/vhost-net`, which is
+`root:kvm 0660` on a stock host — add that account to the `kvm` group. The
+`vhost_net` module autoloads on first open.
+
+If that account lacks access, QEMU exits at startup and the VM never boots. The
+VMM does not pre-check this: QEMU need not share the VMM's credentials, so
+refusing a launch on the VMM's own access would block deployments the host can
+run. It only warns when the device node is missing outright, which is a fact
+about the host rather than about either account.
+
+vhost-net works normally in a TDX guest: the virtio rings and buffers live in
+shared, unencrypted memory precisely so a host-side backend can reach them.
+This is the same mechanism behind `vhost-vsock-pci`, which dstack has always
+used.
+
+On host kernels older than 6.4 the vhost worker is a free-standing kernel
+thread and its CPU time is not charged to the VM's cgroup. Since 6.4 it is a
+`vhost_task` inside the QEMU thread group, so `cpu.max`, cgroup accounting, and
+`top -H` all attribute it to the CVM. Check the host kernel before relying on
+CPU limits for tenant VMs.