docs: document virtio-net data plane tuning

Author: kvinwang

docs/bridge-networking.md

@@ -210,6 +210,6 @@ vmm-cli.py deploy --name my-vm --image dstack-0.5.6 --compose app.yaml --net use
 
 ### vhost-net and multiqueue
 
-Bridge NICs use the host kernel's vhost-net data plane by default, and can expose several virtio-net queue pairs. Both are configurable per node and per VM — see [network-data-plane.md](network-data-plane.md) for the knobs, the mode support matrix, and how to pick a queue count.
+Bridge NICs can run on the host kernel's vhost-net data plane and expose several virtio-net queue pairs. Both are off by default and enabled per node or per VM — see [network-data-plane.md](network-data-plane.md) for the knobs, the enablement checklist, the mode support matrix, and how to pick a queue count.
 
 vhost-net works in a TDX guest: the virtio rings and buffers live in shared, unencrypted memory so that a host-side backend can reach them, which is the same mechanism `vhost-vsock-pci` has always relied on.

docs/network-data-plane.md

@@ -30,9 +30,14 @@ cost twice over.
 it relocates the ceiling rather than removing
 it: packets now arrive faster than a single guest receive queue can drain, and
 the drops reappear at a higher rate. More queue pairs is what removes them,
-which is why both are defaults: vhost everywhere, and a queue count that follows
-the VM's vCPU count. The knob you are more likely to reach for is the other
-direction — see [Choosing a queue count](#choosing-a-queue-count).
+which is why enabling vhost also enables a queue count that follows the VM's
+vCPU count — the two travel together.
+
+vhost is **off by default** and enabled per node (or per VM). Two things make
+it an opt-in rather than a default: turning it on changes the virtio-net device
+of every bridge/macvtap VM on its next boot, and it requires `/dev/vhost-net`
+to be accessible to the account QEMU runs under, which the VMM cannot verify on
+the operator's behalf — see [Enabling vhost on a node](#enabling-vhost-on-a-node).
 
 ## Configuration
 
@@ -47,22 +52,24 @@ bridge = "dstack-br0"
 vhost = true
 ```
 
-Queue pairs are not a node setting. They default to the VM's vCPU count, capped
-at 16, because the useful number follows the VM rather than the host — the guest
-driver uses at most one queue pair per vCPU. A deployment overrides that per VM,
-up to `max_net_queues`.
+Queue pairs are not a node setting. With vhost on they default to the VM's vCPU
+count, capped at 16, because the useful number follows the VM rather than the
+host — the guest driver uses at most one queue pair per vCPU. A deployment
+overrides that per VM, up to `max_net_queues`.
 
 Raising `max_net_queues` above 16 widens what a deployment may ask for without
 moving the default's cap, so a larger VM never silently acquires a worse
 default. Lowering it below 16 does lower the default too, because a node that
 refuses a request for four queue pairs should not hand out sixteen by itself.
 The hard ceiling from any source is 64.
 
-Turning vhost off also turns the multiqueue default off. Without vhost the QEMU
-main loop drains every queue on one thread, so extra queues buy little while
-still costing a netd interface, more MSI-X vectors, and a changed guest device.
-An explicit queue count is still honoured without vhost, since that combination
-is a deliberate request rather than a default.
+Without vhost the default is a single queue pair. The QEMU main loop drains
+every queue on one thread, so extra queues buy little while still costing a
+netd interface, more MSI-X vectors, and a changed guest device. An explicit
+queue count is still honoured without vhost, since that combination is a
+deliberate request rather than a default. The two defaults travelling together
+also means a node that never sets `vhost` keeps building the device its VMs
+have always had.
 
 A VM overrides either value at deploy time, and `UpdateVm` changes them
 afterwards — the new values apply from the VM's next boot:
@@ -128,6 +135,31 @@ vhost state are not part of it, so retuning a NIC does not change app identity
 or require an on-chain update. Adding or removing a NIC does: the NIC count
 changes the guest's ACPI tables and therefore RTMR0.
 
+## Enabling vhost on a node
+
+Setting `vhost = true` in `[cvm.networking]` is a node-wide behaviour change:
+every bridge or macvtap VM that has not pinned its own data plane gets a
+different virtio-net device on its next boot — `vhost=on`, `mq=on` with
+vCPU-scaled queue pairs, and the matching MSI-X vector count. The device is not
+measured, so attestation and app identity are unaffected. Before flipping it:
+
+1. **Verify `/dev/vhost-net` is accessible to the account QEMU runs under.**
+   It is `root:kvm 0660` on Debian-family hosts, where adding the account to
+   the `kvm` group suffices, and `root:root 0600` on several others. If the
+   account lacks access, QEMU exits at launch and every affected VM stops
+   restarting. The VMM warns at startup when its own access fails, but it
+   cannot refuse on that basis — QEMU need not share its credentials.
+
+2. **Restart `netd` before or together with the VMM.** Multiqueue bridge NICs
+   are prepared by `netd`, and the VMM checks that `netd` echoes the queue
+   count it built. An older `netd` fails that check; the launch is rolled back
+   and fails with the reason in the VMM log, but the VM does not start until
+   `netd` is upgraded.
+
+3. **Roll back by setting `vhost = false`.** The node value reaches every VM
+   that did not pin `vhost` explicitly, from its next boot; a VM that pinned
+   `vhost = true` keeps it until updated.
+
 ## What each mode supports
 
 | Mode | netdev | vhost | queues > 1 |
@@ -144,8 +176,8 @@ vhost switches bridge mode to a `tap` netdev driven by the same setuid
 no compiled-in default path for the `tap` netdev, so the VMM probes the known
 distribution locations; set `cvm.qemu_bridge_helper` if yours is elsewhere. If
 no helper is found the NIC falls back to the non-vhost `bridge` netdev with a
-warning, because vhost is a default and a default must not stop a node from
-booting VMs.
+warning, because a node-wide setting must not stop a node from booting VMs
+that never asked for it.
 
 The helper returns exactly one descriptor, which is why more than one queue
 pair in bridge mode is created by `netd` instead: it adds a persistent
@@ -204,6 +236,21 @@ A VM with fewer vCPUs than queues leaves the extra pairs idle — `ethtool -l
 eth0` reports the smaller number. An explicit over-provision is not rejected at
 deployment, because `vmm-cli.py resize` can raise the vCPU count later.
 
+Queue pairs also cost guest memory, and the default follows vCPUs while memory
+does not: each RX ring keeps 256 page-sized buffers posted, so 16 queue pairs
+pin roughly 16 MB of pages plus per-queue NAPI and socket state. On a 1 GB
+guest with 16 queue pairs under sustained load, RX refill failed with atomic
+order-0 page-allocation errors (`try_fill_recv`/`skb_page_frag_refill` in the
+guest log); the same shape with 2 GB ran clean. A many-vCPU VM given very
+little memory should also be given fewer queue pairs — as a working floor,
+keep at least ~128 MB of guest RAM per queue pair beyond the application's own
+needs. The TDX bounce-buffer pool is not the constraint: the guest kernel
+sizes swiotlb at 6% of RAM clamped to [64 MB, 1 GB], while peak demand is
+bounded by ring size at about 2 MB per queue pair — a deliberately undersized
+32 MB pool sustained full multiqueue line rate with zero `swiotlb buffer is
+full` events, so adding a `swiotlb=` boot parameter would only take memory
+away from the allocator that actually runs out.
+
 `vectors` is derived, never configured: `2N + 2`, one vector per queue
 direction plus config and control. One queue pair emits no `mq=on` or
 `vectors=` at all, leaving the guest device line byte for byte identical to the
@@ -223,11 +270,29 @@ interface to describe, so it reports what the next launch would build instead --
 the same calculation, against the node configuration and manifest as they stand
 now, rather than the ones a finished boot ran under.
 
-If that account lacks access, QEMU exits at startup and the VM never boots. The
-VMM does not pre-check this: QEMU need not share the VMM's credentials, so
-refusing a launch on the VMM's own access would block deployments the host can
-run. It only warns when the device node is missing outright, which is a fact
-about the host rather than about either account.
+If that account lacks access, QEMU exits at startup and the VM never boots —
+there is no fallback to the userspace backend at this point, on any QEMU
+version (verified on 8.2.2 and 10.2). What the per-VM launcher log shows
+depends on the version: QEMU 8.2 prints `warning: tap: open vhost char device
+failed: Permission denied` (once per queue) and then dies on `net/net.c:1185:
+net_client_init1: Assertion 'nc' failed` — an upstream bug
+([qemu#1486](https://gitlab.com/qemu-project/qemu/-/issues/1486)); later
+versions exit cleanly with `Could not open '/dev/vhost-net'`. Grep for either.
+The VMM does not refuse a launch over this: QEMU need not share the VMM's
+credentials, so a refusal based on the VMM's own access would block deployments
+the host can run. It warns instead — when the device node is missing outright,
+and when the VMM's own open is denied, since QEMU usually does share its
+account.
+
+QEMU does have a *runtime* fallback, at a different failure point: once the
+netdev initialized with vhost, a later `vhost_net_start()` failure at guest
+driver activation logs `unable to start vhost net: <errno>: falling back on
+userspace virtio` and keeps the NIC working on the userspace data path. That
+path is reachable only after `/dev/vhost-net` was opened successfully at
+launch, so an access problem never lands there. If it does fire, it is the one
+case where `GetInfo` can overstate the data plane — the interface reports the
+vhost state the launch settled while the packets take the userspace path — and
+that QEMU log line is the indicator.
 
 vhost-net works normally in a TDX guest: the virtio rings and buffers live in
 shared, unencrypted memory precisely so a host-side backend can reach them.