docs: document virtio-net data plane tuning

Author: kvinwang

docs/bridge-networking.md

@@ -210,6 +210,6 @@ vmm-cli.py deploy --name my-vm --image dstack-0.5.6 --compose app.yaml --net use
 
 ### vhost-net and multiqueue
 
-Bridge NICs use the host kernel's vhost-net data plane by default, and can expose several virtio-net queue pairs. Both are configurable per node and per VM — see [network-data-plane.md](network-data-plane.md) for the knobs, the mode support matrix, and how to pick a queue count.
+Bridge NICs can run on the host kernel's vhost-net data plane and expose several virtio-net queue pairs. Both are off by default and enabled per node or per VM — see [network-data-plane.md](network-data-plane.md) for the knobs, the enablement checklist, the mode support matrix, and how to pick a queue count.
 
 vhost-net works in a TDX guest: the virtio rings and buffers live in shared, unencrypted memory so that a host-side backend can reach them, which is the same mechanism `vhost-vsock-pci` has always relied on.

docs/network-data-plane.md

@@ -30,9 +30,14 @@ cost twice over.
 it relocates the ceiling rather than removing
 it: packets now arrive faster than a single guest receive queue can drain, and
 the drops reappear at a higher rate. More queue pairs is what removes them,
-which is why both are defaults: vhost everywhere, and a queue count that follows
-the VM's vCPU count. The knob you are more likely to reach for is the other
-direction — see [Choosing a queue count](#choosing-a-queue-count).
+which is why enabling vhost also enables a queue count that follows the VM's
+vCPU count — the two travel together.
+
+vhost is **off by default** and enabled per node (or per VM). Two things make
+it an opt-in rather than a default: turning it on changes the virtio-net device
+of every bridge/macvtap VM on its next boot, and it requires `/dev/vhost-net`
+to be accessible to the account QEMU runs under, which the VMM cannot verify on
+the operator's behalf — see [Enabling vhost on a node](#enabling-vhost-on-a-node).
 
 ## Configuration
 
@@ -47,22 +52,24 @@ bridge = "dstack-br0"
 vhost = true
 ```
 
-Queue pairs are not a node setting. They default to the VM's vCPU count, capped
-at 16, because the useful number follows the VM rather than the host — the guest
-driver uses at most one queue pair per vCPU. A deployment overrides that per VM,
-up to `max_net_queues`.
+Queue pairs are not a node setting. With vhost on they default to the VM's vCPU
+count, capped at 16, because the useful number follows the VM rather than the
+host — the guest driver uses at most one queue pair per vCPU. A deployment
+overrides that per VM, up to `max_net_queues`.
 
 Raising `max_net_queues` above 16 widens what a deployment may ask for without
 moving the default's cap, so a larger VM never silently acquires a worse
 default. Lowering it below 16 does lower the default too, because a node that
 refuses a request for four queue pairs should not hand out sixteen by itself.
 The hard ceiling from any source is 64.
 
-Turning vhost off also turns the multiqueue default off. Without vhost the QEMU
-main loop drains every queue on one thread, so extra queues buy little while
-still costing a netd interface, more MSI-X vectors, and a changed guest device.
-An explicit queue count is still honoured without vhost, since that combination
-is a deliberate request rather than a default.
+Without vhost the default is a single queue pair. The QEMU main loop drains
+every queue on one thread, so extra queues buy little while still costing a
+netd interface, more MSI-X vectors, and a changed guest device. An explicit
+queue count is still honoured without vhost, since that combination is a
+deliberate request rather than a default. The two defaults travelling together
+also means a node that never sets `vhost` keeps building the device its VMs
+have always had.
 
 A VM overrides either value at deploy time, and `UpdateVm` changes them
 afterwards — the new values apply from the VM's next boot:
@@ -128,6 +135,31 @@ vhost state are not part of it, so retuning a NIC does not change app identity
 or require an on-chain update. Adding or removing a NIC does: the NIC count
 changes the guest's ACPI tables and therefore RTMR0.
 
+## Enabling vhost on a node
+
+Setting `vhost = true` in `[cvm.networking]` is a node-wide behaviour change:
+every bridge or macvtap VM that has not pinned its own data plane gets a
+different virtio-net device on its next boot — `vhost=on`, `mq=on` with
+vCPU-scaled queue pairs, and the matching MSI-X vector count. The device is not
+measured, so attestation and app identity are unaffected. Before flipping it:
+
+1. **Verify `/dev/vhost-net` is accessible to the account QEMU runs under.**
+   It is `root:kvm 0660` on Debian-family hosts, where adding the account to
+   the `kvm` group suffices, and `root:root 0600` on several others. If the
+   account lacks access, QEMU exits at launch and every affected VM stops
+   restarting. The VMM warns at startup when its own access fails, but it
+   cannot refuse on that basis — QEMU need not share its credentials.
+
+2. **Restart `netd` before or together with the VMM.** Multiqueue bridge NICs
+   are prepared by `netd`, and the VMM checks that `netd` echoes the queue
+   count it built. An older `netd` fails that check; the launch is rolled back
+   and fails with the reason in the VMM log, but the VM does not start until
+   `netd` is upgraded.
+
+3. **Roll back by setting `vhost = false`.** The node value reaches every VM
+   that did not pin `vhost` explicitly, from its next boot; a VM that pinned
+   `vhost = true` keeps it until updated.
+
 ## What each mode supports
 
 | Mode | netdev | vhost | queues > 1 |
@@ -144,8 +176,8 @@ vhost switches bridge mode to a `tap` netdev driven by the same setuid
 no compiled-in default path for the `tap` netdev, so the VMM probes the known
 distribution locations; set `cvm.qemu_bridge_helper` if yours is elsewhere. If
 no helper is found the NIC falls back to the non-vhost `bridge` netdev with a
-warning, because vhost is a default and a default must not stop a node from
-booting VMs.
+warning, because a node-wide setting must not stop a node from booting VMs
+that never asked for it.
 
 The helper returns exactly one descriptor, which is why more than one queue
 pair in bridge mode is created by `netd` instead: it adds a persistent
@@ -224,10 +256,11 @@ the same calculation, against the node configuration and manifest as they stand
 now, rather than the ones a finished boot ran under.
 
 If that account lacks access, QEMU exits at startup and the VM never boots. The
-VMM does not pre-check this: QEMU need not share the VMM's credentials, so
-refusing a launch on the VMM's own access would block deployments the host can
-run. It only warns when the device node is missing outright, which is a fact
-about the host rather than about either account.
+VMM does not refuse a launch over this: QEMU need not share the VMM's
+credentials, so a refusal based on the VMM's own access would block deployments
+the host can run. It warns instead — when the device node is missing outright,
+and when the VMM's own open is denied, since QEMU usually does share its
+account.
 
 vhost-net works normally in a TDX guest: the virtio rings and buffers live in
 shared, unencrypted memory precisely so a host-side backend can reach them.