|
| 1 | +# virtio-net data plane tuning |
| 2 | + |
| 3 | +Every CVM NIC has two knobs that decide how many packets it can move: whether |
| 4 | +the host kernel's vhost-net data plane is used, and how many virtio-net queue |
| 5 | +pairs the device exposes. Both are set per node and overridable per VM. |
| 6 | + |
| 7 | +## Why it matters |
| 8 | + |
| 9 | +Without vhost-net, QEMU drains every received packet on its single main-loop |
| 10 | +thread. That thread is the ceiling, and it does not grow with vCPUs: |
| 11 | + |
| 12 | +``` |
| 13 | +maximum packets per second ≈ 1 core ÷ per-packet main-loop cost |
| 14 | +``` |
| 15 | + |
| 16 | +The per-packet cost varies with traffic shape — a few microseconds for uniform |
| 17 | +synthetic streams, tens of microseconds for bidirectional short-connection |
| 18 | +traffic — so the ceiling is a property of the workload, not a fixed number. |
| 19 | +What is fixed is the shape of the failure: throughput climbs normally until the |
| 20 | +main loop saturates at 100% of one core, then packets are dropped at the TAP |
| 21 | +before they ever reach the guest. Guest-side counters stay clean, which makes |
| 22 | +the cliff easy to misdiagnose as a network problem. |
| 23 | + |
| 24 | +`vhost=on` moves that work into the host kernel. Guest-side outbound traffic |
| 25 | +uses the same thread, so a busy guest pays the cost twice over. |
| 26 | + |
| 27 | +That returns a whole core, but it relocates the ceiling rather than removing |
| 28 | +it: packets now arrive faster than a single guest receive queue can drain, and |
| 29 | +the drops reappear at a higher rate. Raising `queues` is what removes them. |
| 30 | +Enable vhost everywhere; raise the queue count for VMs that are actually |
| 31 | +throughput-bound. |
| 32 | + |
| 33 | +## Configuration |
| 34 | + |
| 35 | +```toml |
| 36 | +[cvm] |
| 37 | +# Ceiling for both the default and what a deployment may request. |
| 38 | +max_net_queues = 16 |
| 39 | + |
| 40 | +[cvm.networking] |
| 41 | +mode = "bridge" |
| 42 | +bridge = "dstack-br0" |
| 43 | +vhost = true |
| 44 | +``` |
| 45 | + |
| 46 | +Queue pairs are not a node setting. They default to the VM's vCPU count, capped |
| 47 | +at 16, because the useful number follows the VM rather than the host — the guest |
| 48 | +driver uses at most one queue pair per vCPU. A deployment overrides that per VM, |
| 49 | +up to `max_net_queues`. |
| 50 | + |
| 51 | +Raising `max_net_queues` above 16 widens what a deployment may ask for without |
| 52 | +moving the default's cap, so a larger VM never silently acquires a worse |
| 53 | +default. Lowering it below 16 does lower the default too, because a node that |
| 54 | +refuses a request for four queue pairs should not hand out sixteen by itself. |
| 55 | +The hard ceiling from any source is 64. |
| 56 | + |
| 57 | +Turning vhost off also turns the multiqueue default off. Without vhost the QEMU |
| 58 | +main loop drains every queue on one thread, so extra queues buy little while |
| 59 | +still costing a netd interface, more MSI-X vectors, and a changed guest device. |
| 60 | +An explicit queue count is still honoured without vhost, since that combination |
| 61 | +is a deliberate request rather than a default. |
| 62 | + |
| 63 | +A VM overrides either value at deploy time, and `UpdateVm` changes them |
| 64 | +afterwards — the new values apply from the VM's next boot: |
| 65 | + |
| 66 | +```bash |
| 67 | +vmm-cli.py deploy --name my-vm --image dstack-0.5.9 --compose app.yaml \ |
| 68 | + --net bridge --net-queues 4 |
| 69 | +vmm-cli.py deploy --name latency-vm --image dstack-0.5.9 --compose app.yaml \ |
| 70 | + --net bridge --net-no-vhost |
| 71 | +``` |
| 72 | + |
| 73 | +The web UI exposes both per NIC in the deploy and update dialogs, alongside the |
| 74 | +networking mode. Both fields are also on `NetworkingConfig` in the deployment |
| 75 | +and update RPCs. A request that |
| 76 | +sets only `vhost`/`queues` keeps the node's own networking mode, so tuning does |
| 77 | +not force a caller to restate — or be allowed to choose — a backend. `queues` is |
| 78 | +rejected above the node's `max_net_queues`; `vhost` is not otherwise restricted, |
| 79 | +since it only affects the requesting VM. `GetMeta` reports |
| 80 | +`networking.max_queues` so a client can present the real bound. |
| 81 | + |
| 82 | +Only what a deployment explicitly asks for is recorded against the VM. Values it |
| 83 | +inherits stay owned by the node, so changing `[cvm.networking]` later — including |
| 84 | +setting `vhost = false` to roll the whole node back — still reaches VMs that were |
| 85 | +deployed with some other networking override. A request that names a backend |
| 86 | +does pin that backend's bridge or parent for the life of the VM, as before; one |
| 87 | +that only tunes pins nothing. |
| 88 | + |
| 89 | +Neither field changes the CVM's measurement. `mr_config_id` covers the compose |
| 90 | +hash and instance info, so retuning a NIC does not change app identity or |
| 91 | +require an on-chain update. |
| 92 | + |
| 93 | +## What each mode supports |
| 94 | + |
| 95 | +| Mode | netdev | vhost | queues > 1 | |
| 96 | +|---|---|---|---| |
| 97 | +| `user` | `user,...` | no backend, ignored | not supported, ignored | |
| 98 | +| `bridge` | `bridge,br=` or `tap,br=,helper=` | yes | yes, through netd | |
| 99 | +| `bridge` with libvirt filtering | `tap,ifname=` | yes | yes, through netd | |
| 100 | +| `macvtap` | `tap,fd=` / `tap,fds=` | yes | yes | |
| 101 | +| `custom` | operator's own string | operator's own string | device line only | |
| 102 | + |
| 103 | +QEMU's `bridge` netdev accepts neither `vhost=` nor `queues=`, so enabling |
| 104 | +vhost switches bridge mode to a `tap` netdev driven by the same setuid |
| 105 | +`qemu-bridge-helper`. The VMM still needs no network privileges. The helper has |
| 106 | +no compiled-in default path for the `tap` netdev, so the VMM probes the known |
| 107 | +distribution locations; set `cvm.qemu_bridge_helper` if yours is elsewhere. If |
| 108 | +no helper is found the NIC falls back to the non-vhost `bridge` netdev with a |
| 109 | +warning, because vhost is a default and a default must not stop a node from |
| 110 | +booting VMs. |
| 111 | + |
| 112 | +The helper returns exactly one descriptor, which is why more than one queue |
| 113 | +pair in bridge mode is created by `netd` instead: it adds a persistent |
| 114 | +`multi_queue` TAP that QEMU then opens once per queue. That applies whether or |
| 115 | +not libvirt filtering is on, so a bridge node needs `netd` to get the default |
| 116 | +queue count (see [libvirt-network-filter.md](libvirt-network-filter.md)). |
| 117 | +Without it, bridge NICs fall back to a single queue pair with a warning rather |
| 118 | +than failing to launch; a VM that asked for a queue count explicitly still |
| 119 | +fails, so the caller learns their request was not met. `netd` is probed by |
| 120 | +connecting, not by looking for its socket file, because a `netd` that died |
| 121 | +leaves the socket behind. One-shot `dstack-vmm run` has no netd lifecycle at |
| 122 | +all and behaves like a node without it. `netd` reports back the |
| 123 | +queue count it created, and the VMM refuses to launch on a mismatch — a `netd` |
| 124 | +deployed separately as a root service can be older than the VMM asking it for |
| 125 | +multiqueue, and QEMU would otherwise reject the interface from inside the |
| 126 | +per-VM launcher. |
| 127 | + |
| 128 | +For macvtap, the per-VM launcher opens the `/dev/tapN` character device once |
| 129 | +per queue pair and hands QEMU the descriptors as `fds=`. `netd` creates the |
| 130 | +interface with matching `numtxqueues`/`numrxqueues`. |
| 131 | + |
| 132 | +Custom mode owns its whole netdev string, including any `vhost=`/`queues=` |
| 133 | +options. The `queues` field still applies to the generated device line, which |
| 134 | +is what makes a hand-written netdev usable with multiqueue. |
| 135 | + |
| 136 | +## Choosing a queue count |
| 137 | + |
| 138 | +The default suits bandwidth-bound workloads. Latency-sensitive ones should ask |
| 139 | +for fewer: more queues spread receive processing over more vCPUs, and under TDX |
| 140 | +a cross-vCPU wakeup costs an IPI and a VM exit. Measured on one 8-vCPU TDX CVM, |
| 141 | +changing only the guest's channel count: |
| 142 | + |
| 143 | +| Queue pairs | Short-connection throughput | |
| 144 | +|---|---| |
| 145 | +| 1 | 22.3k conn/s | |
| 146 | +| 2 | ~20k conn/s | |
| 147 | +| 4 | 15–21k conn/s | |
| 148 | +| 8 | 6.2–7.7k conn/s | |
| 149 | + |
| 150 | +The same CVM with 8 queues moved 3.0 Mpps of 64-byte UDP with no loss, against |
| 151 | +roughly 600k with one queue. The trade is real in both directions, so a VM |
| 152 | +serving many short connections should set `--net-queues 1` and measure. |
| 153 | + |
| 154 | +A VM with fewer vCPUs than queues leaves the extra pairs idle — `ethtool -l |
| 155 | +eth0` reports the smaller number. An explicit over-provision is not rejected at |
| 156 | +deployment, because `vmm-cli.py resize` can raise the vCPU count later. |
| 157 | + |
| 158 | +`vectors` is derived, never configured: `2N + 2`, one vector per queue |
| 159 | +direction plus config and control. One queue pair emits no `mq=on` or |
| 160 | +`vectors=` at all, leaving the historical command line byte for byte identical. |
| 161 | + |
| 162 | +## Requirements |
| 163 | + |
| 164 | +The account running QEMU must be able to open `/dev/vhost-net`, which is |
| 165 | +`root:kvm 0660` on a stock host — add that account to the `kvm` group. The |
| 166 | +`vhost_net` module autoloads on first open. |
| 167 | + |
| 168 | +`GetInfo` reports the data plane each interface actually got, so a bridge NIC |
| 169 | +that fell back for want of a helper reads as `vhost: false` rather than |
| 170 | +advertising something it is not using. |
| 171 | + |
| 172 | +If that account lacks access, QEMU exits at startup and the VM never boots. The |
| 173 | +VMM does not pre-check this: QEMU need not share the VMM's credentials, so |
| 174 | +refusing a launch on the VMM's own access would block deployments the host can |
| 175 | +run. It only warns when the device node is missing outright, which is a fact |
| 176 | +about the host rather than about either account. |
| 177 | + |
| 178 | +vhost-net works normally in a TDX guest: the virtio rings and buffers live in |
| 179 | +shared, unencrypted memory precisely so a host-side backend can reach them. |
| 180 | +This is the same mechanism behind `vhost-vsock-pci`, which dstack has always |
| 181 | +used. |
| 182 | + |
| 183 | +On host kernels older than 6.4 the vhost worker is a free-standing kernel |
| 184 | +thread and its CPU time is not charged to the VM's cgroup. Since 6.4 it is a |
| 185 | +`vhost_task` inside the QEMU thread group, so `cpu.max`, cgroup accounting, and |
| 186 | +`top -H` all attribute it to the CVM. Check the host kernel before relying on |
| 187 | +CPU limits for tenant VMs. |
0 commit comments