|
| 1 | +# virtio-net data plane tuning |
| 2 | + |
| 3 | +Every CVM NIC has two knobs that decide how many packets it can move: whether |
| 4 | +the host kernel's vhost-net data plane is used, and how many virtio-net queue |
| 5 | +pairs the device exposes. vhost is set per node and overridable per VM; queue |
| 6 | +pairs have no node-wide setting at all, for the reason given under |
| 7 | +Configuration. |
| 8 | + |
| 9 | +## Why it matters |
| 10 | + |
| 11 | +Without vhost-net, QEMU drains every received packet on its single main-loop |
| 12 | +thread. That thread is the ceiling, and it does not grow with vCPUs: |
| 13 | + |
| 14 | +``` |
| 15 | +maximum packets per second ≈ 1 core ÷ per-packet main-loop cost |
| 16 | +``` |
| 17 | + |
| 18 | +The per-packet cost varies with traffic shape — a few microseconds for uniform |
| 19 | +synthetic streams, tens of microseconds for bidirectional short-connection |
| 20 | +traffic — so the ceiling is a property of the workload, not a fixed number. |
| 21 | +What is fixed is the shape of the failure: throughput climbs normally until the |
| 22 | +main loop saturates at 100% of one core, then packets are dropped at the TAP |
| 23 | +before they ever reach the guest. Guest-side counters stay clean, which makes |
| 24 | +the cliff easy to misdiagnose as a network problem. |
| 25 | + |
| 26 | +Guest-side outbound traffic uses the same thread, so a busy guest pays the |
| 27 | +cost twice over. |
| 28 | + |
| 29 | +`vhost=on` moves that work into the host kernel. That returns a whole core, but |
| 30 | +it relocates the ceiling rather than removing |
| 31 | +it: packets now arrive faster than a single guest receive queue can drain, and |
| 32 | +the drops reappear at a higher rate. More queue pairs is what removes them, |
| 33 | +which is why both are defaults: vhost everywhere, and a queue count that follows |
| 34 | +the VM's vCPU count. The knob you are more likely to reach for is the other |
| 35 | +direction — see [Choosing a queue count](#choosing-a-queue-count). |
| 36 | + |
| 37 | +## Configuration |
| 38 | + |
| 39 | +```toml |
| 40 | +[cvm] |
| 41 | +# Ceiling for both the default and what a deployment may request. |
| 42 | +max_net_queues = 16 |
| 43 | + |
| 44 | +[cvm.networking] |
| 45 | +mode = "bridge" |
| 46 | +bridge = "dstack-br0" |
| 47 | +vhost = true |
| 48 | +``` |
| 49 | + |
| 50 | +Queue pairs are not a node setting. They default to the VM's vCPU count, capped |
| 51 | +at 16, because the useful number follows the VM rather than the host — the guest |
| 52 | +driver uses at most one queue pair per vCPU. A deployment overrides that per VM, |
| 53 | +up to `max_net_queues`. |
| 54 | + |
| 55 | +Raising `max_net_queues` above 16 widens what a deployment may ask for without |
| 56 | +moving the default's cap, so a larger VM never silently acquires a worse |
| 57 | +default. Lowering it below 16 does lower the default too, because a node that |
| 58 | +refuses a request for four queue pairs should not hand out sixteen by itself. |
| 59 | +The hard ceiling from any source is 64. |
| 60 | + |
| 61 | +Turning vhost off also turns the multiqueue default off. Without vhost the QEMU |
| 62 | +main loop drains every queue on one thread, so extra queues buy little while |
| 63 | +still costing a netd interface, more MSI-X vectors, and a changed guest device. |
| 64 | +An explicit queue count is still honoured without vhost, since that combination |
| 65 | +is a deliberate request rather than a default. |
| 66 | + |
| 67 | +A VM overrides either value at deploy time, and `UpdateVm` changes them |
| 68 | +afterwards — the new values apply from the VM's next boot: |
| 69 | + |
| 70 | +```bash |
| 71 | +vmm-cli.py deploy --name my-vm --image dstack-0.5.9 --compose app.yaml \ |
| 72 | + --net bridge --net-queues 4 |
| 73 | +vmm-cli.py deploy --name latency-vm --image dstack-0.5.9 --compose app.yaml \ |
| 74 | + --net bridge --net-no-vhost |
| 75 | +``` |
| 76 | + |
| 77 | +The web UI exposes both per NIC in the deploy and update dialogs, alongside the |
| 78 | +networking mode. Both fields are also on `NetworkingConfig` in the deployment |
| 79 | +and update RPCs. A request that |
| 80 | +sets only `vhost`/`queues` keeps the node's own networking mode, so tuning does |
| 81 | +not force a caller to restate — or be allowed to choose — a backend. `queues` is |
| 82 | +rejected above the node's `max_net_queues`; `vhost` is not otherwise restricted, |
| 83 | +since it only affects the requesting VM. `GetMeta` reports |
| 84 | +`networking.max_queues` so a client can present the real bound. |
| 85 | + |
| 86 | +The data plane settings are recorded only when a deployment asks for them. |
| 87 | +Leave one out and it stays owned by the node, so changing `[cvm.networking]` |
| 88 | +later — including setting `vhost = false` to roll the whole node back — still |
| 89 | +reaches VMs deployed with some other networking override. |
| 90 | + |
| 91 | +Naming a backend is different: it pins that NIC's identity, resolved at |
| 92 | +deployment. Its bridge or macvtap parent, its user-mode subnet and DHCP start, |
| 93 | +and its MAC prefix are all fixed for the life of the VM, so a later edit to |
| 94 | +those fields in `[cvm.networking]` does not reach it. A request that only tunes |
| 95 | +pins nothing, including the backend it inherited. |
| 96 | + |
| 97 | +`GetInfo` reports that configuration back, and both `vmm-cli.py update` and the |
| 98 | +web UI read it, change one field, and resend the rest. Two things follow. A |
| 99 | +request may name a bridge or macvtap parent the node itself configured even when |
| 100 | +the allowlists are empty: leaving the field out already yields exactly that |
| 101 | +value, so echoing it grants nothing policy was withholding. And an update may |
| 102 | +restate whatever its own VM already pinned, so that moving the node's default |
| 103 | +out from under a VM does not leave that VM's configuration unsendable. A NIC |
| 104 | +that inherited its backend reports an empty mode, which is the same thing it was |
| 105 | +deployed with. |
| 106 | + |
| 107 | +Neither field reaches the CVM's measurement. The only measurement input the VMM |
| 108 | +controls is `mr_config_id`, which covers the compose hash and instance info, so |
| 109 | +retuning a NIC does not change app identity or require an on-chain update. |
| 110 | + |
| 111 | +## What each mode supports |
| 112 | + |
| 113 | +| Mode | netdev | vhost | queues > 1 | |
| 114 | +|---|---|---|---| |
| 115 | +| `user` | `user,...` | no backend | not supported | |
| 116 | +| `bridge` | `tap,ifname=` via netd, else `tap,br=,helper=`, else `bridge,br=` | yes | yes, through netd | |
| 117 | +| `bridge` with libvirt filtering | `tap,ifname=` | yes | yes, through netd | |
| 118 | +| `macvtap` | `tap,fd=` / `tap,fds=` | yes | yes | |
| 119 | +| `custom` | operator's own string | operator's own string | no, not settable | |
| 120 | + |
| 121 | +QEMU's `bridge` netdev accepts neither `vhost=` nor `queues=`, so enabling |
| 122 | +vhost switches bridge mode to a `tap` netdev driven by the same setuid |
| 123 | +`qemu-bridge-helper`. The VMM still needs no network privileges. The helper has |
| 124 | +no compiled-in default path for the `tap` netdev, so the VMM probes the known |
| 125 | +distribution locations; set `cvm.qemu_bridge_helper` if yours is elsewhere. If |
| 126 | +no helper is found the NIC falls back to the non-vhost `bridge` netdev with a |
| 127 | +warning, because vhost is a default and a default must not stop a node from |
| 128 | +booting VMs. |
| 129 | + |
| 130 | +The helper returns exactly one descriptor, which is why more than one queue |
| 131 | +pair in bridge mode is created by `netd` instead: it adds a persistent |
| 132 | +`multi_queue` TAP that QEMU then opens once per queue. `netd` requires the |
| 133 | +`virsh` binary to be installed even when nothing is filtered, though it does |
| 134 | +not require a reachable `libvirtd`. That applies whether or |
| 135 | +not libvirt filtering is on, so a bridge node needs `netd` to get the default |
| 136 | +queue count (see [libvirt-network-filter.md](libvirt-network-filter.md)). |
| 137 | +Without it, bridge NICs fall back to a single queue pair with a warning rather |
| 138 | +than failing to launch; a VM that asked for a queue count explicitly still |
| 139 | +fails, so the caller learns their request was not met. `netd` is probed by |
| 140 | +connecting, not by looking for its socket file, because a `netd` that died |
| 141 | +leaves the socket behind. One-shot `dstack-vmm run` has no netd lifecycle at |
| 142 | +all and behaves like a node without it. `netd` reports back the |
| 143 | +queue count it created, and the VMM refuses to launch on a mismatch — a `netd` |
| 144 | +deployed separately as a root service can be older than the VMM asking it for |
| 145 | +multiqueue, and QEMU would otherwise reject the interface from inside the |
| 146 | +per-VM launcher. |
| 147 | + |
| 148 | +For macvtap, the per-VM launcher opens the `/dev/tapN` character device once |
| 149 | +per queue pair and hands QEMU the descriptors as `fds=`. `netd` creates the |
| 150 | +interface with matching `numtxqueues`/`numrxqueues`. |
| 151 | + |
| 152 | +Custom mode owns its whole netdev string, including any `vhost=`/`queues=` |
| 153 | +options, and its guest device stays single-queue: the VMM cannot edit that |
| 154 | +string, so it has no way to make a multiqueue device line agree with it. A |
| 155 | +hand-written multiqueue netdev will not pair with a multiqueue guest device |
| 156 | +today. |
| 157 | + |
| 158 | +Naming a backend that cannot carry vhost or a queue count, and then asking for |
| 159 | +one, is refused — the request is yours to correct. Inheriting such a backend is |
| 160 | +not, because the node chose it and may choose another tomorrow; the request |
| 161 | +reads as off, or as one queue pair, until then. |
| 162 | + |
| 163 | +## Choosing a queue count |
| 164 | + |
| 165 | +The default suits bandwidth-bound workloads. Latency-sensitive ones should ask |
| 166 | +for fewer: more queues spread receive processing over more vCPUs, and under TDX |
| 167 | +a cross-vCPU wakeup costs an IPI and a VM exit. Measured on one 8-vCPU TDX CVM, |
| 168 | +changing only the guest's channel count: |
| 169 | + |
| 170 | +| Queue pairs | Short-connection throughput | |
| 171 | +|---|---| |
| 172 | +| 1 | 22.3k conn/s | |
| 173 | +| 2 | ~20k conn/s | |
| 174 | +| 4 | 15–21k conn/s | |
| 175 | +| 8 | 6.2–7.7k conn/s | |
| 176 | + |
| 177 | +The same CVM with 8 queues moved 3.0 Mpps of 64-byte UDP with no loss, against |
| 178 | +roughly 600k with one queue. The trade is real in both directions, so a VM |
| 179 | +serving many short connections should set `--net-queues 1` and measure. |
| 180 | + |
| 181 | +A VM with fewer vCPUs than queues leaves the extra pairs idle — `ethtool -l |
| 182 | +eth0` reports the smaller number. An explicit over-provision is not rejected at |
| 183 | +deployment, because `vmm-cli.py resize` can raise the vCPU count later. |
| 184 | + |
| 185 | +`vectors` is derived, never configured: `2N + 2`, one vector per queue |
| 186 | +direction plus config and control. One queue pair emits no `mq=on` or |
| 187 | +`vectors=` at all, leaving the guest device line byte for byte identical to the |
| 188 | +one before this feature. The `-netdev` half does change wherever vhost is on, |
| 189 | +since that is what selects the backend. |
| 190 | + |
| 191 | +## Requirements |
| 192 | + |
| 193 | +The account running QEMU must be able to open `/dev/vhost-net`, which is |
| 194 | +`root:kvm 0660` on a stock host — add that account to the `kvm` group. The |
| 195 | +`vhost_net` module autoloads on first open. |
| 196 | + |
| 197 | +`GetInfo` reports the data plane each interface actually got, so a bridge NIC |
| 198 | +that fell back for want of a helper reads as `vhost: false` rather than |
| 199 | +advertising something it is not using. For a VM that is not running there is no |
| 200 | +interface to describe, so it reports what the next launch would build instead -- |
| 201 | +the same calculation, against the node configuration and manifest as they stand |
| 202 | +now, rather than the ones a finished boot ran under. |
| 203 | + |
| 204 | +If that account lacks access, QEMU exits at startup and the VM never boots. The |
| 205 | +VMM does not pre-check this: QEMU need not share the VMM's credentials, so |
| 206 | +refusing a launch on the VMM's own access would block deployments the host can |
| 207 | +run. It only warns when the device node is missing outright, which is a fact |
| 208 | +about the host rather than about either account. |
| 209 | + |
| 210 | +vhost-net works normally in a TDX guest: the virtio rings and buffers live in |
| 211 | +shared, unencrypted memory precisely so a host-side backend can reach them. |
| 212 | +This is the same mechanism behind `vhost-vsock-pci`, which dstack has always |
| 213 | +used. |
| 214 | + |
| 215 | +On host kernels older than 6.4 the vhost worker is a free-standing kernel |
| 216 | +thread: it is attached to the owner's cgroups, so `cpu.max` and cgroup |
| 217 | +accounting do apply, but it is outside QEMU's thread group and so invisible to |
| 218 | +`top -H` and to anything reading `/proc/<qemu>/task`. Since 6.4 it is a |
| 219 | +`vhost_task` inside that thread group and shows up everywhere the VM's other |
| 220 | +threads do. |
0 commit comments