|
| 1 | +--- |
| 2 | +title: Migration Stream Encryption with kernel TLS (kTLS) |
| 3 | +layout: default |
| 4 | +design_doc: true |
| 5 | +revision: 1 |
| 6 | +status: proposed |
| 7 | +--- |
| 8 | +# Migration Stream Encryption with kernel TLS (kTLS) |
| 9 | + |
| 10 | +VM-migrate sends the guest memory over a TLS connection so the data is |
| 11 | +encrypted on the wire. Today that TLS is provided by stunnel, an external |
| 12 | +process. This document describes an alternative that keeps the same TLS |
| 13 | +security but removes stunnel from the sender's data path, using the kernel's |
| 14 | +own TLS (kTLS), and so makes vm-migrate and host-evacuate faster. |
| 15 | + |
| 16 | +## Existing transport: stunnel |
| 17 | + |
| 18 | +stunnel is an external process that terminates TLS for the xenguest migration |
| 19 | +stream. As it is a separate process from xenguest, the dom0 kernel has to pipe |
| 20 | +the plaintext between the two, so every byte of guest RAM is copied through an |
| 21 | +extra userspace hop and encrypted in userspace. That wastes dom0 cpu and memory |
| 22 | +throughput, so the result is a slower vm-migrate, and therefore a slower |
| 23 | +host-evacuate for the user. The slowdown is worst exactly when dom0 cpu is the |
| 24 | +bottleneck. |
| 25 | + |
| 26 | +stunnel cannot be removed by linking it into xenguest, as it is not a library: |
| 27 | +it is a configuration wrapper around OpenSSL. |
| 28 | + |
| 29 | +## Solution: kTLS on the sender |
| 30 | + |
| 31 | +The data pipe between xenguest and stunnel disappears if the kernel does the |
| 32 | +bulk TLS encryption in place, on the same socket xenguest already writes to. |
| 33 | +Linux 6.6 supports this through kTLS: once the symmetric key is installed on a |
| 34 | +socket with `setsockopt(SOL_TLS, ...)`, the kernel encrypts/decrypts every |
| 35 | +subsequent `read`/`write` transparently, using AES-NI, producing the same |
| 36 | +byte stream stunnel produces today. |
| 37 | + |
| 38 | +Phase 1 changes only the sender host and already provides the speed-up (see |
| 39 | +Performance), while proving the kTLS sender interoperates with the unchanged |
| 40 | +stunnel receiver on the destination. Benefits: |
| 41 | + |
| 42 | +* significant speed-up: host-evacuate is around 1.5x faster for 10 parallel |
| 43 | + VMs, for less dom0 cpu (see Performance). |
| 44 | +* xenguest is unmodified: no libssl in xenguest, so no extra xen-devel |
| 45 | + upstreaming, maintenance or security reviews on xenguest/libxenguest. |
| 46 | +* same TLS security as stunnel: a small `ktls-helper` does the OpenSSL |
| 47 | + handshake with the same key size, cipher, certificate and verification stunnel |
| 48 | + uses, then asks the kernel to take over the bulk encryption. |
| 49 | +* backwards-compatible & conservative: stunnel stays the default; kTLS is an |
| 50 | + option, introduced gradually until it is proven in the field. |
| 51 | + |
| 52 | +## Principles |
| 53 | + |
| 54 | +* P1. kernel data pipes between userspace processes add latency and cut data |
| 55 | + throughput in dom0. |
| 56 | +* P2. keep the existing security guarantees. |
| 57 | +* P3. backwards-compatibility. |
| 58 | +* P4. guarded new features. |
| 59 | + |
| 60 | +## Use cases |
| 61 | + |
| 62 | +* U1. Admin configuration: |
| 63 | + * U1.1. a new `migration_ktls` option selects the faster datapath for |
| 64 | + vm-migrate and host-evacuate (P1, P2). |
| 65 | + * U1.2. the option can be reverted to the original stunnel datapath (P3). |
| 66 | +* U2. Admin usage when the option is selected: |
| 67 | + * U2.1. vm-migrate is faster, or at least the same (P1). |
| 68 | + * U2.2. host-evacuate is faster, or at least the same (P1). |
| 69 | + * U2.3. vm-migrate and host-evacuate still work between hosts configured |
| 70 | + with different options (P3). |
| 71 | +* U3. Over time the option may graduate to become the default for vm-migrate |
| 72 | + and host-evacuate (P4). |
| 73 | + |
| 74 | +## Requirements |
| 75 | + |
| 76 | +* R1. Admin configuration: |
| 77 | + * R1.1.1. Host-level: `xe-enable-experimental-feature migration_ktls` sets |
| 78 | + `/etc/xenserver/features.d/migration_ktls` to 1 on the host, enabling the |
| 79 | + option there. |
| 80 | + * R1.1.2. Pool-level: when every host has the entry, the matching |
| 81 | + `restrict_migration_ktls` shown by `xe pool-list params=restrictions` |
| 82 | + reads `false`. |
| 83 | + * R1.1.3. Pool-level (future): a helper `xe pool-experimental-feature-set |
| 84 | + name=migration_ktls` sets the entry on each host of the pool. |
| 85 | +* R2. Admin usage: |
| 86 | + * R2.1. vm-migrate: when the option is enabled, xenopsd uses |
| 87 | + `ktls-helper` instead of stunnel to set up kTLS transport for |
| 88 | + xenguest (see Design). |
| 89 | + * R2.2. host-evacuate: the vm-migrate operations it drives inherit R2.1. |
| 90 | +* R3. Future: once kTLS is the default, a XenAPI field |
| 91 | + `pool.migration_transport` could expose `ktls` (default) or `stunnel`. |
| 92 | + |
| 93 | +Implementation note: R1.1.1 describes the intended activation. The current |
| 94 | +implementation does not yet read `features.d/migration_ktls`; it selects the |
| 95 | +transport per host from `xenopsd.conf` instead: |
| 96 | + |
| 97 | +``` |
| 98 | +migration-tls = "ktls" # use the helper |
| 99 | +migration-tls = "stunnel" # explicit default |
| 100 | +migration-tls = "" # currently defaults to "stunnel" |
| 101 | +``` |
| 102 | + |
| 103 | +Wiring `xe-enable-experimental-feature migration_ktls` to this `xenopsd.conf` |
| 104 | +option (so the host flag drives R1.1.1/R1.1.2) is outstanding. |
| 105 | + |
| 106 | +## Design |
| 107 | + |
| 108 | +### Considered designs |
| 109 | + |
| 110 | +The goal is to remove the plaintext data pipe between stunnel and xenguest. |
| 111 | + |
| 112 | +| Design | Summary | Analysis | |
| 113 | +| --- | --- | --- | |
| 114 | +| A: TLS inside xenguest/libxenguest | link libssl into xenguest; xenguest does `SSL_connect`/`SSL_accept` and `SSL_read`/`SSL_write` directly. Needs an SNI dispatcher (eg. sniproxy) on the receiver to share port 443. | Removes the pipe, but embeds TLS in xenguest: lots of changes, lots of xen-devel upstreaming, and ongoing security-maintenance on xenguest/libxenguest. Too expensive to create and maintain. | |
| 115 | +| B: xenguest as a stunnel SNI backend | use stunnel's SNI dispatch to route migration traffic to xenguest. | Easy (stunnel config only), but the plaintext pipe between stunnel and the backend remains, so it defeats the goal. | |
| 116 | +| C: kTLS via a small TLS helper | `ktls-helper` does the OpenSSL handshake, asks OpenSSL to enable kTLS on the socket, and hands the kTLS socket fd to xenopsd over SCM_RIGHTS. xenguest then sees only plaintext; the kernel encrypts transparently. | xenguest is unmodified, stunnel leaves the sender data path, the helper reuses stunnel's OpenSSL config/keys/certificates, and kTLS can later be offloaded to hardware NICs. | |
| 117 | + |
| 118 | +Conclusion: design C (kTLS) seems superior, as it reaches the goal with fewer |
| 119 | +changes, is toolstack-only (no xen-devel upstream loop, no future xenguest |
| 120 | +security maintenance), is a small focused tool rather than a change to the |
| 121 | +highly-complex xenguest, leaves the stunnel option in place, and opens the way |
| 122 | +to kTLS hardware offload later. |
| 123 | + |
| 124 | +### kTLS sender |
| 125 | + |
| 126 | +Performance: the kernel encrypts the migration stream in the xenguest context, |
| 127 | +so dom0 no longer copies the whole guest RAM between xenguest and stunnel. The |
| 128 | +effect is largest where dom0 cpu, not the wire, is the bottleneck. |
| 129 | + |
| 130 | +Security: the helper uses OpenSSL exactly as stunnel does, so the bulk kTLS |
| 131 | +stream is byte-identical to stunnel's and is accepted by the unchanged |
| 132 | +destination stunnel. |
| 133 | + |
| 134 | +* same handshake: TLS 1.2, with the cipher list and ECDHE curve taken from the |
| 135 | + pool TLS policy (`Stunnel.Openssl`, the same source the stunnel client reads), |
| 136 | + renegotiation disabled, authenticated with `SSL_VERIFY_PEER` |
| 137 | + against the destination's pool-internal certificate (CN = host uuid, |
| 138 | + SNI = `pool`). |
| 139 | +* single source of truth: the helper's verification is derived from the same |
| 140 | + configuration the stunnel fallback would use, so `--cert-bundle-file`/`--sni` come from it |
| 141 | + and `--no-verify` is sent only when verification is disabled pool-wide. SNI is |
| 142 | + always sent, so the destination serves its pool-internal certificate either |
| 143 | + way. |
| 144 | +* the TLS 1.2 pin is necessary for kTLS, as a TLS 1.3 post-handshake |
| 145 | + KeyUpdate cannot be carried by the kernel kTLS data path. |
| 146 | +* no new security code in xenguest: after the handshake OpenSSL installs the |
| 147 | + symmetric key in the kernel, and everything xenguest writes as plaintext is |
| 148 | + encrypted transparently. |
| 149 | +* the helper hands the socket over only after confirming the kernel actually |
| 150 | + enabled kTLS in both directions (`BIO_get_ktls_send` and `BIO_get_ktls_recv`), |
| 151 | + as the migration channel is bidirectional; otherwise it errors and the sender |
| 152 | + falls back to stunnel. |
| 153 | + |
| 154 | +Backwards-compatibility: the new behaviour is hidden behind the per-host option, |
| 155 | +and a kTLS sender works with a stunnel receiver, so the worst case is exactly |
| 156 | +today's behaviour. |
| 157 | + |
| 158 | +### The helper |
| 159 | + |
| 160 | +`ktls-helper` does only what stunnel cannot, then gets out of the way: |
| 161 | + |
| 162 | +1. perform the TLS handshake with OpenSSL (same protocol/cipher/cert/verification |
| 163 | + as stunnel). |
| 164 | +2. ask OpenSSL to enable kTLS on the socket. |
| 165 | +3. confirm the kernel took over send and receive (`BIO_get_ktls_send`/`_recv`); |
| 166 | + if not, error so the sender falls back to stunnel. |
| 167 | +4. hand the now-encrypting socket to xenopsd over SCM_RIGHTS, then exit. |
| 168 | + |
| 169 | +The helper is not a proxy and is not in the data path: it is a short-lived "set |
| 170 | +up the encrypted socket, hand it over, get out of the way" step. xenopsd brokers |
| 171 | +the fd to xenguest just as it brokers the stunnel fd today, and xenguest writes |
| 172 | +the migration stream as plaintext while the kernel encrypts each `write` in |
| 173 | +place. |
| 174 | + |
| 175 | +### Why stunnel cannot remove the data pipe |
| 176 | + |
| 177 | +stunnel is a general-purpose TLS proxy, and a proxy must receive plaintext on |
| 178 | +one side to encrypt it on the other, so the xenguest -> stunnel plaintext hop is |
| 179 | +structural and cannot be removed while stunnel is in the path. stunnel has no |
| 180 | +pass-through mode and cannot hand its encrypted socket to another process, so its |
| 181 | +bulk encryption stays in userspace. |
| 182 | + |
| 183 | +An alternative considered but rejected was to make stunnel use kTLS and then pull |
| 184 | +its kernel-encrypting socket out of the stunnel process with `pidfd_getfd(2)`. |
| 185 | +This has lots of issues: it needs ptrace-level privilege over stunnel and racy |
| 186 | +fd-table scraping; there is no clean ownership handoff, as the TLS sequence |
| 187 | +number and any partial record are per-socket and two writers corrupt the stream; |
| 188 | +and nobody owns the TLS control path, so an inbound TLS 1.3 KeyUpdate stalls the |
| 189 | +receive side. The safe form of "hand a kTLS socket to the datapath" is precisely |
| 190 | +the dedicated helper above. |
| 191 | + |
| 192 | +## Performance |
| 193 | + |
| 194 | +Single host-evacuate, 10 VMs migrating in parallel, sender kTLS vs sender |
| 195 | +stunnel, same host pair. The hosts have Intel Xeon Gold 6430 cpus (128 pCPU, |
| 196 | +16 dom0 vCPUs) on a 25 GbE link, and each guest is Windows Server 2019 with |
| 197 | +64 GB RAM and 12 vCPUs. The improvement grows with guest load, as a busier |
| 198 | +guest has more memory to transmit and pushes dom0 cpu harder: |
| 199 | + |
| 200 | +| Guest load | stunnel | kTLS | Improvement | |
| 201 | +| --- | --- | --- | --- | |
| 202 | +| idle | 185s | 163s | 1.13x | |
| 203 | +| medium (windows apps) | 249s | 171s | 1.46x | |
| 204 | +| high (synthetic page thrasher) | 1341s | 835s | 1.60x | |
| 205 | + |
| 206 | +host-evacuate improvement is around 1.5x for 10 parallel VMs, for |
| 207 | +lower dom0 cpu (eg. on the medium load, mean dom0 cpu drops from ~0.73 to ~0.53 |
| 208 | +of the 16 dom0 vCPUs). The improvement is smaller for idle guests, as dom0 is |
| 209 | +then not the bottleneck. |
| 210 | + |
| 211 | +## Implementation |
| 212 | + |
| 213 | +The sender path is in xenopsd. Outside the helper itself, the change is small: |
| 214 | + |
| 215 | +* `ocaml/ktls-helper/helper/` — the standalone C helper, with its |
| 216 | + Makefile and README. It is built on its own (`make`) and deployed to |
| 217 | + `/usr/libexec/xapi/ktls-helper`. |
| 218 | +* `ocaml/xenopsd/lib/migrate_connect.ml` — a drop-in replacement for |
| 219 | + `Open_uri.with_open_uri` that spawns the helper when `migration-tls = "ktls"` |
| 220 | + and falls back to `Open_uri` otherwise. |
| 221 | +* `ocaml/xenopsd/lib/xenops_server.ml` — the three migration fd call sites |
| 222 | + (vm, mem, vgpu) in the `VM_migrate` branch go through `Migrate_connect`. |
| 223 | +* `ocaml/xenopsd/lib/xenopsd.ml`, `ocaml/xenopsd/xc/xc_resources.ml`, |
| 224 | + `ocaml/xenopsd/xenopsd.conf` — register the `migration-tls` option and the |
| 225 | + `xenopsd-tls-helper` resource, and document them. |
| 226 | + |
| 227 | +Fallback: the sender falls back to stunnel only when the kTLS path fails to |
| 228 | +produce the fd (helper spawn, TLS handshake, kTLS install or SCM_RIGHTS). It logs |
| 229 | +a single warn line and uses stunnel for that one connection, so the migration |
| 230 | +still proceeds. Once the fd is handed to the migration, any later exception is a |
| 231 | +migration-layer failure and propagates unchanged, as silently retrying it over a |
| 232 | +fresh stunnel socket would re-enter the in-progress receive on the destination |
| 233 | +and corrupt its state. |
| 234 | + |
| 235 | +## Upgrade |
| 236 | + |
| 237 | +The kTLS sender produces the same TLS stream as stunnel and the receiver is |
| 238 | +unchanged, so a kTLS-enabled host migrates to a stunnel host with no |
| 239 | +coordination. xapi only permits migration from older to newer toolstacks, so a |
| 240 | +newer kTLS sender is never required by an older receiver. The option is off by |
| 241 | +default, so an upgrade changes nothing until an admin enables it. |
| 242 | + |
| 243 | +## Applicability |
| 244 | + |
| 245 | +Phase 1 covers the sender side of the guest-memory migration stream (the vm, |
| 246 | +mem and vgpu fds in `VM_migrate`). It leaves the receiver and the other TLS |
| 247 | +users unchanged: SMAPIv1 storage migration uses `sparse_dd` and SMAPIv3 uses |
| 248 | +stunnel, both untouched, as are RRD and other stunnel traffic. Phase 2 (the |
| 249 | +kTLS receiver) would extend the same treatment to the destination. |
| 250 | + |
| 251 | +## Outlook |
| 252 | + |
| 253 | +* Phase 2: a kTLS receiver, so the destination also drops stunnel from the data |
| 254 | + path. It needs an SNI dispatcher (eg. sniproxy) to share port 443 between the |
| 255 | + receiver and the existing stunnel endpoint. |
| 256 | +* kTLS hardware-offload NICs (`CONFIG_TLS_DEVICE=y` in kernel) could move the bulk |
| 257 | + encryption off the cpu. This depends on the NIC and the network topology: the |
| 258 | + offload applies where the migration socket terminates in a TLS-capable NIC (for |
| 259 | + example a raw or bonded NIC), but not where the traffic is bridged (kernel or Open |
| 260 | + vSwitch). The gain grows with line rate: at 100 GbE and above, software AES-GCM |
| 261 | + (even with AES-NI/AVX2 or VAES/AVX-512) becomes the bottleneck on a single core. |
| 262 | +* `pool.migration_transport` (R3) once kTLS becomes the default. |
| 263 | +* mutual TLS (client-certificate auth) and destination-hostname pinning exceed |
| 264 | + the stunnel migration client and can be done with the Phase 2 receiver work. |
| 265 | + |
0 commit comments