Skip to content

Commit f19c423

Browse files
committed
docs: cut the host-provisioning detour from the AMD section
Section 9.4 had grown into a walkthrough of tooling that cannot be used -- snphost subcommands, a QEMU property dump, and an account of why the earlier recommendation was wrong. A runbook is read by someone mid-test; none of that helps them. What is left is the part that changes what a reader does: KDS is a single rate-limited endpoint, its 429 looks like a verification failure, the cache is in-process only, and the host-certificate route is a dead end on a stock QEMU so nobody should spend an afternoon on it.
1 parent cee7b5b commit f19c423

1 file changed

Lines changed: 34 additions & 87 deletions

File tree

docs/rc-testing-runbook.md

Lines changed: 34 additions & 87 deletions
Original file line numberDiff line numberDiff line change
@@ -429,87 +429,40 @@ certificate** — the onboarded KMS re-issues its own cert, so the PEM differs
429429
while `openssl x509 -pubkey` is identical. A KMS that bootstrapped independently
430430
differs in both.
431431

432-
### 9.4 Provision host certificates first
433-
434-
Verification needs the ASK and VCEK certificates. By default they are fetched
435-
from `https://kdsintf.amd.com`, which is a **single global endpoint with no
436-
mirror**, is aggressively rate-limited (one identical request per ~10s), and was
437-
completely unreachable for an entire test round — TCP timeouts and 100% ICMP loss
438-
from five vantage points across four autonomous systems and two continents, while
439-
every other external service answered normally. DNS was healthy and unchanged
440-
throughout, so this was the service, not a migration.
441-
442-
The rate limit is not theoretical: back-to-back negative tests against the same
443-
chip hit `HTTP status client error (429)` on the VCEK URL, which reads exactly
444-
like a verification failure until you notice the status code. Space repeat
445-
verifications ~25s apart, or provision the host.
446-
447-
dstack already supports the alternative: `normalize_kernel_cert_table` reads ASK
448-
and VCEK from the SNP extended report's certificate table, so a host that has
449-
been provisioned needs no KDS access at all. AMD's own specification recommends
450-
caching at the host for exactly this reason.
451-
452-
**There is no KDS cache service in this repo, and nothing to point one at.** The
453-
only caching is in-process: `AmdKdsClient` holds a `moka` cache of 16 CA chains
454-
and 1024 VCEKs, bounded by capacity with no TTL and no persistence, so it is warm
455-
only for the lifetime of a single KMS or verifier process. A one-shot
456-
`dstack-verifier --verify` starts cold every time, which is exactly how the 429
457-
above was earned. `[core.attestation.urls] amd_kds` accepts any KDS-compatible
458-
base URL, so a mirror would drop in — but you would have to write it. Compare
459-
`nvidia-attest-proxy`, which is the persistent, on-disk, PCCS-like cache the
460-
NVIDIA collateral got and AMD did not. (`crates/mock-attestation` serves
461-
KDS-shaped routes but is a test fixture with fake collateral, not a mirror.)
462-
463-
The guest side of the cert-table path needs nothing configured: `sev-snp-attest`
464-
reads `certs`, `cert_chain` or `auxblob` from the configfs TSM report, falling
465-
back to the extended-report ioctl, and passes whatever it finds up with the
466-
quote. An unprovisioned host simply yields an empty chain, and the verifier then
467-
goes to KDS — which is what happened here.
468-
469-
**Check that your VMM can actually deliver those certificates before planning
470-
around it.** On the host used for this round it cannot, and the earlier revision
471-
of this section recommended provisioning without checking.
472-
473-
The tooling half is fine. `snphost` (virtee) is the SEV-SNP host CLI:
474-
`snphost fetch ca [der|pem] DIR` and `snphost fetch vek [der|pem] DIR` pull the
475-
CA chain and the VCEK from KDS, and `snphost import DIR CERT-FILE` packs them
476-
into the GHCB-formatted blob that, in its own words, "can then be provided to
477-
QEMU to perform extended attestation on guests". (`sevctl` is *not* part of this
478-
— it manages the pre-SNP SEV platform and its OCA certificate chain. A previous
479-
revision of this section listed the two together; that was wrong.)
480-
481-
The delivery half is the problem. Providing that blob to QEMU needs a
482-
`certs-path`-style property on the `sev-snp-guest` object, and stock QEMU does
483-
not have one:
484-
485-
```
486-
$ qemu-system-x86_64 -object sev-snp-guest,help # QEMU 10.2.1
487-
sev-snp-guest options:
488-
author-key-enabled=<bool> guest-visible-workarounds=<string>
489-
host-data=<string> id-auth=<string>
490-
id-block=<string> kernel-hashes=<bool>
491-
policy=<uint64> sev-device=<string>
492-
vcek-disabled=<bool>
493-
```
494-
495-
`strings` on the binary confirms it — no `certs-path` anywhere. The host-side
496-
`SNP_SET_EXT_CONFIG` ioctl, the other way certificates used to be staged, is
497-
absent from upstream `linux/psp-sev.h` as well; it lived in AMD's out-of-tree
498-
fork. The guest half is present and working — `linux/sev-guest.h` still has
499-
`SNP_GET_EXT_REPORT` and its `certs_address` — so the guest asks and gets an
500-
empty table back.
501-
502-
Which is exactly what was observed: `sev-snp-attest` found no `certs`,
503-
`cert_chain` or `auxblob` content, so the attestation travelled with an empty
504-
chain and the verifier went to KDS every time.
505-
506-
**So, on a stock Ubuntu/Debian QEMU, KDS is not optional.** The realistic
507-
mitigations are the boring ones: keep a long-lived KMS so its in-process cache
508-
stays warm, and space repeat verifications out. Removing the dependency properly
509-
means either a QEMU that can carry the cert blob, or a KDS mirror behind
510-
`[core.attestation.urls] amd_kds`. Both are work, not configuration. And note
511-
the bootstrap problem either way — obtaining the VCEK requires KDS access once,
512-
from a network that can reach it.
432+
### 9.4 KDS is on the critical path, and stays there
433+
434+
Verification needs the ASK and VCEK certificates. They come from
435+
`https://kdsintf.amd.com`, a **single global endpoint with no mirror**, rate
436+
limited to roughly one identical request per 10s, and unreachable for an entire
437+
test round — TCP timeouts and 100% ICMP loss from five vantage points across four
438+
autonomous systems and two continents, while every other external service
439+
answered normally and DNS stayed healthy throughout.
440+
441+
The rate limit is the one that will bite you mid-test: back-to-back verifications
442+
of the same chip return `HTTP status client error (429)` on the VCEK URL, which
443+
reads exactly like a verification failure until you notice the status code. Space
444+
them ~25s apart.
445+
446+
Caching is in-process only. `AmdKdsClient` keeps a `moka` cache of 16 CA chains
447+
and 1024 VCEKs, capacity-bounded, no TTL, no persistence — warm for the life of
448+
one KMS or verifier process and cold in every one-shot `dstack-verifier --verify`.
449+
There is no KDS cache service in this repo; `[core.attestation.urls] amd_kds`
450+
would accept a mirror, but nothing implements one. (Compare `nvidia-attest-proxy`,
451+
the persistent on-disk cache the NVIDIA collateral got and AMD did not.)
452+
453+
Do not plan on serving the certificates from the host instead. The verifier side
454+
supports it — `normalize_kernel_cert_table` reads ASK and VCEK from the SNP
455+
extended report's certificate table — and so does the guest, but nothing can put
456+
them there: `sev-snp-guest` in stock QEMU has no `certs-path`-style property, and
457+
the host-side `SNP_SET_EXT_CONFIG` ioctl is not in upstream `psp-sev.h`. Guests
458+
ask via `SNP_GET_EXT_REPORT` and get an empty table, which is why every
459+
verification in this round went to KDS.
460+
461+
A note for whoever touches this code: the ARK is already compiled in, and the ARK
462+
fetched from KDS is discarded (`let (_fetched_ark, ask) = ...`). The network
463+
request exists only to obtain the ASK, which is equally static per product family.
464+
Bundling it would remove the `cert_chain` request entirely, leaving only the
465+
per-chip VCEK.
513466

514467
### 9.5 Reaching the API that holds the keys
515468

@@ -539,12 +492,6 @@ these JSON bodies are **hex strings**, not base64 and not byte arrays. Base64
539492
fails with `Invalid character 'w' at position 5`, which names the symptom and not
540493
the cause.
541494

542-
A related note for whoever touches this code: the ARK is already compiled in, and
543-
the ARK fetched from KDS is discarded (`let (_fetched_ark, ask) = ...`). The
544-
network request exists only to obtain the ASK, which is equally static per product
545-
family. Bundling it would remove the `cert_chain` request entirely, leaving only
546-
the per-chip VCEK.
547-
548495
---
549496

550497
## 10. Fixed in 0.6.0-rc0 testing — do not re-diagnose

0 commit comments

Comments
 (0)