Skip to content

Commit a9db617

Browse files
authored
Merge pull request #1132 from Dstack-TEE/feat/certbot-dns-persist-01
feat(certbot): issue certificates via `dns-persist-01`
2 parents 027c62b + 4fcd3d2 commit a9db617

25 files changed

Lines changed: 3860 additions & 1511 deletions

File tree

CHANGELOG.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
2525
- guest-agent: v1 `Attest` accepts `include_boottime_gpu_evidence` and returns the boot-time GPU attestation evidence in `AttestResponse.boottime_gpu_evidence`, so a verifier fetches the attestation and the GPU evidence in one round trip. It arrives as a `GpuEvidenceBundle` list -- the same shape `AttestGpu` returns, so a consumer writes one bundle parser and dispatches on `format`: `nvidia-nvattest-boottime-json-v1` is the record written at boot, `nvidia-nvattest-collect-evidence-json-v1` is collected on demand against a caller's nonce, and a verifier for one does not appraise the other. Absence is the empty list. The bundle's `evidence` is the nvattest output byte for byte as read from disk, because the only thing binding it to the boot is sha256 over precisely those bytes against the measured `gpu-attestation` event. `Attest` is also v1's sole CVM attestation entry point: the `VersionedAttestation` it returns already carries the TDX quote and event log, and unlike `GetQuote` it answers on every supported platform
2626
- sdk: `AppCompose` in the Go SDK gained `init_script`, `storage_fs`, `swap_size`, `event_log_version`, `port_policy` and `verity_volumes`, and `Requirements` gained `gpu_policy` in the Go and Python SDKs
2727
- shared API authentication (`dstack-api-auth`) protecting the full VMM HTTP/pRPC/UI surface and unifying Gateway/KMS admin auth: bearer/`X-Admin-Token`/HTTP Basic/bcrypt htpasswd, constant-time verification (#796)
28+
- certbot/gateway: opt-in `dns-persist-01` certificate validation ([draft-ietf-acme-dns-persist-01](https://datatracker.ietf.org/doc/html/draft-ietf-acme-dns-persist-01)), which issues without a DNS provider credential at all. `dns-01` needs write access to the zone on every order, so a gateway CVM holds a Cloudflare API token for the life of the deployment -- a token that rewrites the whole zone, not just `_acme-challenge`. Under `dns-persist-01` the zone owner publishes one `_validation-persist.<name>` TXT record naming the CA and the ACME account; nothing about it changes between orders, so certbot only ever reads DNS and the zone can be hosted anywhere, with no provider integration. Set `challenge = "dns-persist-01"` in `certbot.toml`, or `challenge` on a gateway ZT domain; the default stays `dns-01` and existing deployments are untouched. `certbot dns-records` prints the records to publish, `GetZtDomain`/`ListZtDomains` return them in `required_dns_records`, and the gateway logs them wherever it would otherwise have written DNS. CAA records carry `validationmethods=dns-persist-01` to match, so switching methods means republishing both. Two gateway operations change shape for such a domain: `SetCaa` skips it, having nothing to reconcile without write access, and `RotateAcmeCredentials` leaves it broken until the operator republishes -- the record still names the old account -- so the response now returns the new records in `required_dns_records`. A rotation that registers the account but fails to re-pin some domain's CAA now reports those domains in `repin_failed_domains` instead of failing the call: the account exists and the cluster is using it, so raising an error there reads as "nothing happened" and invites a retry that registers another account. **Experimental**: the draft is still changing, and Let's Encrypt serves the challenge on staging only pending an open working-group issue. Documented in `docs/certbot-dns-persist-01.md`.
29+
30+
Two settings changed shape for every deployment, not only `dns-persist-01` ones. `issuer_domain_name` now names the CA in the CAA records written for `dns-01` too -- its default is `letsencrypt.org`, so an untouched configuration writes exactly what it wrote before -- and it is validated where it is set, since it lands verbatim in an RFC 8659 `issue-value` and CAA is republished by deleting the old records first. The pre-order DNS self-check is now capped at half of `renew_timeout` rather than at its own configured value: both defaulted to 300s in the gateway, and the CLI wrapped a 300s wait in a 120s budget, so the timeout around the order always fired first and a missing record was reported as `certificate request timed out` instead of by name
2831
- gateway: `Admin.Status` reports `health_gating`, so an operator can see whether this node's health polling is switched on. With it off, instances that opted in sit at `unknown` forever and are all in rotation, which is otherwise indistinguishable on the dashboard from being held out pending a first answer
2932
- gateway: `Admin.SetInstanceReady` takes a CVM instance out of its app's load-balancing rotation without stopping it; instance-id routing stays open so the instance can still be investigated, and the setting survives re-registration
3033
- gateway: operator-set per-instance overrides now live under their own KV keys — `admin/<instance_id>/ready` and `admin/<instance_id>/port_policy` — instead of inside the instance record, so a CVM re-registration can no longer drop them and setting one cannot discard a peer's unsynced change to the other. An override left in an instance record by an earlier build is moved across on load

docs/certbot-dns-persist-01.md

Lines changed: 240 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,240 @@
1+
# Certificate issuance without a DNS credential (`dns-persist-01`)
2+
3+
`dns-01` asks certbot to write a fresh `_acme-challenge` TXT record for every
4+
order, so whatever runs certbot holds a DNS API token with write access to the
5+
zone, forever. In dstack that token lives inside the gateway CVM. Attestation
6+
covers what the CVM is running, but a token is a token: anything that gets hold
7+
of it can rewrite the zone, including records that have nothing to do with
8+
certificates.
9+
10+
`dns-persist-01` moves the proof out of the issuance loop. The zone owner
11+
publishes one record naming the CA and the ACME account allowed to issue:
12+
13+
```dns
14+
_validation-persist.example.com. IN TXT "letsencrypt.org; accounturi=https://acme-v02.api.letsencrypt.org/acme/acct/1234567890"
15+
```
16+
17+
The account key proves who is asking, the record proves the zone owner agreed,
18+
and neither changes between orders. certbot reads DNS and never writes it, so
19+
the CVM holds no DNS credential and the zone can be hosted anywhere — no
20+
Cloudflare account, no API token, no provider integration.
21+
22+
> **Experimental.** `dns-persist-01` is specified in
23+
> [draft-ietf-acme-dns-persist-01][draft], which is still changing: an open
24+
> working-group issue may add a client-key-derived value to the record, and
25+
> Let's Encrypt has said it will not deploy the challenge to production until
26+
> that is resolved. It is live on Let's Encrypt **staging** and in
27+
> [Pebble][pebble]. Treat the record format as unstable, and expect to
28+
> republish when the draft settles.
29+
30+
[draft]: https://datatracker.ietf.org/doc/html/draft-ietf-acme-dns-persist-01
31+
[pebble]: https://github.com/letsencrypt/pebble
32+
33+
## What the record means
34+
35+
| Part | Effect |
36+
| --- | --- |
37+
| `letsencrypt.org` | Issuer Domain Name. A CA ignores records naming a different issuer, so one label can hold records for several CAs. |
38+
| `accounturi=` | The ACME account authorized to issue. Compared byte for byte — no case folding, no URI normalization. |
39+
| `policy=wildcard` | Extends the record to `*.example.com`. Without it the CA authorizes `example.com` alone and refuses wildcard orders. |
40+
| `persistUntil=` | Optional UNIX timestamp after which the CA stops accepting the record. |
41+
42+
A wildcard order authorizes from its base name — `*.example.com` is validated
43+
against `_validation-persist.example.com`, not
44+
`_validation-persist.*.example.com` — so one record covers a name and its
45+
wildcard.
46+
47+
Two things about the syntax bite in practice, because a CA rejects the whole
48+
record rather than ignoring the offending part: **no trailing semicolon**, and
49+
**no whitespace inside a value**. certbot renders records that satisfy both;
50+
copy them verbatim rather than retyping.
51+
52+
The scope stops at the names above. Let's Encrypt does not walk up the tree, so
53+
a record on `example.com` does not authorize `sub.example.com` — give each base
54+
name its own record.
55+
56+
## Standalone certbot
57+
58+
`certbot` never writes DNS in this mode, so setup is: create the account, read
59+
the records off it, publish them, then issue.
60+
61+
```toml
62+
# certbot.toml
63+
workdir = "/var/lib/certbot"
64+
acme_url = "https://acme-staging-v02.api.letsencrypt.org/directory"
65+
challenge = "dns-persist-01"
66+
issuer_domain_name = "letsencrypt.org"
67+
# auto_set_caa promises certbot keeps CAA in sync, which it cannot do without
68+
# write access. Leave it off and publish the CAA records below by hand.
69+
auto_set_caa = false
70+
domains = ["example.com", "*.example.com"]
71+
renew_interval = 3600
72+
renew_days_before = 10
73+
renew_timeout = 120
74+
# Capped at half of renew_timeout, so this one waits 60s, not 300s. The whole
75+
# order -- the wait, the authorizations, finalize, and fetching the certificate
76+
# -- has to fit inside renew_timeout, and a wait that outlasts it takes the
77+
# timeout with it, losing the warning naming the record that was missing.
78+
# Raising this alone changes nothing; raise renew_timeout with it.
79+
max_dns_wait = 300
80+
```
81+
82+
`cf_api_token` is unused and can be left out; certbot warns if one is set.
83+
84+
```console
85+
$ certbot init -c certbot.toml
86+
INFO certbot::bot: creating new ACME account
87+
INFO certbot::bot: created new ACME account: https://acme-staging-v02.api.letsencrypt.org/acme/acct/1234567890
88+
89+
$ certbot dns-records -c certbot.toml
90+
_validation-persist.example.com. IN TXT "letsencrypt.org; accounturi=https://acme-staging-v02.api.letsencrypt.org/acme/acct/1234567890; policy=wildcard"
91+
example.com. IN CAA 0 issue "letsencrypt.org;validationmethods=dns-persist-01;accounturi=https://acme-staging-v02.api.letsencrypt.org/acme/acct/1234567890"
92+
example.com. IN CAA 0 issuewild "letsencrypt.org;validationmethods=dns-persist-01;accounturi=https://acme-staging-v02.api.letsencrypt.org/acme/acct/1234567890"
93+
```
94+
95+
Publish all three, wait for them to propagate, then issue:
96+
97+
```console
98+
$ certbot renew --once -c certbot.toml
99+
INFO certbot::acme_client: requesting new certificates for example.com, *.example.com
100+
INFO certbot::bot: created new certificate
101+
```
102+
103+
Renewals need nothing further. The record stays, and `certbot renew` reuses it
104+
for every order.
105+
106+
The CAA records are optional but recommended — they stop any other account, at
107+
Let's Encrypt or elsewhere, from being issued for your name. Note the
108+
`validationmethods=dns-persist-01` in them: a CAA record left pinned to
109+
`dns-01` refuses every `dns-persist-01` order, so switching methods means
110+
updating CAA and the validation record together.
111+
112+
### When issuance fails
113+
114+
certbot checks its own resolver before starting an order, and says exactly what
115+
it expected to find:
116+
117+
```
118+
WARN certbot::acme_client: no TXT record at _validation-persist.example.com matches the expected value: letsencrypt.org; accounturi=https://acme-staging-v02.api.letsencrypt.org/acme/acct/1234567890; policy=wildcard
119+
Error: order is invalid: API error: Checking DNS-PERSIST-01 challenge TXT record with issuer-domain-name "letsencrypt.org": accounturi mismatch: expected "https://acme-staging-v02.api.letsencrypt.org/acme/acct/1234567890", got "https://acme-staging-v02.api.letsencrypt.org/acme/acct/9876543210" (urn:ietf:params:acme:error:unauthorized)
120+
```
121+
122+
The check is advisory and never blocks an order: certbot's resolver is not the
123+
CA's, and its expectation can be stricter than what the CA would accept. A
124+
warning with a successful issuance underneath it is a resolver difference, not
125+
a problem. A record that genuinely does not match costs the whole DNS budget
126+
before the order is sent, because the check waits it out first -- that is
127+
`max_dns_wait`, or half of `renew_timeout` when that is smaller, which for the
128+
configuration above is 60s. certbot logs the budget it settled on at `debug`.
129+
130+
If the CA rejects the order, compare the published record against
131+
`certbot dns-records` character by character. The usual causes are a stale
132+
`accounturi` after the account was recreated, a missing `policy=wildcard` on a
133+
wildcard order, and an issuer domain name that does not match the CA.
134+
135+
## dstack-gateway
136+
137+
A ZT domain picks its method with the `challenge` field, which defaults to
138+
`dns-01` — existing domains are unaffected:
139+
140+
```json
141+
{ "domain": "app.example.com", "port": 443, "challenge": "dns-persist-01" }
142+
```
143+
144+
Such a domain needs no `dns_cred_id`, and the gateway CVM never receives a DNS
145+
credential for it. `GetZtDomain` and `ListZtDomains` return the records to
146+
publish in `required_dns_records`, and the gateway logs them whenever it cannot
147+
write DNS itself:
148+
149+
```
150+
WARN cert[app.example.com]: publish this record by hand: _validation-persist.app.example.com. IN TXT "letsencrypt.org; accounturi=...; policy=wildcard"
151+
```
152+
153+
The gateway issues for `*.{domain}` only, so each domain needs one validation
154+
record with `policy=wildcard`.
155+
156+
### Setup order
157+
158+
The records name the ACME account, so the account comes first:
159+
160+
1. `SetCertbotConfig``acme_url`, and `issuer_domain_name` if the CA is not
161+
Let's Encrypt.
162+
2. `RotateAcmeCredentials` — registers the account and returns its `account_uri`.
163+
It needs no ZT domain: registration proves nothing about a domain. Every call
164+
registers a *new* account, so call it once here; running it again later is a
165+
rotation, with the consequences described below.
166+
3. `AddZtDomain` with `challenge: "dns-persist-01"`. No `dns_cred_id`: the
167+
domain needs none, and the gateway CVM never receives one for it.
168+
4. Read the records — the dashboard shows them when the domain is added and
169+
again under its **Records** button, and `GetZtDomain`, `ListZtDomains` and
170+
the gateway log all carry them — then publish the `_validation-persist` TXT
171+
record, plus the CAA records if you want issuance pinned to this account.
172+
5. `RenewZtDomainCert`. Renewals from here need nothing further.
173+
174+
Adding another dns-persist-01 domain later starts at step 3: the account already
175+
exists, so the new domain's records are available as soon as it is. A domain's
176+
challenge is chosen when it is added — in the dashboard's ZT-Domain form as well
177+
as over the API — and carried forward by every edit. `UpdateZtDomain` replaces
178+
the whole record, so `challenge` is optional there and omitting it means "leave
179+
it as it is": a caller that predates the field — a cached dashboard bundle, a
180+
script, an older SDK — cannot downgrade a `dns-persist-01` domain to `dns-01` by
181+
editing something else, which its hand-published CAA would then refuse. Send the
182+
field explicitly to switch a domain over.
183+
184+
### Naming the CA
185+
186+
For a non-production ACME server, set `issuer_domain_name` in the global certbot
187+
config — `Admin.SetCertbotConfig`, or the field of that name in the dashboard's
188+
Certbot Configuration — to whatever that server puts in `issuer-domain-names`,
189+
`pebble.letsencrypt.org` for Pebble. Empty means `letsencrypt.org`.
190+
191+
It also names the CA in the CAA records certbot writes for `dns-01` domains, so
192+
one setting covers both challenges rather than pinning CAA to Let's Encrypt
193+
while orders go somewhere else. `acme_url` is one setting for the whole
194+
deployment; this follows it.
195+
196+
The value is checked when it is set. It ends up verbatim inside an RFC 8659
197+
`issue-value`, and publishing CAA deletes the records it replaces before writing
198+
the new ones, so a name carrying a space or a `;` would leave the zone holding a
199+
malformed `issue` property with nothing valid behind it — CAA that forbids every
200+
issuer. Anything that is not a DNS name is refused by `SetCertbotConfig`, and by
201+
`certbot` at startup.
202+
203+
Publishing CAA installs a temporary `;` guard, deletes the records it replaces,
204+
writes the new ones and drops the guard. A run interrupted part-way leaves the
205+
guard behind, which denies every issuer until a later run finishes the job — so
206+
each run now sweeps stale guards before installing its own, and a rerun is
207+
enough to recover.
208+
209+
### How long the check waits
210+
211+
The self-check before each order waits for the record to become visible, but
212+
never longer than half of `renew_timeout`, the budget for the whole renewal.
213+
Past that the outer timeout fires first and the renewal ends as `certificate
214+
request timed out` with no mention of the record it could not find — the
215+
opposite of what the check is for. Both values defaulted to 300s in the gateway
216+
(and the CLI wrapped a 300s wait in a 120s budget), so this was the default
217+
behaviour rather than an edge case.
218+
219+
Two operations behave differently on these domains:
220+
221+
- **`SetCaa`** skips them. There is nothing to reconcile without write access;
222+
the records are logged instead, and the summary reports how many were left to
223+
the operator.
224+
- **`RotateAcmeCredentials`** is not self-service. Rotation moves the cluster
225+
to a new ACME account, and every `_validation-persist` record still names the
226+
old one, so orders for those domains fail until the operator republishes. The
227+
response returns the new records in `required_dns_records` and the gateway
228+
logs them; publish before the next renewal comes due.
229+
230+
A rotation that registers the account but cannot re-pin every `dns-01`
231+
domain's CAA still succeeds — the account exists and the cluster is on it —
232+
and names those domains in `repin_failed_domains`. Finish with `SetCaa`
233+
rather than another rotation, which would register yet another account
234+
against a rate-limited quota. The dashboard reports both lists in one dialog
235+
after a rotation, with a **Run SetCaa** button for the second.
236+
237+
## Related
238+
239+
- [dstack-gateway](dstack-gateway.md) — gateway architecture and TLS termination
240+
- [deployment.md](deployment.md#4-zero-trust-https-optional) — the `dns-01` setup this replaces

0 commit comments

Comments
 (0)