Certificates remain stale after wake from sleep

[Bilge]

Issue Details

Every morning, when I resume my computer from sleep, I am unable to access local Caddy sites (on HTTPS) for 30-60 minutes while the short-lease expired certificates take a while to refresh. Eventually they refresh and the sites are accessible again, but this 30-60 minute delay is way too long; Caddy should notice this and refresh them sooner, especially if a request comes in that it can't serve with a valid certificate.

Assistance Disclosure

AI not used

If AI was used, describe the extent to which it was used.

No response

[steadytao]

This will need a bit more detail before we can tell whether this is a bug or a config/lifetime issue.

Could you provide:

• your Caddyfile/config
• whether you are using local_certs, tls internal, a custom lifetime or a custom renew_interval
• the Caddy version you're using
• debug logs from immediately after wake, especially tls / certmagic logs around certificate selection and renewal
• the certificate lifetime and expiration time for one affected local site

For context, internal/local certs are intentionally short-lived by default and renewal is timer-driven while Caddy is running. If the machine sleeps through most or all of the renewal window, the cert may be expired on wake. That said, an expired managed cert should normally trigger renewal during the TLS handshake so a 30-60 minute outage seems longer than I would expect and will require logs to narrow down.

[steadytao]

@Bilge Will be closing this as needs info for now but if you could please provide us the necessary details to investigate, more then happy for this to be reopened.

[Bilge]

dev.steam250.com local.dev.steam250.com {
    tls internal

    root "/usr/share/Steam 250/site"
    file_server
    try_files {path}.html

    header Access-Control-Allow-Origin *
}

caddy -v: v2.10.2 h1:g/gTYjGMD0dec+UgMw8SnfmJ3I9+M2TdvoRL/Ovu6U8=

Logs

Details

{"level":"info","ts":1778827131.7333145,"logger":"tls.renew","msg":"releasing lock","identifier":"s.club.local.dev.steam250.com"}

{"level":"info","ts":1778827131.7332132,"logger":"tls.renew","msg":"certificate renewed successfully","identifier":"s.steam250.com","issuer":"local"}

{"level":"info","ts":1778827131.7333937,"logger":"tls.renew","msg":"certificate renewed successfully","identifier":"club.local.dev.steam250.com","issuer":"local"}

{"level":"info","ts":1778827131.73343,"logger":"tls.renew","msg":"releasing lock","identifier":"club.local.dev.steam250.com"}

{"level":"info","ts":1778827131.7333577,"logger":"tls.renew","msg":"certificate renewed successfully","identifier":"headless.dev.steam250.com","issuer":"local"}

{"level":"info","ts":1778827131.733497,"logger":"tls.renew","msg":"releasing lock","identifier":"s.steam250.com"}

{"level":"info","ts":1778827131.733523,"logger":"tls.renew","msg":"releasing lock","identifier":"headless.dev.steam250.com"}

{"level":"info","ts":1778827131.7337868,"logger":"tls.renew","msg":"certificate renewed successfully","identifier":"local.dev.steam250.com","issuer":"local"}

{"level":"info","ts":1778827131.7338352,"logger":"tls.renew","msg":"releasing lock","identifier":"local.dev.steam250.com"}

{"level":"info","ts":1778827131.7338994,"logger":"tls.renew","msg":"releasing lock","identifier":"club.steam250.com"}

{"level":"info","ts":1778827131.7339118,"logger":"tls","msg":"reloading managed certificate","identifiers":["s.club.local.dev.steam250.com"]}

{"level":"info","ts":1778827131.7339344,"logger":"tls","msg":"reloading managed certificate","identifiers":["club.steam250.com"]}

{"level":"info","ts":1778827131.7339435,"logger":"tls.renew","msg":"releasing lock","identifier":"s.club.dev.steam250.com"}

{"level":"info","ts":1778827131.7339768,"logger":"tls","msg":"reloading managed certificate","identifiers":["s.club.dev.steam250.com"]}

{"level":"warn","ts":1778827131.7340567,"logger":"tls","msg":"stapling OCSP","identifiers":["s.headless.club.dev.steam250.com"]}

{"level":"info","ts":1778827131.7340677,"logger":"tls.cache","msg":"replaced certificate in cache","subjects":["s.headless.club.dev.steam250.com"],"new_expiration":1778870331}

{"level":"info","ts":1778827131.7332132,"logger":"tls.renew","msg":"certificate renewed successfully","identifier":"headless.club.dev.steam250.com","issuer":"local"}

{"level":"info","ts":1778827131.7341084,"logger":"tls.renew","msg":"releasing lock","identifier":"dev.steam250.com"}

{"level":"info","ts":1778827131.7341397,"logger":"tls","msg":"reloading managed certificate","identifiers":["dev.steam250.com"]}

{"level":"warn","ts":1778827131.7341464,"logger":"tls","msg":"stapling OCSP","identifiers":["s.club.local.dev.steam250.com"]}

{"level":"warn","ts":1778827131.734136,"logger":"tls","msg":"stapling OCSP","identifiers":["s.club.dev.steam250.com"]}

{"level":"info","ts":1778827131.734155,"logger":"tls.cache","msg":"replaced certificate in cache","subjects":["s.club.local.dev.steam250.com"],"new_expiration":1778870331}

{"level":"info","ts":1778827131.7339125,"logger":"tls","msg":"reloading managed certificate","identifiers":["club.local.dev.steam250.com"]}

{"level":"info","ts":1778827131.7341597,"logger":"tls.cache","msg":"replaced certificate in cache","subjects":["s.club.dev.steam250.com"],"new_expiration":1778870331}

{"level":"info","ts":1778827131.7333555,"logger":"tls","msg":"reloading managed certificate","identifiers":["api.dev.steam250.com"]}

{"level":"info","ts":1778827131.7341752,"logger":"tls.renew","msg":"releasing lock","identifier":"headless.club.dev.steam250.com"}

{"level":"info","ts":1778827131.734208,"logger":"tls","msg":"reloading managed certificate","identifiers":["headless.club.dev.steam250.com"]}

{"level":"info","ts":1778827131.7342484,"logger":"tls","msg":"reloading managed certificate","identifiers":["s.steam250.com"]}

{"level":"warn","ts":1778827131.734348,"logger":"tls","msg":"stapling OCSP","identifiers":["headless.club.dev.steam250.com"]}

{"level":"info","ts":1778827131.7343574,"logger":"tls.cache","msg":"replaced certificate in cache","subjects":["headless.club.dev.steam250.com"],"new_expiration":1778870331}

{"level":"warn","ts":1778827131.734087,"logger":"tls","msg":"stapling OCSP","identifiers":["club.steam250.com"]}

{"level":"info","ts":1778827131.7343652,"logger":"tls.cache","msg":"replaced certificate in cache","subjects":["club.steam250.com"],"new_expiration":1778870331}

{"level":"warn","ts":1778827131.734388,"logger":"tls","msg":"stapling OCSP","identifiers":["club.local.dev.steam250.com"]}

{"level":"info","ts":1778827131.734414,"logger":"tls.cache","msg":"replaced certificate in cache","subjects":["club.local.dev.steam250.com"],"new_expiration":1778870331}

{"level":"warn","ts":1778827131.7344062,"logger":"tls","msg":"stapling OCSP","identifiers":["dev.steam250.com"]}

{"level":"info","ts":1778827131.734426,"logger":"tls.cache","msg":"replaced certificate in cache","subjects":["dev.steam250.com"],"new_expiration":1778870331}

{"level":"warn","ts":1778827131.734484,"logger":"tls","msg":"stapling OCSP","identifiers":["api.dev.steam250.com"]}

{"level":"info","ts":1778827131.7345045,"logger":"tls.cache","msg":"replaced certificate in cache","subjects":["api.dev.steam250.com"],"new_expiration":1778870331}

{"level":"warn","ts":1778827131.7345035,"logger":"tls","msg":"stapling OCSP","identifiers":["s.steam250.com"]}

{"level":"info","ts":1778827131.7345173,"logger":"tls.cache","msg":"replaced certificate in cache","subjects":["s.steam250.com"],"new_expiration":1778870331}

{"level":"info","ts":1778827131.7345607,"logger":"tls","msg":"reloading managed certificate","identifiers":["local.dev.steam250.com"]}

{"level":"info","ts":1778827131.73456,"logger":"tls","msg":"reloading managed certificate","identifiers":["headless.dev.steam250.com"]}

{"level":"warn","ts":1778827131.734927,"logger":"tls","msg":"stapling OCSP","identifiers":["local.dev.steam250.com"]}

{"level":"info","ts":1778827131.7349453,"logger":"tls.cache","msg":"replaced certificate in cache","subjects":["local.dev.steam250.com"],"new_expiration":1778870331}

{"level":"warn","ts":1778827131.734926,"logger":"tls","msg":"stapling OCSP","identifiers":["headless.dev.steam250.com"]}

{"level":"info","ts":1778827131.7349536,"logger":"tls.cache","msg":"replaced certificate in cache","subjects":["headless.dev.steam250.com"],"new_expiration":1778870331}

[steadytao]

Thanks, that seems like enough detail to reopen for investigation.

The config confirms this is using tls internal so these are short-lived local certificates. The logs also show successful renewal and cache replacement with the renewed certs expiring 12 hours later which matches the internal issuer default lifetime.

The remaining question is where the 30-60 minute delay is coming from. Managed cert maintenance is periodic but an expired managed certificate should normally trigger renewal during the TLS handshake rather than remaining stale until a later background scan. To make this actionable, the most useful next data would be debug logs starting immediately after wake including the first failed request, any tls / tls.renew / CertMagic messages and the later successful renewal. It would also help to know whether this reproduces on the current release since v2.10.2 is not latest, we are now on v2.11.3 (I would suggest working with v.2.11.2 with this issue temporarily as v2.11.3 has a mTLS bug which can be found within #7724).

I would not call this confirmed yet but it is legitimate enough to investigate now.

[Bilge]

How do I get "debug logs", though?

[mholt]

In the Caddyfile, add debug to your global options block at the top.