Skip to content

feat(prober): add CRL certificate revocation checking - #1583

Open
sebastiangaiser wants to merge 1 commit into
prometheus:masterfrom
sebastiangaiser:feat/crl
Open

feat(prober): add CRL certificate revocation checking#1583
sebastiangaiser wants to merge 1 commit into
prometheus:masterfrom
sebastiangaiser:feat/crl

Conversation

@sebastiangaiser

@sebastiangaiser sebastiangaiser commented May 19, 2026

Copy link
Copy Markdown
Contributor

Check the full certificate chain against CRLs across the HTTP, TCP, gRPC, and Unix probes. For each cert with CRLDistributionPoints, fetch the CRL, verify its signature against the issuer, reject CRLs that are stale or not yet valid, and look up the cert serial in the revocation list. Enabled per probe via the opt-in check_revoked flag (default false); the CRL fetch is bounded by the probe timeout, no separate knob. DNS over TLS is not covered yet.

What this PR does / Which issue(s) does the PR fix:

Does this PR introduce a user-facing change?

CRL certificate revocation checking

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG added in release-notes section of PR Desc.

Comment thread prober/crl_test.go
Comment thread config/config.go Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds opt-in CRL certificate revocation checks and metrics for TLS-enabled probes.

Changes:

  • Fetches and validates CRLs across certificate chains.
  • Exposes CRL availability, revocation, staleness, timing, and publication metrics.
  • Adds configuration and tests for CRL behavior.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
config/config.go Adds CRL configuration types and probe fields.
config/testdata/blackbox-good.yml Adds valid HTTP CRL configuration.
prober/crl.go Implements CRL fetching, validation, and metrics.
prober/crl_test.go Tests CRL processing and metrics.
prober/dns.go Adapts DNS TLS configuration wrapper.
prober/grpc.go Integrates CRL checks into gRPC probes.
prober/grpc_test.go Updates gRPC TLS test configuration.
prober/handler_test.go Updates handler TLS test configuration.
prober/http.go Integrates CRL checks into HTTP probes.
prober/http_test.go Tests unavailable CRL reporting.
prober/query_response.go Integrates CRL checks for TCP and Unix TLS.
prober/tcp.go Adapts TCP TLS configuration.
prober/tcp_test.go Updates TCP TLS test configuration.
prober/unix.go Adapts Unix TLS configuration.
Suppressed comments (2)

prober/crl.go:174

  • The timer stops as soon as response headers arrive. A slow or large CRL body can consume most of the probe timeout while probe_ssl_crl_fetch_time_seconds reports only time-to-first-byte; measure through ReadAll so the metric represents the fetch.
	start := time.Now()
	resp, err := client.Do(req)
	fetchTime := time.Since(start).Seconds()
	if err != nil {

config/config.go:315

  • Without a JSON tag, non-HTTP TLS configurations serialize this property as CRLCheck, while the YAML and all neighboring JSON config fields use snake_case. Give both configuration formats the same public key.
	CRLCheck         CRLCheckConfig `yaml:"crl_check,omitempty"`

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread prober/crl.go Outdated
Comment thread prober/crl.go Outdated
Comment thread config/config.go Outdated
Comment thread config/config.go Outdated
Comment thread config/config.go Outdated
Comment thread prober/query_response.go Outdated
Comment thread prober/grpc.go Outdated
@sebastiangaiser
sebastiangaiser force-pushed the feat/crl branch 3 times, most recently from 6859675 to 0e0b293 Compare August 13, 2026 20:05
@sebastiangaiser sebastiangaiser changed the title feat(prober,http,grpc): add CRL certificate revocation checking feat(prober): add CRL certificate revocation checking Aug 13, 2026
@sebastiangaiser
sebastiangaiser force-pushed the feat/crl branch 2 times, most recently from 742ecbe to 4533283 Compare August 13, 2026 20:09
@sebastiangaiser

Copy link
Copy Markdown
Contributor Author

@electron0zero thanks for the review. I've reworked the PR along your suggestions:

  • Added a single opt-in flag. Dropped the CRLCheckConfig type and the TLSConfigWithCRL wrapper entirely. Each probe now has a plain check_revoked: <bool> (default false) next to tls/tls_config; tls_config is back to the plain config.TLSConfig. HTTP/TCP/gRPC/Unix share the same switch.
  • Removed the separate CRL fetch timeout, the fetch is now bounded by the probe timeout.
  • Regarding DNS-over-TLS. I reverted DNS to config.TLSConfig and left DoT out for now (the TLS ConnectionState isn't readily accessible via the miekg/dns client), so there's no silently-ignored knob. Not sure how we should follow-up on that one.

I also addressed the Copilot findings: probe_ssl_crl_fetch_time_seconds now accumulates across distribution points and measures through the body read, and not-yet-valid CRLs (thisUpdate in the future) are rejected. Tests were refactored to t.Helper()/t.Fatalf() and I added integration tests that actually enable CRL and assert the metrics for TCP (direct TLS + StartTLS), Unix, and gRPC. check_revoked is now documented in CONFIGURATION.md.

Comment thread prober/crl_test.go Outdated
Comment thread prober/crl_test.go Outdated
Comment thread prober/crl_test.go Outdated
Comment thread prober/crl_test.go Outdated
Comment thread prober/crl_test.go Outdated
Comment thread config/config.go
Comment thread prober/http_test.go Outdated
Comment thread prober/crl.go Outdated
Comment thread prober/crl.go Outdated
Comment thread CONFIGURATION.md
tls_config:
[ <tls_config> ]

# Check each certificate in the TLS chain against its CRL and emit probe_ssl_crl_* metrics.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thinking about this, one thing come to my mind. When someone sets check_revoked to true, do they just expect the metrics or they also want the probe to fail when the Cert is revoked?

What is check_revoked meant to promise? A cert confirmed on its issuer's CRL still gives probe_success=1, but the config name reads like enforcement.

we have flags like fail_if_ssl / fail_if_not_ssl / fail_if_body_matches_regexp for enforcement and failing the probe.

if your use-case and intent is to fail the probe, then we should add support for a fail_if_cert_revoked flag as well, and when that's set, we should fail the probe like we do for the fail_if_ssl and many other flags.

if you just want the metrics, then current setup is fine as is, and we can keep publishing the metrics and probe_success is left as is.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our use case is alerting on the metrics, with the revocation signal deliberately kept separate from probe_success:

  • probe_ssl_crl_revoked == 1 -> critical after 5m
  • probe_ssl_crl_available == 0 (restricted to crl_url!="") -> warning after 15m
  • probe_ssl_crl_stale == 1 -> warning after 1h

probe_success then keeps its own meaning: the target is reachable and answered as expected. That separation is the useful part. A CRL distribution point being down is not the same outage as the service being down, and the two want different severities and different runbooks. If check_revoked also failed the probe, every dead or slow CRL responder would look like a total outage and we'd lose the ability to grade "revoked" (act now) differently from "revocation status unknown" (look into it).

For the same reason we alert on the per-phase probe_http_duration_seconds rather than the total probe_duration_seconds for latency, since the total includes the CRL fetch and a slow responder would otherwise inflate it.

You're right that the name alone doesn't convey that, so I've spelled the semantics out in CONFIGURATION.md for all four probes: the flag only reports revocation state and never fails the probe, a revoked certificate or an unreachable CRL leaves probe_success untouched, and alerting is expected to happen on the metrics. If you'd rather have that stated somewhere more prominent than the per-probe options, I'm happy to add a short section on the CRL metrics and how to alert on them.

And if hard enforcement turns out to be wanted later, a separate fail_if_cert_revoked can be added on top without changing this behaviour.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

makes sense, I am good with current setup and we can always add fail_if_cert_revoked flag in the future.

@sebastiangaiser
sebastiangaiser force-pushed the feat/crl branch 2 times, most recently from 4a4fad8 to ae9cf0b Compare August 18, 2026 14:36
@sebastiangaiser

Copy link
Copy Markdown
Contributor Author

Thanks for the detailed second pass @electron0zero. I've pushed changes for all of it.

  • Metric renamed to probe_ssl_crl_next_update_timestamp_seconds as suggested. I also renamed the corresponding struct field to NextUpdate so it lines up with x509.RevocationList.NextUpdate.
  • crlCertResult is no longer exported, it was only ever used inside the package.
  • check_revoked is now rejected when TLS is off, so the pair can't be configured in a way that silently does nothing.
  • CRL fetches use the probe's proxy configuration, including proxy_connect_header.
  • Test helpers moved to prober/utils_test.go and rebuilt on top of generateCertificateTemplate. crl_test.go no longer carries any fixture helpers of its own.
  • TestCheckChainCRL and TestRegisterCRLMetrics are now table driven, which removed most of the duplication you pointed at.
  • Added a root -> intermediate -> leaf test that gives each level its own CRL and asserts the intermediate is checked against the root's CRL.
  • Dropped the stale comment about the CRL fetch timeout, that was left over from removing the timeout.

Three things where I deviated from the review, all for reasons worth flagging:

Proxy config only exists on the HTTP probe: Only HTTPProbe carries an HTTPClientConfig (which embeds ProxyConfig); TCP, Unix and gRPC only have a TLSConfig, so there is no proxy to honour there and they pass nil. Worth noting this changes behaviour slightly for HTTP: the fetch previously used Go's default transport, which picks up HTTP_PROXY from the environment. It now follows the probe's proxy configuration, so getting the environment proxy needs proxy_from_environment: true, the same as for the probe request itself. Let me know if you'd rather keep the old implicit behaviour.

TCP and Unix accept check_revoked with tls: false if a query_response step uses starttls: CRL checking runs on the STARTTLS-upgraded connection too, so rejecting purely on tls: false would have broken valid STARTTLS setups. gRPC has no STARTTLS path and is validated on tls alone.

The new certificate helper doesn't reuse generateSignedCertificate: That helper takes a parentCert but discards it (generateCertificate signs template with template as parent), so certificates come out with self-referential issuer names. That's fine for the existing tests but not for a chain test where each certificate has to be checked against its actual issuer's CRL, so generateCRLTestCert honours the parent. Happy to fix generateCertificate instead if you'd prefer that, though it touches tests beyond this PR.

Check the full certificate chain against CRLs across the HTTP, TCP, gRPC, and Unix probes. For each cert with CRLDistributionPoints, fetch the CRL, verify its signature against the issuer, reject CRLs that are stale or not yet valid, and look up the cert serial in the revocation list. Enabled per probe via the opt-in `check_revoked` flag (default false); the CRL fetch is bounded by the probe timeout, no separate knob. DNS over TLS is not covered yet.

Signed-off-by: Sebastian Gaiser <sebastiangaiser@users.noreply.github.com>

@electron0zero electron0zero left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, thanks for the PR :shipit:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants