HTTPS probes with TLS 1.2 fail on 0.26.0 and higher

[rodecker]

Since version 0.26.0, HTTPS probes to TLS1.2 targets (older Dell iDRAC interfaces) are failing.

version:

blackbox_exporter, version 0.26.0 (branch: HEAD, revision: 444e3d089ae1aeeed385712ef84a5fa4f0c083ec)
  build user:       root@87341586bc85
  build date:       20250226-12:29:43
  go version:       go1.23.6
  platform:         linux/amd64
  tags:             unknown

config:

  https_oob_2xx:                  
    prober: http                        
    timeout: 60s                 
    http:                         
      valid_status_codes: [] #default to 2xx
      method: GET                                 
      headers:                              
        Accept-Encoding: gzip                                
      fail_if_not_ssl: true                               
      preferred_ip_protocol: "ip6"                          
      ip_protocol_fallback: true                             
      tls_config:
        insecure_skip_verify: true

output:

/ # blackbox_exporter --web.listen-address=:9142 --log.level=debug --log.prober="debug"
time=2025-07-03T08:37:11.501Z level=INFO source=main.go:86 msg="Starting blackbox_exporter" version="(version=0.26.0, branch=HEAD, revision=444e3d089ae1aeeed385712ef84a5fa4f0c083ec)"
time=2025-07-03T08:37:11.501Z level=INFO source=main.go:87 msg="(go=go1.23.6, platform=linux/amd64, user=root@87341586bc85, date=20250226-12:29:43, tags=unknown)"
time=2025-07-03T08:37:11.501Z level=ERROR source=main.go:90 msg="Error loading config" err="error reading config file: open blackbox.yml: no such file or directory"
/ # blackbox_exporter --config.file /etc/blackbox/blackbox.yml --web.listen-address=:9142 --log.level=debug --log.prober="debug"
time=2025-07-03T08:37:24.137Z level=INFO source=main.go:86 msg="Starting blackbox_exporter" version="(version=0.26.0, branch=HEAD, revision=444e3d089ae1aeeed385712ef84a5fa4f0c083ec)"
time=2025-07-03T08:37:24.137Z level=INFO source=main.go:87 msg="(go=go1.23.6, platform=linux/amd64, user=root@87341586bc85, date=20250226-12:29:43, tags=unknown)"
time=2025-07-03T08:37:24.138Z level=INFO source=main.go:99 msg="Loaded config file"
time=2025-07-03T08:37:24.138Z level=DEBUG source=main.go:114 msg=http://4abf5ec952d9:9142
time=2025-07-03T08:37:24.138Z level=DEBUG source=main.go:128 msg=/
time=2025-07-03T08:37:24.139Z level=INFO source=tls_config.go:347 msg="Listening on" address=[::]:9142
time=2025-07-03T08:37:24.139Z level=INFO source=tls_config.go:350 msg="TLS is disabled." http2=false address=[::]:9142
time=2025-07-03T08:37:37.818Z level=INFO source=handler.go:122 msg="Beginning probe" module=https_oob_2xx target=https://my.target/ probe=http timeout_seconds=60
time=2025-07-03T08:37:37.818Z level=INFO source=utils.go:61 msg="Resolving target address" module=https_oob_2xx target=https://my.target/ target=my.target ip_protocol=ip6
time=2025-07-03T08:37:37.819Z level=INFO source=utils.go:130 msg="Resolved target address" module=https_oob_2xx target=https://my.target/ target=my.target ip=x.x.x.x
time=2025-07-03T08:37:37.819Z level=INFO source=http.go:153 msg="Making HTTP request" module=https_oob_2xx target=https://my.target/ url=https://x.x.x.x/ host=my.target
time=2025-07-03T08:37:37.876Z level=ERROR source=http.go:474 msg="Error for HTTP request" module=https_oob_2xx target=https://my.target/ err="Get \"https://x.x.x.x/\": remote error: tls: handshake failure"
time=2025-07-03T08:37:37.876Z level=INFO source=http.go:601 msg="Response timings for roundtrip" module=https_oob_2xx target=https://my.target/ roundtrip=0 start=2025-07-03T08:37:37.820Z dnsDone=2025-07-03T08:37:37.820Z connectDone=2025-07-03T08:37:37.842Z gotConn=0001-01-01T00:00:00.000Z responseStart=0001-01-01T00:00:00.000Z tlsStart=2025-07-03T08:37:37.842Z tlsDone=2025-07-03T08:37:37.876Z end=0001-01-01T00:00:00.000Z
time=2025-07-03T08:37:37.876Z level=ERROR source=handler.go:135 msg="Probe failed" module=https_oob_2xx target=https://my.target/ duration_seconds=0.058089777

Setting tlsconfig.min_version and tlsconfig.max_version to TLS12 does not help.

Version 0.27.0 has the same problem.

On version 0.25.0 the probe succeeds:

/ # blackbox_exporter --config.file /etc/blackbox/blackbox.yml --web.listen-address=:9142 --log.level=debug --log.prober="debug"                                                                                                                          
ts=2025-07-03T08:46:41.366Z caller=main.go:87 level=info msg="Starting blackbox_exporter" version="(version=0.25.0, branch=HEAD, revision=ef3ff4fef195333fb8ee0039fb487b2f5007908f)"                                                                      
ts=2025-07-03T08:46:41.366Z caller=main.go:88 level=info build_context="(go=go1.22.2, platform=linux/amd64, user=root@47d5b0d99f18, date=20240409-12:58:39, tags=unknown)"                                                                                
ts=2025-07-03T08:46:41.368Z caller=main.go:100 level=info msg="Loaded config file"
ts=2025-07-03T08:46:41.368Z caller=main.go:115 level=debug externalURL=http://76edb406fd2e:9142
ts=2025-07-03T08:46:41.368Z caller=main.go:129 level=debug routePrefix=/
ts=2025-07-03T08:46:41.368Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9142
ts=2025-07-03T08:46:41.368Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9142
ts=2025-07-03T08:47:15.936Z caller=level.go:71 module=https_oob_2xx target=https://my.target/ level=info msg="Beginning probe" probe=http timeout_seconds=60
ts=2025-07-03T08:47:15.936Z caller=level.go:71 module=https_oob_2xx target=https://my.target/ level=info msg="Resolving target address" target=my.target ip_protocol=ip6
ts=2025-07-03T08:47:15.937Z caller=level.go:71 module=https_oob_2xx target=https://my.target/ level=info msg="Resolved target address" target=my.target ip=x.x.x.x
ts=2025-07-03T08:47:15.938Z caller=level.go:71 module=https_oob_2xx target=https://my.target/ level=info msg="Making HTTP request" url=https://x.x.x.x/ host=my.target
ts=2025-07-03T08:47:16.298Z caller=level.go:71 module=https_oob_2xx target=https://my.target/ level=info msg="Received redirect" location=https://my.target/start.html
ts=2025-07-03T08:47:16.298Z caller=level.go:71 module=https_oob_2xx target=https://my.target/ level=info msg="Making HTTP request" url=https://my.target/start.html host=
ts=2025-07-03T08:47:16.298Z caller=level.go:71 module=https_oob_2xx target=https://my.target/ level=info msg="Address does not match first address, not sending TLS ServerName" first=x.x.x.x address=my.target
ts=2025-07-03T08:47:16.660Z caller=level.go:71 module=https_oob_2xx target=https://my.target/ level=info msg="Received HTTP response" status_code=200
ts=2025-07-03T08:47:16.660Z caller=level.go:71 module=https_oob_2xx target=https://my.target/ level=info msg="Response timings for roundtrip" roundtrip=0 start=2025-07-03T08:47:15.9380914Z dnsDone=2025-07-03T08:47:15.9380914Z connectDone=2025-07-03T08:47:15.959242001Z gotConn=2025-07-03T08:47:16.272380147Z responseStart=2025-07-03T08:47:16.298528926Z tlsStart=2025-07-03T08:47:15.95927469Z tlsDone=2025-07-03T08:47:16.27235491Z end=0001-01-01T00:00:00Z
ts=2025-07-03T08:47:16.660Z caller=level.go:71 module=https_oob_2xx target=https://my.target/ level=info msg="Response timings for roundtrip" roundtrip=1 start=2025-07-03T08:47:16.298993668Z dnsDone=2025-07-03T08:47:16.299920125Z connectDone=2025-07-03T08:47:16.320924522Z gotConn=2025-07-03T08:47:16.628307118Z responseStart=2025-07-03T08:47:16.660030683Z tlsStart=2025-07-03T08:47:16.320947084Z tlsDone=2025-07-03T08:47:16.628287292Z end=2025-07-03T08:47:16.660831925Z
ts=2025-07-03T08:47:16.661Z caller=level.go:71 module=https_oob_2xx target=https://my.target/ level=info msg="Probe succeeded" duration_seconds=0.724730993

[JDA88]

We do have the same exact issue running in Windows. Logs do not show why the handshake failed.

How can I get more info to help you fix this? Rolling back to 0.25 in the meantime.

[valutcizen]

Connection reset by peer with IIS simple page

[ricardomolendijk]

Bump, we are also experiencing this issue.

[arminpkathrein]

Same here, I reverted to 0.25 for the "probe_success" to work. 0.26 nor 0.27 are working at the moment.

[JDA88]

Anyone had a chance to test with 0.28?

[electron0zero]

If min_version is unset, we will use Go default minimum version, which is TLS 1.2.

I am looking at the https://pkg.go.dev/crypto/tls#Config, which sets the default TLS Version and it says MinVersion is 1.2 by default and MaxVersion is 1.3 by default.

see this part of the https://pkg.go.dev/crypto/tls#Config page:

	// MinVersion contains the minimum TLS version that is acceptable.
	//
	// By default, TLS 1.2 is currently used as the minimum. TLS 1.0 is the
	// minimum supported by this package.
	//
	// The server-side default can be reverted to TLS 1.0 by including the value
	// "tls10server=1" in the GODEBUG environment variable.
	MinVersion uint16

	// MaxVersion contains the maximum TLS version that is acceptable.
	//
	// By default, the maximum version supported by this package is used,
	// which is currently TLS 1.3.
	MaxVersion uint16

you can also check probe_tls_version_info metric to see what's the TLS version that's being used by the probe and debug the failure. that's will help you figure out if the correct TLS version is being sent to server or not.

We do force the TLS to be 1.3 if you are using HTTP/3 with enable_http3 set to true. we set it to false by default, and I am adding a log line to surface this action in the logs.