Support for happy eyeballs multi-IP failover for HTTP probes (probe_success false negatives)

[tadkarshirish]

Blackbox exporter probes only one DNS A record (no failover), causing false negatives for multi-AZ ALB endpoints

Description
As per document ,blackbox_exporter performs a single DNS resolution and attempts a single connection to one IP address. There is no “happy eyeballs” style parallelization or retry.

This behavior causes issues when monitoring DNS names backed by multiple IPs (for example AWS ALB across AZs), where one IP may be temporarily unreachable while others are healthy.

What did you do?

Deployed blackbox_exporter to monitor an HTTPS endpoint (https://www.url.com) that is fronted by an AWS Application Load Balancer (ALB) spanning two Availability Zones.

The DNS name resolves to two IPs:

$ nslookup www.url.com
Name: www.url.com
Address: 123.12.12.01
Name: www.url.com
Address: 123.12.12.02

One of the IPs intermittently times out from the EKS cluster (cross-AZ), while the other IP is healthy.

When probing with blackbox exporter:

curl "http://blackbox:9115/probe?target=https://www.url.com&module=http_2xx"

the exporter attempts only one of the IPs and fails if that IP times out.

What did you expect to see?

probe_success = 1

when any resolved IP responds successfully (similar to standard HTTP client behavior such as curl or browsers, which retry/fail over to another IP).

What did you see instead? Under which circumstances?

probe_success = 0 when the first chosen IP times out, even though another IP for the same DNS name is healthy and serving traffic.

This happens when:

DNS name has multiple A records

One IP is unreachable or slow

Blackbox exporter does not retry or attempt the next IP

Reproduction steps

Configure an ALB-backed DNS name with multiple A records

Ensure one IP is slow/unreachable from the blackbox exporter pod

Run:

nslookup www.url.com
curl -v https://www.url.com        # succeeds after retrying another IP
curl http://blackbox:9115/probe?target=https://www.url.com&module=http_2xx

Observe:
curl/browser succeeds
blackbox exporter returns probe_success = 0

Additional context

We understand that this behavior may be intentional and that blackbox exporter is designed to be deterministic and infrastructure-focused.

However, this commonly leads to confusion and “false negative” alerts when monitoring:

AWS ALBs / NLBs
Multi-AZ services
DNS-based load balancing

We are looking for:

Recommended best practices for monitoring such endpoints

[roidelapluie]

This is expected behavior from blackbox_exporter: it performs a single DNS resolution and probes only one IP address, with no retry or failover across multiple A records. For DNS-load-balanced endpoints such as AWS ALBs, this can result in false negatives when one IP is unhealthy while others are still serving traffic.

The recommended approach is to probe each DNS A record individually using dns_sd_configs, and then express the desired semantics at the PromQL / alerting layer. This avoids flakiness, makes failures explicit, and still allows detection of partial AZ or IP outages.

With the configuration above, Prometheus creates one target per resolved IP, while preserving the original hostname for the HTTP Host header and TLS SNI via __meta_dns_name.

You can then alert using clear aggregation rules:

• Endpoint is up if any IP is healthy

  max without (instance) (probe_success) == 1
  

• Partial failure detected (one or more IPs failing)

  min without (instance) (probe_success) == 0
  

This aligns blackbox_exporter with multi-AZ ALBs and DNS-based load balancing, without changing its intentionally deterministic probing model.

Prometheus config:

# Prometheus: probe every A record, using __meta_dns_name as the hostname (Host + SNI)
scrape_configs:
  - job_name: blackbox-per-a-record
    metrics_path: /probe
    params:
      module: [http_2xx]

    dns_sd_configs:
      - names: ["www.url.com"]
        type: A
        port: 443

    relabel_configs:
      # Probe the discovered IP:port
      - source_labels: [__address__]
        target_label: __param_target
        replacement: https://$1

      # Use the original DNS name for Host header + TLS SNI
      - source_labels: [__meta_dns_name]
        target_label: __param_hostname

      # Keep IP:port as instance label
      - source_labels: [__address__]
        target_label: instance

      # Send scrape to blackbox exporter
      - target_label: __address__
        replacement: blackbox.monitoring.svc:9115

[tadkarshirish]

Thank you for this quick response, will try this approach.

This is expected behavior from blackbox_exporter: it performs a single DNS resolution and probes only one IP address, with no retry or failover across multiple A records. For DNS-load-balanced endpoints such as AWS ALBs, this can result in false negatives when one IP is unhealthy while others are still serving traffic.

The recommended approach is to probe each DNS A record individually using dns_sd_configs, and then express the desired semantics at the PromQL / alerting layer. This avoids flakiness, makes failures explicit, and still allows detection of partial AZ or IP outages.

With the configuration above, Prometheus creates one target per resolved IP, while preserving the original hostname for the HTTP Host header and TLS SNI via __meta_dns_name.

You can then alert using clear aggregation rules:

• Endpoint is up if any IP is healthy

  max without (instance) (probe_success) == 1
  

• Partial failure detected (one or more IPs failing)

  min without (instance) (probe_success) == 0
  

This aligns blackbox_exporter with multi-AZ ALBs and DNS-based load balancing, without changing its intentionally deterministic probing model.

Prometheus config:

# Prometheus: probe every A record, using __meta_dns_name as the hostname (Host + SNI)
scrape_configs:
  - job_name: blackbox-per-a-record
    metrics_path: /probe
    params:
      module: [http_2xx]

    dns_sd_configs:
      - names: ["www.url.com"]
        type: A
        port: 443

    relabel_configs:
      # Probe the discovered IP:port
      - source_labels: [__address__]
        target_label: __param_target
        replacement: https://$1

      # Use the original DNS name for Host header + TLS SNI
      - source_labels: [__meta_dns_name]
        target_label: __param_hostname

      # Keep IP:port as instance label
      - source_labels: [__address__]
        target_label: instance

      # Send scrape to blackbox exporter
      - target_label: __address__
        replacement: blackbox.monitoring.svc:9115

[tadkarshirish]

@roidelapluie

this config is working fine , giving proper metrics thank you , but we have multiple urls with has custom labels

like
ur1
app: appname1
env: environmantname1
svc: servicename1

url2
app: appname2
env: environmantname2
svc: servicename2

Any way to add those custome labels to metrics ?

# Prometheus: probe every A record, using __meta_dns_name as the hostname (Host + SNI)
scrape_configs:
  - job_name: blackbox-per-a-record
    metrics_path: /probe
    params:
      module: [http_2xx]

    dns_sd_configs:
      - names: ["www.url.com"]
        type: A
        port: 443

    relabel_configs:
      # Probe the discovered IP:port
      - source_labels: [__address__]
        target_label: __param_target
        replacement: https://$1

      # Use the original DNS name for Host header + TLS SNI
      - source_labels: [__meta_dns_name]
        target_label: __param_hostname

      # Keep IP:port as instance label
      - source_labels: [__address__]
        target_label: instance

      # Send scrape to blackbox exporter
      - target_label: __address__
        replacement: blackbox.monitoring.svc:9115

[roidelapluie]

You mean just use:

- target_labels: app
  replacement: app1

or, if you want multiples names in the same job:

dns_sd_configs:
  - names: ["www.url.com", "prometheus.io"]
    type: A
    port: 443
relabel_configs:
- source_labels: [__meta_dns_name]
  regex: example.com
  target_labels: app
  replacement: app1
- source_labels: [__meta_dns_name]
  regex: prometheus.io
  target_labels: app
  replacement: app2

[tadkarshirish]

@roidelapluie we have existing url monitoring job which has more than 50-60 urls , and

For those
We have added targets using file_sd_config for probe job

`- targets:
- www.url1.com
labels:
app: app1
env: env1
svc: svc1

• targets:
  - www.url2.com
  labels:
  app: app2
  env: env2
  svc: svc2
• targets:
  - www.url3.com
  labels:
  app: app3
  env: env3
  svc: svc3`

These labels appear dynamically in the metrics. Adding relabel_configs with replacements for all 60 URLs would make the config very large and confusing.

Any way to provide labels for each Target which can feteched and added automatically to metrics.

Example

`dns_sd_configs:

• names: ["www.url.com"]
  type: A
  port: 443
  labels:
  app:app1
  env:env1
• names: ["www.url2.com"]
  type: A
  port: 443
  labels:
  app:app2
  env:env2`

You mean just use:

- target_labels: app
  replacement: app1

or, if you want multiples names in the same job:

dns_sd_configs:
  - names: ["www.url.com", "prometheus.io"]
    type: A
    port: 443
relabel_configs:
- source_labels: [__meta_dns_name]
  regex: example.com
  target_labels: app
  replacement: app1
- source_labels: [__meta_dns_name]
  regex: prometheus.io
  target_labels: app
  replacement: app2

[roidelapluie]

There is no way to do this easily in plain Prometheus today.

dns_sd_configs does not support attaching arbitrary per-target labels (unlike file_sd_configs), and Prometheus does not support chaining service discovery mechanisms (e.g. file_sd → dns_sd). As a result, per-URL labels must either be encoded via relabel_configs or split across multiple scrape jobs.

The closest workarounds, while staying fully within Prometheus, are:

• Offload the scrape configuration itself into separate files using scrape_config_files:.
  This allows you to define multiple scrape jobs with static labels in a way that is syntactically close to your existing file_sd_config, without polluting the main prometheus.yml.

  Example:

  scrape_config_files:
    - /etc/prometheus/blackbox-scrapes/*.yml

  Each file can then define one labeled scrape job using dns_sd_configs.
  If you want to keep the exact same metric shape as before, you can override the automatically assigned job label in relabel_configs (e.g. setting it back to the original job name), so dashboards and alerts continue to work unchanged.
  This keeps per-URL labels readable and maintainable, at the cost of having multiple scrape jobs internally.

• Alternatively, keep a single scrape job and encode all per-URL label mappings in relabel_configs, also offloaded into a separate file via scrape_config_files:.
  This preserves a single job, but still requires explicitly enumerating or grouping hostnames with relabel rules.

Beyond these options, there is no native Prometheus solution that combines:

• per-A-record probing via DNS SD, and
• per-target arbitrary labels,

without either relabel rule enumeration or external service discovery generation.