Skip to content

PostgresCollector emits no metrics at all — not even pg_scrape_collector_success=0 — when the per-scrape connection fails, while pg_up stays 1 #1387

Open

Description

@Lunatik-006

Version

v0.20.1 (postgres_exporter_build_info{version="0.20.1", revision="867fbcac31cd18c143e244190ea9168cca069827"}), PostgreSQL 17.10.

What happens

When PostgresCollector cannot open its per-scrape connection, it emits nothing at all — not even pg_scrape_collector_success{collector="…"} 0. The failure exists only as a log line.

https://github.com/prometheus-community/postgres_exporter/blob/v0.20.1/collector/collector.go#L183-L195

func (p PostgresCollector) Collect(ch chan<- prometheus.Metric) {
	// copy the instance so that concurrent scrapes have independent instances
	inst := p.instance.copy()

	// Set up the database connection for the collector.
	err := inst.setup()
	defer inst.Close()
	if err != nil {
		p.logger.Error("Error opening connection to database", "err", err)
		return
	}
	p.collectFromConnection(inst, ch)
}

pg_scrape_collector_success is produced by execute(), which is only reached from collectFromConnection. So the return above skips the one metric that is supposed to report a collector problem.

At the same time the other half of the exporter — the Exporter registered alongside PostgresCollector in cmd/postgres_exporter/main.go — holds a persistent connection per server. It therefore keeps reporting pg_up 1 and pg_exporter_last_scrape_error 0 while every --collector.* metric is missing.

Net effect: the exporter reports "up, no scrape errors" while roughly half of its metric surface has silently disappeared. The only in-band signal is the absence of series.

Why it matters (measured)

On 2026-09-07 a shared development PostgreSQL of ours refused new connections for just under three hours (FATAL: sorry, too many clients already, ~39.7k occurrences). Over that window:

Source Behaviour
all 18 --collector.* collectors pg_scrape_collector_success = 0 on 49 samples, plus outright gaps 05:16→07:49 (scrapes where inst.setup() failed and Collect returned early)
pg_up, pg_exporter_last_scrape_error 2400 samples at 15s, no gaps, values 1 and 0 for the entire outage
a --extend.query-path user query 2399 samples at 15s, no gaps

Because collector metrics vanish rather than report a failure, an alert written on them (in our case connection saturation, sum(pg_stat_activity_count) / pg_settings_max_connections > 0.7) evaluates to "no data" and resolves itself in the middle of the incident, exactly when the condition is at its worst. pg_up-based and pg_exporter_last_scrape_error-based alerts do not compensate, because those come from the other registration and stay green.

Connection exhaustion is only the most obvious trigger; anything that makes inst.setup() fail (auth change, pooler saturation, TLS) has the same shape.

Suggestion

Make the failure representable in metrics rather than only in logs. Either of:

  1. On inst.setup() error, emit pg_scrape_collector_success{collector=…} 0 for every enabled collector before returning; or
  2. Add a dedicated gauge, e.g. pg_scrape_connection_error, set to 1 on that path.

(1) needs no new metric name and makes existing dashboards/alerts on pg_scrape_collector_success correct. Happy to send a PR if the maintainers prefer one of the two.

Activity

  1. added 2 commits that reference this issue on Sep 24, 2026
    d921fdd
    ed5f6bb
  2. added a commit that references this issue on Oct 7, 2026
    f64d04e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions