You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
PostgresCollector emits no metrics at all — not even pg_scrape_collector_success=0 — when the per-scrape connection fails, while pg_up stays 1 #1387
When PostgresCollector cannot open its per-scrape connection, it emits nothing at all — not even pg_scrape_collector_success{collector="…"} 0. The failure exists only as a log line.
func (pPostgresCollector) Collect(chchan<- prometheus.Metric) {
// copy the instance so that concurrent scrapes have independent instancesinst:=p.instance.copy()
// Set up the database connection for the collector.err:=inst.setup()
deferinst.Close()
iferr!=nil {
p.logger.Error("Error opening connection to database", "err", err)
return
}
p.collectFromConnection(inst, ch)
}
pg_scrape_collector_success is produced by execute(), which is only reached from collectFromConnection. So the return above skips the one metric that is supposed to report a collector problem.
At the same time the other half of the exporter — the Exporter registered alongside PostgresCollector in cmd/postgres_exporter/main.go — holds a persistent connection per server. It therefore keeps reporting pg_up 1 and pg_exporter_last_scrape_error 0 while every --collector.* metric is missing.
Net effect: the exporter reports "up, no scrape errors" while roughly half of its metric surface has silently disappeared. The only in-band signal is the absence of series.
Why it matters (measured)
On 2026-09-07 a shared development PostgreSQL of ours refused new connections for just under three hours (FATAL: sorry, too many clients already, ~39.7k occurrences). Over that window:
Source
Behaviour
all 18 --collector.* collectors
pg_scrape_collector_success = 0 on 49 samples, plus outright gaps 05:16→07:49 (scrapes where inst.setup() failed and Collect returned early)
pg_up, pg_exporter_last_scrape_error
2400 samples at 15s, no gaps, values 1 and 0 for the entire outage
a --extend.query-path user query
2399 samples at 15s, no gaps
Because collector metrics vanish rather than report a failure, an alert written on them (in our case connection saturation, sum(pg_stat_activity_count) / pg_settings_max_connections > 0.7) evaluates to "no data" and resolves itself in the middle of the incident, exactly when the condition is at its worst. pg_up-based and pg_exporter_last_scrape_error-based alerts do not compensate, because those come from the other registration and stay green.
Connection exhaustion is only the most obvious trigger; anything that makes inst.setup() fail (auth change, pooler saturation, TLS) has the same shape.
Suggestion
Make the failure representable in metrics rather than only in logs. Either of:
On inst.setup() error, emit pg_scrape_collector_success{collector=…} 0 for every enabled collector before returning; or
Add a dedicated gauge, e.g. pg_scrape_connection_error, set to 1 on that path.
(1) needs no new metric name and makes existing dashboards/alerts on pg_scrape_collector_success correct. Happy to send a PR if the maintainers prefer one of the two.
Activity
collector: report pg_scrape_collector_success=0 on connection failure
collector: report pg_scrape_collector_success=0 on connection failure
collector: report pg_scrape_collector_success=0 on connection failure