PostgreSQL 17: pg_stat_bgwriter column changes break metrics collection

[gmastakar]

In PostgreSQL 17, several columns have been removed from pg_stat_bgwriter, breaking the default PostgreSQL integration for Datadog when attempting to collect metrics. Upgrading databases to PostgreSQL 17 results in numerous errors from the Datadog agent due to these missing columns. A new view, pg_stat_checkpointer, introduced in PostgreSQL 17 appears to include checkpoint-related metrics, although under different column names.

Details
PostgreSQL (version < 17) columns in pg_stat_bgwriter:

checkpoints_timed
checkpoints_req
checkpoint_write_time
checkpoint_sync_time
buffers_checkpoint
buffers_backend
buffers_backend_fsync
buffers_clean
maxwritten_clean
buffers_alloc
stats_reset

PostgreSQL 17 columns in pg_stat_bgwriter:

buffers_clean
maxwritten_clean
buffers_alloc
stats_reset

Removed Columns in PostgreSQL 17:

• checkpoints_timed
• checkpoints_req
• checkpoint_write_time
• checkpoint_sync_time
• buffers_checkpoint
• buffers_backend
• buffers_backend_fsync

New View pg_stat_checkpointer in PostgreSQL 17:
The newly added pg_stat_checkpointer view appears to include most (or all) checkpoint-related data previously part of pg_stat_bgwriter, although with updated terminology:

 num_timed
 num_requested
 restartpoints_timed
 restartpoints_req
 restartpoints_done
 write_time
 sync_time 
 buffers_written
 stats_reset

Potential mapping to removed columns:
num_timed → checkpoints_timed
num_requested → checkpoints_req
restartpoints_timed → Possibly encompasses prior checkpoint work
write_time → checkpoint_write_time
sync_time → checkpoint_sync_time

Impact

• After upgrading several production databases to PostgreSQL 17, the Datadog PostgreSQL integration has been throwing continuous errors about missing columns in pg_stat_bgwriter.
• This prevents critical checkpoint and writer metrics from being collected for monitoring or analysis.
• Additionally, pg_stat_checkpointer contains data that should now be collected to avoid a loss of visibility into checkpoint-related metrics. Without this fix, monitoring critical health indicators like checkpoint timing, restarts, and buffers written remains broken for PostgreSQL 17 instances.

Current Behavior
The Datadog PostgreSQL integration fails to collect pg_stat_bgwriter metrics in PostgreSQL 17 due to missing columns, resulting in continuous errors from the Datadog agent.

PostgreSQL 17 Compatibility Work in Progress:
We noticed that there is an existing PR in progress to address PostgreSQL 17 compatibility. Additionally, similar column changes have been handled in previous PostgreSQL version upgrades. Given the introduction of pg_stat_checkpointer and changes to pg_stat_bgwriter, additional functionality is needed to incorporate metrics from pg_stat_checkpointer transparently. Seems like this is the PR for PG17 compatibility changes that is in progress: #19625.

Do we know when the PostgreSQL 17 compatibility updates, particularly for pg_stat_bgwriter and integration of pg_stat_checkpointer metrics, will be delivered? We would like to understand the timeline for when this fix will be live so we can plan our monitoring dependencies accordingly.