Skip to content

database: add failover-safe pool defaults and pgxpool health checks for Postgres/AlloyDB#490

Open
stevemsmith wants to merge 4 commits intomoov-io:masterfrom
stevemsmith:fix/alloydb-failover-recovery
Open

database: add failover-safe pool defaults and pgxpool health checks for Postgres/AlloyDB#490
stevemsmith wants to merge 4 commits intomoov-io:masterfrom
stevemsmith:fix/alloydb-failover-recovery

Conversation

@stevemsmith
Copy link

Summary

  • Switch Postgres/AlloyDB connections from sql.Open("pgx") to pgxpool with stdlib.OpenDBFromPool() — enables HealthCheckPeriod (1s) to proactively evict dead connections after a failover, while keeping the *sql.DB return type so no downstream changes are needed
  • Add safe connection pool defaults (MaxLifetime=5m, MaxIdleTime=30s, MaxOpen=25, MaxIdle=5) that are applied automatically when services don't explicitly configure them
  • Add IsRetryablePostgresError() and RetryPostgres() utilities for services that want to retry transient connection errors on in-flight queries

Problem

During AlloyDB monthly maintenance switchovers (< 1s downtime), Go services using this library fail to recover because:

  1. No health checkingdatabase/sql with pgx as a driver has no background connection health checks. After a switchover, dead connections sit in the pool and get handed to the application, causing errors.
  2. No pool limits by defaultConnectionsConfig defaults to all zero values, meaning database/sql holds connections forever. Dead connections from before the switchover are never evicted.
  3. No retry logic — in-flight queries that fail due to the connection being severed have no retry path.

Changes

postgres.go

  • Replace sql.Open("pgx", connStr) with pgxpool.NewWithConfig() + stdlib.OpenDBFromPool()
  • Set HealthCheckPeriod = 1s — pgxpool pings idle connections every second and evicts dead ones before the app sees them
  • Refactor getAlloyDBConnectorConnStr()buildAlloyDBPoolConfig() to return *pgxpool.Config instead of a connection string
  • Add IsRetryablePostgresError() — classifies PG error codes (57P01 admin_shutdown, 57P03 cannot_connect_now, 08xxx connection class) and network errors (EOF, broken pipe, connection reset) as retryable
  • Add RetryPostgres() — opt-in retry wrapper with linear backoff for critical DB operations

model_config.go

  • Add DefaultPostgresConnectionsConfig() with failover-safe defaults

database.go

  • Add ApplyPostgresConnectionsConfig() that fills in defaults for zero-valued config fields
  • Postgres path in New() now uses this instead of the generic ApplyConnectionsConfig

Non-breaking

  • Return type of New() is still *sql.DB — no downstream changes needed
  • Services that explicitly set pool config values keep their values; defaults only fill in zeros
  • MySQL and Spanner paths are untouched
  • getPostgresConnStr() is unchanged (just moved in file)
  • No new external dependencies — pgxpool and stdlib are both part of pgx/v5

Test plan

  • Unit tests for IsRetryablePostgresError — all PG error codes, network errors, non-retryable cases
  • Unit tests for RetryPostgres — success, retry-then-succeed, non-retryable short-circuit, context cancellation, attempt exhaustion
  • Unit tests for DefaultPostgresConnectionsConfig and ApplyPostgresConnectionsConfig
  • go build ./database/... and go vet ./database/... pass
  • Integration test during next AlloyDB maintenance window

🤖 Generated with Claude Code

stevemsmith and others added 3 commits March 17, 2026 12:20
…oyDB

During AlloyDB maintenance switchovers (< 1s downtime), services using
database/sql with pgx fail to recover because:
1. No connection pool limits are set by default, so dead connections
   persist indefinitely
2. No retry logic exists for transient connection errors

This adds:
- DefaultPostgresConnectionsConfig() with MaxLifetime=5m, MaxIdleTime=30s
  to ensure dead connections are evicted quickly after failover
- ApplyPostgresConnectionsConfig() that fills in safe defaults when
  services don't explicitly configure pool settings
- IsRetryablePostgresError() to classify transient PG/network errors
- RetryPostgres() for services to wrap critical DB operations

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Switch from sql.Open("pgx") to pgxpool.NewWithConfig() wrapped with
stdlib.OpenDBFromPool(). This gives us pgxpool's HealthCheckPeriod
(set to 5s) which proactively pings idle connections and evicts dead
ones — the most important fix for surviving AlloyDB maintenance
switchovers. The return type remains *sql.DB so no downstream changes
are needed.

Also cleans up the dialer TODO (dialer lifecycle is now tied to the
pool) and removes the unused pgx.ParseConfig import path.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
For sub-second AlloyDB switchovers, 1s health checks detect and evict
dead connections faster with negligible overhead.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@gemini-code-assist
Copy link

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly improves the robustness of database connections for Postgres and AlloyDB by implementing proactive health checks, establishing intelligent default connection pool settings, and providing utilities to automatically retry operations affected by transient connection issues. These changes aim to enhance service stability and recovery during database failovers or brief network disruptions, without requiring modifications to existing application code.

Highlights

  • Enhanced Postgres/AlloyDB Connection Resilience: Switched Postgres/AlloyDB connections from sql.Open("pgx") to pgxpool with stdlib.OpenDBFromPool(), enabling HealthCheckPeriod (1s) to proactively evict dead connections after a failover, while maintaining *sql.DB return type for compatibility.
  • Failover-Safe Connection Pool Defaults: Added safe connection pool defaults (MaxLifetime=5m, MaxIdleTime=30s, MaxOpen=25, MaxIdle=5) that are automatically applied when services do not explicitly configure them, preventing dead connections from lingering.
  • Transient Error Retry Utilities: Introduced IsRetryablePostgresError() to classify transient connection errors (e.g., admin_shutdown, cannot_connect_now, network errors) and RetryPostgres() for opt-in retry logic with linear backoff for critical database operations.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog
  • database/database.go
    • Updated the New function to utilize ApplyPostgresConnectionsConfig for Postgres connections.
    • Added ApplyPostgresConnectionsConfig to apply default connection pool settings for Postgres/AlloyDB, filling in zero-valued configuration fields with predefined safe defaults.
  • database/database_test.go
    • Added TestApplyPostgresConnectionsConfig_Defaults to verify that default connection settings are correctly applied.
    • Added TestApplyPostgresConnectionsConfig_Overrides to confirm that explicitly set connection values override defaults.
    • Added TestDefaultPostgresConnectionsConfig to validate the properties of the default Postgres connection configuration.
  • database/model_config.go
    • Introduced DefaultPostgresConnectionsConfig function, providing failover-tuned default values for Postgres connection pool settings.
  • database/postgres.go
    • Refactored postgresConnection to leverage pgxpool.NewWithConfig and stdlib.OpenDBFromPool for enhanced connection management and health checking.
    • Configured pgxpool with a HealthCheckPeriod of 1 second to proactively ping and evict dead connections.
    • Renamed and refactored getAlloyDBConnectorConnStr to buildAlloyDBPoolConfig to return a *pgxpool.Config.
    • Introduced buildPgxPoolConfig to abstract the creation of pgxpool.Config for both standard Postgres and AlloyDB connections.
    • Added IsRetryablePostgresError function to identify transient PostgreSQL error codes and network errors that are safe to retry.
    • Implemented RetryPostgres utility for executing database operations with linear backoff on retryable errors.
  • database/postgres_test.go
    • Added comprehensive unit tests for IsRetryablePostgresError, covering various PostgreSQL error codes, network errors, and non-retryable scenarios.
    • Added unit tests for RetryPostgres to validate successful execution, retry behavior, non-retryable error handling, context cancellation, and attempt exhaustion.
Activity
  • No specific activity (comments, reviews, or progress updates) has been recorded for this pull request yet.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces robust failover handling for Postgres/AlloyDB connections by switching to pgxpool, which enables proactive health checks to evict dead connections. It also adds sensible default connection pool settings tuned for failover scenarios and an opt-in retry mechanism for transient connection errors. The changes are well-implemented, non-breaking for consumers of the library, and thoroughly tested. My feedback includes a couple of suggestions to improve code conciseness and strengthen test assertions.

adamdecaf
adamdecaf previously approved these changes Mar 17, 2026
@stevemsmith
Copy link
Author

CI is failing on a pre-existing vulnerability in golang.org/x/net@v0.50.0 (GO-2026-4559) — unrelated to the changes in this PR. Looks like #487 (Renovate) already bumps this dependency.

- Collapse switch cases in IsRetryablePostgresError for readability
- Make context cancellation test more specific (assert context.Canceled
  and exact call count)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@adamdecaf
Copy link
Member

Can we try this in some apps before merging?

@adamdecaf
Copy link
Member

I'll work with gosec/golangci-lint on getting the linter fix released.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants