feat(RHINENG-21958): Implement database migration locking

[ezr-ondrej]

Adds PostgreSQL advisory lock mechanism to prevent race conditions when multiple pods attempt to run migrations simultaneously during deployments.

• Lock acquired before any migration work (including version table checks)
• Configurable timeout (MIGRATION_LOCK_TIMEOUT, default 30 min)
• Retry logic with configurable interval (MIGRATION_LOCK_RETRY_INTERVAL, default 15s)
• Enhanced logging for lock acquisition, wait times, and release
• Graceful pod exit if lock cannot be acquired
• Lock always released in finally block to prevent deadlocks

🤖 Generated with Claude Code

This PR is being created to address RHINENG-21958.

Summary by Sourcery

Serialize database migrations across multiple pods using PostgreSQL advisory locks, with configurable timeout and retry behavior and improved logging to ensure safe and observable deployments

New Features:

• Introduce PostgreSQL advisory locking for Alembic migrations to prevent concurrent execution
• Add configurable lock timeout (MIGRATION_LOCK_TIMEOUT) and retry interval (MIGRATION_LOCK_RETRY_INTERVAL) via environment variables
• Implement retry logic and exit the pod gracefully if the migration lock cannot be obtained

Enhancements:

• Enhance logging for lock acquisition attempts, wait status, migration start/completion, and lock release
• Guarantee advisory lock release in a finally block to avoid deadlocks

[sourcery-ai]

Reviewer's Guide

This PR adds a PostgreSQL advisory lock mechanism around Alembic migrations to prevent concurrent migrations, featuring a computed lock ID, configurable timeout and retry interval, retry logic with detailed logging, graceful exit on failure, and guaranteed lock release in a finally block.

Sequence diagram for migration lock acquisition and release during Alembic migrations

sequenceDiagram
    participant Pod
    participant DB["PostgreSQL"]
    participant Alembic
    Pod->DB: Try to acquire advisory lock (pg_try_advisory_lock)
    alt Lock acquired
        Pod->Alembic: Run migrations
        Alembic->DB: Execute migration SQL
        Pod->DB: Release advisory lock (pg_advisory_unlock)
    else Lock not acquired (timeout)
        Pod->DB: Retry until timeout
        Pod->Pod: Log and exit gracefully
    end

Loading

Class diagram for migration lock logic in env.py

classDiagram
    class env {
        +acquire_migration_lock(conn: Connection, timeout: int) bool
        +release_migration_lock(conn: Connection)
        MIGRATION_LOCK_ID: int
        MIGRATION_LOCK_TIMEOUT: int
        MIGRATION_LOCK_RETRY_INTERVAL: int
    }
    class AlembicMigration {
        +migrate_alembic_version_table(conn: Connection)
    }
    env <|-- AlembicMigration

Loading

File-Level Changes

Change	Details	Files
Add migration lock configuration	Compute a unique MIGRATION_LOCK_ID from a fixed string using MD5 Introduce MIGRATION_LOCK_TIMEOUT and MIGRATION_LOCK_RETRY_INTERVAL from env vars	migrations/env.py
Implement lock acquisition utility	Define acquire_migration_lock(conn, timeout) with retry loop Use pg_try_advisory_lock for non-blocking lock attempts Log attempts, wait status, and error on timeout	migrations/env.py
Implement lock release utility	Define release_migration_lock(conn) to call pg_advisory_unlock Log success, warning if lock not held, and catch exceptions	migrations/env.py
Integrate advisory lock into migration flow	Call acquire_migration_lock before any migration work and exit on failure Remove inline SQL locks and replace with lock utilities Wrap migrations in try/finally to always call release_migration_lock Add logging for migration start and completion	migrations/env.py

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

New security issues found

[sourcery-ai]

security (python.sqlalchemy.security.sqlalchemy-execute-raw-query): Avoiding SQL string concatenation: untrusted input concatenated with raw SQL query can result in SQL Injection. In order to execute raw query safely, prepared statement should be used. SQLAlchemy provides TextualSQL to easily used prepared statement with named parameters. For complex SQL composition, use SQL Expression Language or Schema Definition Language. In most cases, SQLAlchemy ORM will be a better option.

Source: opengrep

[sourcery-ai]

security (python.sqlalchemy.security.sqlalchemy-execute-raw-query): Avoiding SQL string concatenation: untrusted input concatenated with raw SQL query can result in SQL Injection. In order to execute raw query safely, prepared statement should be used. SQLAlchemy provides TextualSQL to easily used prepared statement with named parameters. For complex SQL composition, use SQL Expression Language or Schema Definition Language. In most cases, SQLAlchemy ORM will be a better option.

Source: opengrep

[ezr-ondrej]

Ah, we already do have a lock! Interesting :)