Skip to content

Parameterise PostgreSQL schema throughout the backend#590

Merged
will-langdale merged 7 commits into
uktrade:mainfrom
will-langdale:claude/awesome-mccarthy-p5Eb8
Jun 9, 2026
Merged

Parameterise PostgreSQL schema throughout the backend#590
will-langdale merged 7 commits into
uktrade:mainfrom
will-langdale:claude/awesome-mccarthy-p5Eb8

Conversation

@will-langdale

@will-langdale will-langdale commented Jun 8, 2026

Copy link
Copy Markdown
Collaborator

MB__SERVER__POSTGRES__DB_SCHEMA was only respected by the SQLAlchemy ORM metadata. All Alembic migrations, several raw SQL strings, and the ingest utilities were hardcoded to "mb".

🛠️ Changes proposed in this pull request

  • env.py: pass db_schema into Alembic config so migrations can read it
  • All 25 migration files: replace schema="mb" kwargs, inline FK strings (e.g. ["mb.table.col"]), and raw SQL table references with the parameterised value via op.get_context().config.get_main_option()
  • Adds an Alembic migration hook to ensure this happens in future migrations
  • First migration: remove CREATE/DROP SCHEMA — the operator is now responsible for creating the schema before running migrations
  • db.py: parameterise the pre-alembic recovery DROP SCHEMA
  • orm.py: fix hardcoded schema on the upload_stages Enum type
  • insert.py, adapter/eval.py: fix schema_name="mb" on temp table calls
  • justfile: use $MB__SERVER__POSTGRES__DB_SCHEMA in the drop recipe

👀 Guidance to review

Alembic post migration generation hook in action:

(matchbox-db) willlangdale@DBT001350 ~/D/matchbox (claude/awesome-mccarthy-p5Eb8)> just migrate generate "Add level_2."
uv run alembic --config "src/matchbox/server/postgresql/alembic.ini" revision --autogenerate -m "Add level_2."
...
INFO  [alembic.autogenerate.compare.tables] Detected added column 'mb.step_from.level_2'
  Generating /Users/willlangdale/Dev/matchbox/src/matchbox/server/postgresql/alembic/versions/5e33e5844258_add_level_2.py ...  done
  Running post write hook 'fix_schema' ...
  done
  Running post write hook 'ruff_format' ...
1 file reformatted
  done
  Running post write hook 'ruff_lint' ...
Found 2 errors (2 fixed, 0 remaining).
  done
"""Add level_2.

Revision ID: 5e33e5844258
Revises: 65dd8d1fe119
Create Date: 2026-06-09 13:11:20.645400

"""

from collections.abc import Sequence

import sqlalchemy as sa
from alembic import op

# revision identifiers, used by Alembic.
revision: str = "5e33e5844258"
down_revision: str | None = "65dd8d1fe119"
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None


def upgrade() -> None:
    """Upgrade schema."""
    schema = op.get_context().config.get_main_option("db_schema")
    # ### commands auto generated by Alembic - please adjust! ###
    op.add_column(
        "step_from", sa.Column("level_2", sa.INTEGER(), nullable=False), schema=schema
    )
    # ### end Alembic commands ###


def downgrade() -> None:
    """Downgrade schema."""
    schema = op.get_context().config.get_main_option("db_schema")
    # ### commands auto generated by Alembic - please adjust! ###
    op.drop_column("step_from", "level_2", schema=schema)
    # ### end Alembic commands ###

🤖 AI declaration

AI written, human reviewed.

✅ Checklist:

  • This is the smallest, simplest solution to the problem
  • I've read our code standards and this code follows them
  • All new code is tested
  • I've updated all relevant documentation (select all that apply)
    • API documentation (docstrings and indexes)
    • Tutorials
    • Developer docs

MB__SERVER__POSTGRES__DB_SCHEMA was only respected by the SQLAlchemy
ORM metadata. All Alembic migrations, several raw SQL strings, and
the ingest utilities were hardcoded to "mb".

- env.py: pass db_schema into Alembic config so migrations can read it
- All 25 migration files: replace schema="mb" kwargs, inline FK strings
  (e.g. ["mb.table.col"]), and raw SQL table references with the
  parameterised value via op.get_context().config.get_main_option()
- First migration: remove CREATE/DROP SCHEMA — the operator is now
  responsible for creating the schema before running migrations
- db.py: parameterise the pre-alembic recovery DROP SCHEMA
- orm.py: fix hardcoded schema on the upload_stages Enum type
- insert.py, adapter/eval.py: fix schema_name="mb" on temp table calls
- justfile: use $MB__SERVER__POSTGRES__DB_SCHEMA in the drop recipe

https://claude.ai/code/session_01NBvRFT32ceNrae2sUCjpzb
@will-langdale will-langdale requested a review from a team as a code owner June 8, 2026 15:56
@datadog-uktrade

This comment has been minimized.

Since migration 1 no longer creates the PostgreSQL schema, it must be
provisioned externally before migrations run.

- db.py: run_migrations() now creates the schema when bootstrapping a
  fresh database (pre-alembic path — the path test fixtures always hit)
- docker-compose.yml: mount a postgres initdb.d script and forward
  MB__SERVER__POSTGRES__DB_SCHEMA into the postgres container
- docker/postgres-init/01-create-schema.sh: new init script that runs
  CREATE SCHEMA IF NOT EXISTS on first container startup
- docs/server/install.md: document the schema creation prerequisite
  for production deployments

https://claude.ai/code/session_01NBvRFT32ceNrae2sUCjpzb
@will-langdale will-langdale force-pushed the claude/awesome-mccarthy-p5Eb8 branch from 48f26ae to 1b71ec5 Compare June 8, 2026 16:16
Comment thread docker-compose.yml Outdated
claude and others added 3 commits June 9, 2026 11:52
- script.py.mako: inject schema variable at the top of upgrade() and
  downgrade() so new migrations always have it available
- alembic.ini: add fix_schema post-write hook (runs before ruff) that
  reads MB__SERVER__POSTGRES__DB_SCHEMA from the environment and
  replaces hardcoded schema strings with the schema variable
- fix_schema_hook.py: the hook script (~8 lines)

https://claude.ai/code/session_01NBvRFT32ceNrae2sUCjpzb
Signed-off-by: DBT pre-commit check
Signed-off-by: DBT pre-commit check
@will-langdale will-langdale merged commit f6c4fd3 into uktrade:main Jun 9, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants