Skip to content

Fix PostgreSQL 13→16 upgrade path in deploy-dev#668

Merged
ehelms merged 7 commits into
theforeman:masterfrom
jeremylenz:fix-pg-upgrade-path
Jul 17, 2026
Merged

Fix PostgreSQL 13→16 upgrade path in deploy-dev#668
ehelms merged 7 commits into
theforeman:masterfrom
jeremylenz:fix-pg-upgrade-path

Conversation

@jeremylenz

@jeremylenz jeremylenz commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Why are you introducing these changes? (Problem description, related links)

Commit ca470b8 ("Switch to PostgreSQL 16 image") introduced issues that cause forge deploy-dev to fail when re-deploying over an existing PostgreSQL 13 database.

Three problems were identified:

  1. The upgrade.yaml stop task assumes foreman.target exists, but deploy-dev never created it (the systemd_target role was not included)
  2. Without foreman.target, services like pulp-api aren't stopped before re-deploy, causing handler conflicts
  3. The host's pg_dump stays at version 13 after the containerized PG upgrades to 16, breaking db:migrate schema dumps

What are the changes introduced in this pull request?

  • Add failed_when: false to the "Stop foreman target" task in upgrade.yaml — on first deploy, foreman.target doesn't exist yet (it's created by systemd_target which runs after postgresql)
  • Add systemd_target role to deploy-dev.yaml — creates foreman.target so all backing services can be managed as a group, matching the production deploy
  • Add a deploy-dev pre-task to switch the postgresql dnf module to 16 if needed — handles dev re-deploys where the host still has PG 13 client tools

How to test this pull request

  1. Have an existing dev environment deployed with PG 13
  2. Run forge deploy-dev — should complete without errors
  3. Verify cat /var/lib/pgsql/data/userdata/PG_VERSION shows 16
  4. Run forge deploy-dev again — idempotent re-run should succeed

🤖 Generated with Claude Code

Fixes multiple issues preventing deploy-dev from running over existing PG 13 databases:
- Add failed_when: false to service stops (handles missing foreman.target)
- Change upgrade container volume from :Z to :z (fixes SELinux permission denied)
- Replace creates: guard with proper stream check for dnf module switch-to

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
The deploy-dev playbook was missing the systemd_target role, so
foreman.target was never created. This meant services couldn't be
stopped cleanly before re-deploying, causing handler failures (e.g.
pulp-api.socket refusing to start because pulp-api.service was
already active).

Add the systemd_target role (matching production deploy) and a
pre-task to stop foreman.target before re-deploying.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Comment thread src/roles/postgresql/tasks/upgrade.yaml Outdated
Comment thread development/playbooks/deploy-dev/deploy-dev.yaml Outdated
Comment thread src/roles/postgresql/tasks/upgrade.yaml Outdated
Comment thread src/roles/pre_install/tasks/main.yaml Outdated
jeremylenz and others added 3 commits July 17, 2026 09:54
Remove the deploy-dev pre-task that stopped foreman.target before
deployment - the roles should handle idempotent re-deploys.

Remove the explicit postgresql stop from the upgrade path since
postgresql is PartOf=foreman.target and stopping the target
cascades to it.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Revert pre_install changes since production always deploys on clean
systems. Instead, handle the postgresql module switch as a deploy-dev
pre-task for dev re-deploy scenarios.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The SELinux issue was caused by the main postgresql container not
being stopped before the upgrade container ran, leading to
conflicting MCS categories. With foreman.target properly stopping
services first, :Z (private label) is correct.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@jeremylenz

Copy link
Copy Markdown
Contributor Author

addressed comments and also updated the PR description to reflect the current state.

Comment thread src/roles/postgresql/tasks/upgrade.yaml Outdated
Move the systemd_target role before postgresql in deploy-dev so
foreman.target exists when the upgrade task needs to stop it.
This eliminates the need for failed_when: false.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Comment thread development/playbooks/deploy-dev/deploy-dev.yaml Outdated
Move the dnf module switch-to task from the deploy-dev pre-tasks
into the foreman_development role where it belongs alongside
other dev-specific module setup like nodejs.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@ehelms
ehelms merged commit 5bc3cc7 into theforeman:master Jul 17, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants