Merge pull request #2705 from bcgov/DESENG-925-capture-patroni-cluster-in-helm

DESENG-925: Capture Patroni cluster in Helm charts

Author: NatSquared

.github/workflows/infrastructure-deploy.yml

@@ -21,7 +21,7 @@ on:
         default: "all"
         type: choice
         options:
-          - met-db
+          - engagement-db
           - notify-api
           - met-api
           - met-web
@@ -93,7 +93,7 @@ jobs:
         run: |
           # Define chart directories
           declare -A chart_dirs=(
-            [met-db]="openshift/db"
+            [engagement-db]="openshift/db"
             [notify-api]="openshift/notify"
             [met-api]="openshift/api"
             [met-web]="openshift/web"

.github/workflows/infrastructure-lint.yml

@@ -15,7 +15,7 @@ on:
         default: "all"
         type: choice
         options:
-          - met-db
+          - engagement-db
           - notify-api
           - met-api
           - met-web
@@ -78,7 +78,7 @@ jobs:
         run: |
           # Define chart directories
           declare -A chart_dirs=(
-            [met-db]="openshift/db"
+            [engagement-db]="openshift/db"
             [notify-api]="openshift/notify"
             [met-api]="openshift/api"
             [met-web]="openshift/web"

.gitignore

@@ -142,4 +142,7 @@ met-api/.DS_Store
 
 # VSCode config and Chrome extension user data
 .vscode/chrome
-met-web/.vscode/
+met-web/.vscode/
+
+# Helm dependency files
+openshift/**/charts/

CHANGELOG.MD

@@ -1,3 +1,14 @@
+## December 11, 2025
+
+- **Feature** Capture Patroni cluster in Helm charts [🎟️ DESENG-925](https://citz-gdx.atlassian.net/browse/DESENG-925)
+  - Updated Helm charts for `openshift/db` to include Patroni cluster configuration.
+  - Patroni cluster state is now managed by Helm via Github Actions CI/CD pipeline.
+  - Leverages https://github.com/bcgov/nr-patroni-chart/ as a subchart for Patroni deployment.
+  - Renamed DB to `engagement-patroni` to reflect ongoing branding changes.
+  - Updated backup and restore scripts to work with Patroni clusters.
+  - Updated backup/restore documentation in `openshift/README.md`.
+  - Confirmed app still runs correctly after DB reconfiguration.
+
 ## December 9, 2025
 
 - **Feature** Add CI pipeline for ETL workflow [🎟️ DESENG-880](https://citz-gdx.atlassian.net/browse/DESENG-880)
@@ -86,7 +97,6 @@
 ## August 22, 2025
 
 - **Chore** Resolve Dependabot vulnerability reports [🎟️ DESENG-845](https://citz-gdx.atlassian.net/browse/DESENG-845)
-
   - Update several packages to the latest version, resolving Dependabot PRs that are too old to be rebased.
 
 - **Bugfix** Finish upgrading DeploymentConfigs -> Deployments cross-app [🎟️ DESENG-847](https://citz-gdx.atlassian.net/browse/DESENG-847)
@@ -245,7 +255,6 @@
 - **Bugfix** Resolve conflicting versions for Dagster ETL pipeline [🎟️ DESENG-677](https://citz-gdx.atlassian.net/browse/DESENG-677)
   - Tweak dagster versions and python base image
 - **Feature** Redesign event widget [🎟️ DESENG-698](https://citz-gdx.atlassian.net/browse/DESENG-698)
-
   - Modified api to accomodate additional field required in event widget design
   - Modified forms and cards on admin side to show new field
   - Implemented front-end design
@@ -262,7 +271,6 @@
 ## April 22, 2025
 
 - **Feature** Redesign poll widget [🎟️ DESENG-697](https://citz-gdx.atlassian.net/browse/DESENG-697)
-
   - Refactor PollWidget.tsx implementing Figma design
     - Improved layout, similar to DocumentWidget
   - Load data more efficiently
@@ -272,15 +280,13 @@
 ## April 15, 2025
 
 - **Feature** Redesign document widget [🎟️ DESENG-696](https://citz-gdx.atlassian.net/browse/DESENG-696)
-
   - Significant refactor to DocumentWidget.tsx implementing Figma design
   - Use recursive rendering to support theoretically infinite subfolders in future
   - Many accessibility/QoL improvements
   - Improved layout - minimizes layout shift by forcing the list widget to be at least as tall as the copy it appears beside.
   - Improved skeleton while loading
 
 - **Chore** Update GitHub Actions workflows to use ubuntu-latest instead of ubuntu-20.04
-
   - This is to stay ahead of the deprecation of ubuntu-20.04 in GitHub Actions as of today's date
 
 ## _Interregnum break_: _October 2024 - March 2025_
@@ -290,7 +296,6 @@
 ## October 2, 2024
 
 - **Feature** New Who is Listening widget front end [🎟️ DESENG-695](https://citz-gdx.atlassian.net/browse/DESENG-695)
-
   - Implemented Figma design
   - Added widget_listening table to db for widget instance data
   - Added option to enter widget description text (below title, above contacts)
@@ -300,7 +305,6 @@
   - Updated Who is Listening widget form
 
 - **Feature** New Video Widget front end [🎟️ DESENG-692](https://citz-gdx.atlassian.net/browse/DESENG-692)
-
   - Changed icon colour from yellow to white to accomodate company branding requests
 
 - **Feature** New Map Widget front end [🎟️ DESENG-693](https://citz-gdx.atlassian.net/browse/DESENG-693)
@@ -315,7 +319,6 @@
 ## September 25, 2024
 
 - **Feature** New Video Widget front end [🎟️ DESENG-692](https://citz-gdx.atlassian.net/browse/DESENG-692)
-
   - Removed unneeded tables from db
   - Updated all other met_api and met_web logic to accomodate this
 
@@ -334,7 +337,6 @@
 ## September 18, 2024
 
 - **Feature** New Video Widget front end [🎟️ DESENG-692](https://citz-gdx.atlassian.net/browse/DESENG-692)
-
   - Implemented Figma design
   - Created custom layover bar for videos that shows video provider and has a custom logo
   - Transcripts will not be implemented at this time
@@ -392,7 +394,6 @@
 ## August 15, 2024
 
 - **Feature** Add engagement configuration summary [🎟️ DESENG-667](https://citz-gdx.atlassian.net/browse/DESENG-667)
-
   - Added a tabbed layout to the new engagement view page
   - Added a new "Configuration" tab to display the engagement's configuration details
     - The other tabs are blank for now, but will be filled in future tickets
@@ -432,7 +433,6 @@
 ## July 30, 2024
 
 - **Feature** New admin header [🎟️ DESENG-660](https://citz-gdx.atlassian.net/browse/DESENG-660)
-
   - Added a new API endpoint to fetch a list of a user's tenants, required for the new header
   - Redesigned the admin header to match the new design system
   - Added a new dropdown to the header to allow users to switch between tenants
@@ -450,7 +450,6 @@
 ## July 17, 2024
 
 - **Feature** Admin authoring experience - language selector [🎟️ DESENG-657](https://citz-gdx.atlassian.net/browse/DESENG-657)
-
   - Created loader for engagement creation wizard
   - Created language selector field in engagement creation wizard
 

met-api/src/met_api/services/authorization.py

@@ -29,7 +29,7 @@ def check_auth(**kwargs):
     if not user_from_db:
         abort(HTTPStatus.FORBIDDEN, 'User not found')
 
-    # Retrieve tenant specific user roles from met-db
+    # Retrieve tenant specific user roles from OIDC token info
     user_roles = current_app.config['JWT_ROLE_CALLBACK'](user_from_context.token_info)
 
     if not user_roles:

openshift/README.md

@@ -176,24 +176,37 @@ roleRef:
 
 ## Database Configuration
 
-Inntall an instance of patroni using helm chart:
+Create a highly available PostgreSQL database using Patroni:
 
-```
-helm repo add patroni-chart https://bcgov.github.io/nr-patroni-chart
-helm install -n <namespace> met-patroni patroni-chart/patroni
+```bash
+helm install engagement-db . --values values_<env>.yaml
 ```
 
-If HA is not necessary create a instance of a postgresql database:
+If the cluster is not starting, remove any existing ConfigMaps and PersistentVolumeClaims:
 
+```bash
+oc delete configmap -l app.kubernetes.io/instance=engagement-db
+oc delete pvc -l app.kubernetes.io/instance=engagement-db
 ```
-oc new-app --template=postgresql-persistent -p POSTGRESQL_DATABASE=app -p DATABASE_SERVICE_NAME=met-postgresql
+
+For technical reasons, we cannot load secrets from Vault during the initial database setup. Therefore, a Secret must be created in each namespace containing the initial database user passwords. These values should match those stored in Vault.
+
+```bash
+oc create secret generic engagement-patroni \
+--from-literal=superuser-username=postgres \
+--from-literal=superuser-password=<superuser-password> \
+--from-literal=replication-username=replicator \
+--from-literal=replication-password=<replication-password> \
+--from-literal=app-db-username=<app-username> \
+--from-literal=app-db-password=<app-password> \
+--from-literal=app-db-name=app \
+-n e903c2-<env>
 ```
 
 ### Setup
 
 1. Users Setup script is located at /tools/postgres/init/00_postgresql-user-setup.sql
 1. Initial database setup script is located at /tools/postgres/init/01_postgresql-schema-setup.sql
-1. Openshift secret yaml is located at ./database-users.secret.yml
 
 ## IP Whitelist Management
 
@@ -266,69 +279,72 @@ This should be stored at:
 
 Backups are generated daily by the "backup" deployment in each namespace and are saved as a gzipped SQL script containing the database structure + data.
 
-Backups are verified nightly at 4 AM by creating a temporary database instance, restoring the backup into it, and checking for tables in the restored database. This is configured in the `backup.conf` file, mounted from the `met-db-backup-config` ConfigMap.
+Backups are verified nightly at 4 AM by creating a temporary database instance, restoring the backup into it, and checking for tables in the restored database. This is configured in the `backup.conf` file, mounted from the `engagement-db-backup` ConfigMap.
 
 Backups are also uploaded to S3-compatible storage for offsite retention.
 
-## Restoring from S3
+### Restoring from S3
 
 **Note**: The backup container does not support restoring directly from S3. If a backup has already been removed from the local FS, you must download it from S3 to the local `/backups/` directory first, then restore using the standard process.
 
 To restore a backup stored in S3:
 
 ```bash
 # Connect to backup container
-oc rsh deploy/met-db-backup
+oc rsh deploy/engagement-db-backup
 
 # Source S3 credentials
 source /vault/secrets/s3
 
+# Configure mc (MinIO Client) for S3 access
+mc alias set dell_s3 $S3_ENDPOINT $S3_USER $S3_PASSWORD
+
 # List backups in S3 bucket
-mc ls minio_s3/engagement-dev-backup/
+mc ls dell_s3/engagement-dev-backup/
 
 # Download the desired backup file
-mc cp minio_s3/engagement-dev-backup/met-patroni-app_YYYY-MM-DD_HH-MM-SS.sql.gz /backups/
+mc cp dell_s3/engagement-dev-backup/engagement-patroni-app_YYYY-MM-DD_HH-MM-SS.sql.gz /backups/
 
 # Now proceed with normal restore process (see scenarios below)
 ```
 
-## Scenarios
+### Scenarios
 
-### 1. Normal Production Restore (Roles Exist)
+#### 1. Normal Production Restore (Roles Exist)
 
 **Use case**: Restoring to existing Patroni cluster where roles are already defined.
 
 ```bash
 # Connect to backup container
-oc rsh deploy/met-db-backup
+oc rsh deploy/engagement-db-backup
 
 # Add credentials to environment
-source /vault/secrets/met-patroni
+source /vault/secrets/engagement-patroni
 
 # List available backups
 ./backup.sh -l # press Enter if prompted for password
 
 # Restore with -I flag to ignore duplicate role errors
-./backup.sh -I -r postgres=met-patroni:5432/app -f /backups/<period>/<YYYY-MM-DD>/met-patroni-app_<YYYY-MM-DD_HH-MM-SS>.sql.gz
+./backup.sh -I -r postgres=engagement-patroni:5432/app -f /backups/<period>/<YYYY-MM-DD>/engagement-patroni-app_<YYYY-MM-DD_HH-MM-SS>.sql.gz
 ```
 
 **Why `-I`?**: The backup contains `CREATE ROLE` statements at the end. Since roles already exist in production, these will fail but can be safely ignored.
 
 ---
 
-### 2. Emergency Restore (Fresh Database)
+#### 2. Emergency Restore (Fresh Database)
 
 **Use case**: Restoring to a completely new PostgreSQL instance that has no existing roles.
 
 ```bash
 # Connect to backup container
-oc rsh deploy/met-db-backup
+oc rsh deploy/engagement-db-backup
 
 # Source credentials
-source /vault/secrets/met-patroni
+source /vault/secrets/engagement-patroni
 
 # Use the emergency restore script
-./emergency-restore.sh /backups/daily/YYYY-MM-DD/met-patroni-app_YYYY-MM-DD_HH-MM-SS.sql.gz app
+./emergency-restore.sh /backups/daily/YYYY-MM-DD/engagement-patroni-app_YYYY-MM-DD_HH-MM-SS.sql.gz app
 ```
 
 **What it does**:
@@ -342,7 +358,7 @@ Note: the users are created without passwords; you will need to set passwords ma
 
 ```bash
 # Connect as superuser
-oc rsh pod/met-patroni-0
+oc rsh pod/engagement-patroni-0
 psql -U postgres -d app
 
 # Set passwords for each role

openshift/db/Chart.lock

@@ -0,0 +1,6 @@
+dependencies:
+- name: patroni
+  repository: https://bcgov.github.io/nr-patroni-chart
+  version: 0.0.4
+digest: sha256:d7bc2a92b8b07c58f069af160f02486c5f1b6f8d80b00e016473501f84aee502
+generated: "2025-12-10T13:43:11.389446729-08:00"

openshift/db/Chart.yaml

@@ -11,7 +11,9 @@ description: |
 type: application
 version: 0.1.0
 appVersion: "1.0"
-# dependencies:
-#   - name: patroni
-#     version: 0.0.1
-#     repository: https://bcgov.github.io/nr-patroni-chart
+dependencies:
+  - name: patroni
+    alias: postgres
+    version: 0.0.4
+    repository: https://bcgov.github.io/nr-patroni-chart
+    condition: postgres.enabled

openshift/db/files/backup.conf

@@ -1,4 +1,4 @@
 # Daily backup at 1 AM
 0 1 * * * default ./backup.sh -s
 # Daily verification at 4 AM using verify connection spec with -I to ignore role errors
-0 4 * * * default ./backup.sh -I -s -v postgres=verify:5432/app -f met-patroni-app
+0 4 * * * default ./backup.sh -I -s -v postgres=verify:5432/app -f engagement-patroni-app

openshift/db/files/emergency-restore.sh

@@ -11,16 +11,16 @@ set -e
 
 if [[ $# -ne 2 ]]; then
     echo "Usage: $0 <backup-file.sql.gz> <database-name>"
-    echo "Example: $0 /backups/daily/2025-11-25/met-patroni-app_2025-11-25_10-34-33.sql.gz app"
+    echo "Example: $0 /backups/daily/2025-11-25/engagement-patroni-app_2025-11-25_10-34-33.sql.gz app"
     exit 1
 fi
 
 BACKUP_FILE="$1"
 DATABASE="$2"
 
-source /vault/secrets/met-patroni
+source /vault/secrets/engagement-patroni
 export PGPASSWORD="$MET_PATRONI_PASSWORD"
-export DB_HOST="${DATABASE_SERVICE_NAME:-met-patroni}"
+export DB_HOST="${DATABASE_SERVICE_NAME:-engagement-patroni}"
 
 # Check the file exists
 if [[ ! -f "$BACKUP_FILE" ]]; then