Merge pull request #168 from bcgov/feature/oc-backup-db-to-unc

scripts: UNC-share pg_dump backup + artifactory rotation fix

Author: kmandryk

.github/workflows/deploy-instance.yml

@@ -520,12 +520,14 @@ jobs:
           INSTANCE: ${{ needs.metadata.outputs.instance-name }}
         run: |
           set -euo pipefail
-          # For prod, rotate the SHA-tagged rollback handles: keep the 10 most
-          # recent `<instance>-*` named tags per image, delete the rest. Then
-          # orphan cleanup reclaims underlying layers.
+          # For prod, rotate the SHA-tagged rollback handles: keep the 3 most
+          # recent `<instance>-<12hex>` named tags per image, delete the rest.
+          # The 12-`?` glob matches only the commit-SHA-suffixed tags, so
+          # `<instance>` and `<instance>-test` are never rotated. Orphan cleanup
+          # then reclaims underlying layers.
           # For test/dev, just orphan cleanup — no SHA tags accumulate.
           if [[ "${ENVIRONMENT}" == "prod" ]]; then
-            bash ./scripts/artifactory-cleanup.sh --delete --keep 10 --match "${INSTANCE}-*"
+            bash ./scripts/artifactory-cleanup.sh --delete --keep 3 --match "${INSTANCE}-????????????"
           else
             bash ./scripts/artifactory-cleanup.sh --delete
           fi

docs-md/openshift-deployment/BACKUP_TO_NETWORK_SHARE.md

@@ -0,0 +1,102 @@
+# Backing up the production database to a Windows network share
+
+This page describes `scripts/oc-backup-db-to-unc.sh`, a variant of the standard
+backup script that streams a `pg_dump` of an instance's PostgreSQL database
+straight to a Windows UNC share — without ever creating a file on the WSL host.
+
+Use it when the backup must not be persisted on the local machine (for example,
+when capturing production into a managed file-share location such as
+`\\widget\SDPRDocuments`).
+
+## When to use this vs. `oc-backup-db.sh`
+
+| Question | Use `oc-backup-db.sh` | Use `oc-backup-db-to-unc.sh` |
+|----------|-----------------------|-------------------------------|
+| Output destination | `./backups/<file>.pgc` on the WSL host | `\\server\share\<file>.pgc` on Windows |
+| Local file allowed? | Yes | No — nothing is written locally |
+| Platform | Any host with `oc` | WSL only (uses `powershell.exe`) |
+| Restore workflow | Pass local file to `oc-restore-db.sh --from` | File must first be copied somewhere local before restore |
+
+## Prerequisites
+
+- WSL with Windows interop enabled (`powershell.exe` reachable at
+  `/mnt/c/Windows/System32/WindowsPowerShell/v1.0/powershell.exe`).
+- The UNC share is reachable from the Windows side and the current Windows
+  user has write access to it.
+- `.oc-deploy/token` exists — created by `scripts/oc-setup-sa.sh` against the
+  target namespace (e.g. `fd34fb-prod`).
+- The service account has `pods/exec` permission (granted during setup).
+
+## Usage
+
+```bash
+# Production (instance bcgov-di in namespace fd34fb-prod)
+./scripts/oc-backup-db-to-unc.sh --instance bcgov-di --dest '\\widget\SDPRDocuments'
+
+# Defaulting the instance to the current git branch
+./scripts/oc-backup-db-to-unc.sh --dest '\\widget\SDPRDocuments'
+```
+
+Output file: `<dest>\<instance>-<YYYYMMDD-HHMMSS>.pgc`, e.g.
+`\\widget\SDPRDocuments\bcgov-di-20260515-082000.pgc`.
+
+The dump uses `pg_dump -Fc --clean --if-exists` (custom format with drop/recreate
+statements). Restore with `pg_restore` or the project's `oc-restore-db.sh`.
+
+## How "no local file" is achieved
+
+The standard `oc-backup-db.sh` writes the dump to `/tmp/...` inside the pod,
+then `oc cp`s it to the local filesystem. This script replaces the second
+step with a direct stream:
+
+1. `oc exec <pod> -c database -- pg_dump -Fc ...` writes the dump to stdout
+   (no TTY, so the binary custom-format stream is preserved end-to-end).
+2. Bash pipes that stdout into `powershell.exe -NoProfile -Command '...'`.
+3. The PowerShell side reads raw bytes via `[Console]::OpenStandardInput()`
+   and writes them to `[System.IO.File]::Create('<UNC path>')`. Using the
+   stream APIs (not `Set-Content`) keeps the byte sequence intact — no
+   CRLF translation, no implicit decoding.
+4. PowerShell then queries `Get-Item <dest>` for `.Length` and reports the
+   byte count back. The dump itself is never opened for reading.
+
+No file is created on the WSL filesystem or on a local Windows drive — the
+bytes traverse the pipe and land on the network share.
+
+## Before-stream safety check
+
+Before invoking `pg_dump`, the script writes a 2-byte probe file to the
+destination via PowerShell and deletes it again. If the probe fails (share
+unreachable, no write permission, path doesn't exist), the script aborts
+before any database work begins. This avoids running a multi-minute dump only
+to discover the destination is unwritable.
+
+## Restoring a dump that lives on a network share
+
+`scripts/oc-restore-db.sh` expects a local file path, so to restore you must
+first copy the `.pgc` file to a location the restore script can read.
+
+```bash
+# From WSL, copy back via powershell.exe (or any Windows file-copy tool)
+powershell.exe -NoProfile -Command "Copy-Item '\\widget\SDPRDocuments\bcgov-di-20260515-082000.pgc' 'C:\temp\restore.pgc'"
+./scripts/oc-restore-db.sh --instance bcgov-di-test --from /mnt/c/temp/restore.pgc
+```
+
+The restore script uses `pg_restore` and respects the `--clean --if-exists`
+flags baked into the dump, so existing data in the target instance is
+dropped and replaced.
+
+## What is *not* backed up
+
+The script captures only the PostgreSQL database. Azure Blob Storage content
+(uploaded documents, attachments, etc.) is **not** included; it must be
+backed up separately. This matches the behavior of `oc-backup-db.sh`.
+
+## Troubleshooting
+
+| Symptom | Likely cause |
+|---------|--------------|
+| `powershell.exe not found at /mnt/c/Windows/...` | Not running under WSL, or Windows interop is disabled. |
+| `Destination not writable` | UNC share is unreachable, or the Windows user lacks write permission. Try `powershell.exe -Command "Test-Path '<dest>'"` to verify. |
+| `Backup stream failed` | Either `oc exec` lost its connection, or the PowerShell side errored. A partial file may exist at the destination — delete it before retrying. |
+| `Token may have expired` | Re-run `./scripts/oc-setup-sa.sh --namespace <namespace>` to issue a fresh token. |
+| Backup file is much smaller than expected | The database may genuinely be small, but also check stderr from `pg_dump` (printed inline) for table-level errors. |

scripts/README.md

@@ -206,6 +206,33 @@ Notes:
 
 ---
 
+### oc-backup-db-to-unc.sh — Database Backup to Windows UNC share
+
+Streams a `pg_dump` of an instance's PostgreSQL database directly to a Windows
+UNC share (e.g. `\\widget\SDPRDocuments`) without writing any file on the WSL
+host. Use this when the backup must stay off the local machine.
+
+```bash
+# Stream prod backup to a UNC share
+./scripts/oc-backup-db-to-unc.sh --instance bcgov-di --dest '\\widget\SDPRDocuments'
+```
+
+| Option | Short | Required | Description |
+|--------|-------|----------|-------------|
+| `--dest` | `-d` | Yes | UNC directory to write into (e.g. `\\server\share`) |
+| `--instance` | `-i` | No | Instance to back up (default: from git branch) |
+| `--help` | `-h` | No | Show help |
+
+Output: `<UNC-dest>\<instance>-<timestamp>.pgc` (pg_dump custom format, `-Fc`)
+
+Notes:
+- WSL-only — requires Windows interop with `powershell.exe` reachable
+- Verifies the destination is writable before starting the dump
+- Pipes `oc exec ... pg_dump` stdout through `powershell.exe` to a `[System.IO.File]::Create` stream — no intermediate file on either Linux or Windows local disk
+- See [docs-md/openshift-deployment/BACKUP_TO_NETWORK_SHARE.md](../docs-md/openshift-deployment/BACKUP_TO_NETWORK_SHARE.md) for the longer write-up
+
+---
+
 ### oc-restore-db.sh — Database Restore
 
 Restores a PostgreSQL database from a local SQL dump file into any instance.

scripts/artifactory-cleanup.sh

@@ -146,9 +146,11 @@ fi
 if [[ -n "${KEEP_N}" ]]; then
   log_info "Rotation: keeping ${KEEP_N} most-recent '${MATCH_GLOB}' tags per image..."
 
+  # Match both `manifest.json` (legacy single-platform) and `list.manifest.json`
+  # (OCI image index — what buildx writes for multi-platform pushes).
   ROTATION_AQL=$(curl -sf -u "${AUTH}" -X POST "${BASE_URL}/api/search/aql" \
     -H "Content-Type: text/plain" \
-    -d "items.find({\"repo\":\"${ARTIFACTORY_REPO}\",\"type\":\"file\",\"name\":\"manifest.json\"}).include(\"repo\",\"path\",\"name\",\"created\")" 2>&1) || {
+    -d "items.find({\"repo\":\"${ARTIFACTORY_REPO}\",\"type\":\"file\",\"\$or\":[{\"name\":\"manifest.json\"},{\"name\":\"list.manifest.json\"}]}).include(\"repo\",\"path\",\"name\",\"created\")" 2>&1) || {
     log_error "Rotation AQL query failed."
     exit 1
   }
@@ -160,21 +162,31 @@ keep = int('${KEEP_N}')
 glob = '${MATCH_GLOB}'
 
 # Each result has path like 'backend-services/bcgov-di-abc123' (the tag dir)
-# plus name='manifest.json'. Extract (image, tag, created).
+# plus name like 'manifest.json' or 'list.manifest.json'. A tag may have both;
+# dedupe by (image, tag), keeping the newest created timestamp.
 by_image = {}
+seen = {}
 for r in data['results']:
     parts = r['path'].split('/')
     if len(parts) < 2:
         continue
-    image = parts[0]
-    tag = parts[1]
+    image, tag = parts[0], parts[1]
     if tag.startswith('sha256__') or tag.startswith('sha256:') or tag == '_uploads':
         continue
     if not fnmatch.fnmatch(tag, glob):
         continue
-    by_image.setdefault(image, []).append({'tag': tag, 'created': r.get('created', '')})
+    created = r.get('created', '')
+    key = (image, tag)
+    if key in seen:
+        if created > seen[key]:
+            seen[key] = created
+        continue
+    seen[key] = created
+    by_image.setdefault(image, []).append({'tag': tag, 'key': key})
 
 for image, tags in by_image.items():
+    for t in tags:
+        t['created'] = seen[t['key']]
     tags.sort(key=lambda t: t['created'], reverse=True)
     for t in tags[keep:]:
         print(f\"{image}\t{t['tag']}\")