AI-1823 PVC Size Changes & Logging Changes

[dbarkowsky]

Summary

One thing I noticed during this. When the backup PVC is full, the PVC for database data begins to fill with the write-ahead files. As soon as I expanded the backup PVC, Crunchy relieved the data PVC of all those files. As a result, the data PVC probably didn't need to be increased, but at least this gives room for expected growth.

Changes

• Changed many defaults for PVCs at 10Gi down to 2Gi.
  • But also prod overrides for larger PVCs needed.
• Also increased the size for the app-pg PVC to permit the DB restore.
• Lowered restore timeline from 30 to 14 days.
• Added drop conditions to the promtail configs. The goal is to drop repetitive logging that we're not interested in (Loki-wise). That means stuff like /health or /metrics.
• Upped db pod replicas to 2. This seems like a reasonable requirement if there's going to be high-traffic use in production.

Testing

Checklist

By submitting this pull request, I acknowledge that I have attempted to meet the following:

• a self-review of my code
• commented code particularly in hard-to-understand areas
• corresponding changes to the documentation where required
• changes tested to the best of my ability
• no new errors or non-functional code

[alex-struk]

While cleaning up an old branch (#169) that also touched temporal/postgrescluster.yml, I dropped its infra hunk in favour of this PR — but one piece isn't covered here yet and I think it belongs in this stack: the temporal-pg repo1 (pgBackRest backup) PVC.

Checked live fd34fb-prod today:

	Applied spec	Live PVC	Used
pgdata	5Gi	5Gi	647M (13%)
repo1 (backups)	10Gi	22Gi	12G (51%)

The repo1 PVC was already expanded in-place to 22Gi (~125d ago), but nothing patches repo1 for prod, so the applied spec is still the base 10Gi. That mismatch is live on the cluster:

$ oc -n fd34fb-prod get postgrescluster bcgov-di-temporal-pg \
    -o jsonpath='{range .status.conditions[*]}{.type}={.status} :: {.reason}{"\n"}{end}'
...
PersistentVolumeResizing=False :: Invalid

Same shrink-conflict signature as the pgdata one this PR already documents — desired(10Gi) < actual(22Gi), operator won't shrink. And it's a real requirement, not just leftover: repo1 is using 12G, above the 10Gi base, even after block-incremental + 14-day retention.

Suggested fix (mirrors the pgdata patch already in prod-resources): add a prod-only repo1 patch so base stays 10Gi for test/ephemeral —

  - target:
      kind: PostgresCluster
      name: temporal-pg
    patch: |
      - op: replace
        path: /spec/backups/pgbackrest/repos/0/volume/volumeClaimSpec/resources/requests/storage
        value: 22Gi

(#169's original approach edited the base to 22Gi, which would also push fd34fb-test + ephemeral instances up — hence flagging it here instead. Not urgent; the volume is healthy at 51%.)

Unrelated note from the same look: the recent temporal-pg-repo1-full job Error entries are pgBackRest lock contention (unable to acquire lock … is another pgBackRest process running?), i.e. a double-scheduled full, not a space problem.