Skip to content

K8SPG-691: add repo host autoscaling#1689

Open
pooknull wants to merge 4 commits into
mainfrom
K8SPG-691
Open

K8SPG-691: add repo host autoscaling#1689
pooknull wants to merge 4 commits into
mainfrom
K8SPG-691

Conversation

@pooknull

@pooknull pooknull commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

DESCRIPTION

This PR adds autoscaling for pgbackrest PVCs. It follows the pgdata PVC autoscaling logic.

When the AutoGrowVolumes feature gate is enabled and spec.backups.pgbackrest.repos[].volume.volumeClaimSpec.resources.limits.storage is configured:

  • A script running in the pgbackrest-config container checks disk usage every five seconds.
  • When usage exceeds 75%, the script calculates a new desired size by increasing the current volume size by 50% and writes it to the repo-host pod annotation suggested-pgbackrest-<repo-name>-pvc-size
  • The annotation update triggers reconcile. The operator increases the PVCs resources.requests.storage to the suggested size
  • The operator saves the latest suggestion in status.pgbackrest.repos[].desiredRepoVolume

CHECKLIST

Jira

  • Is the Jira ticket created and referenced properly?
  • Does the Jira ticket have the proper statuses for documentation (Needs Doc) and QA (Needs QA)?
  • Does the Jira ticket link to the proper milestone (Fix Version field)?

Tests

  • Is an E2E test/test case added for the new feature/change?
  • Are unit tests added where appropriate?

Config/Logging/Testability

  • Are all needed new/changed options added to default YAML files?
  • Are all needed new/changed options added to the Helm Chart?
  • Did we add proper logging messages for operator actions?
  • Did we ensure compatibility with the previous version or cluster upgrade process?
  • Does the change support oldest and newest supported PG version?
  • Does the change support oldest and newest supported Kubernetes version?

@pooknull
pooknull marked this pull request as ready for review July 20, 2026 20:23

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds repo-host (pgBackRest) PVC autoscaling support to the operator, mirroring the existing pgdata autogrow behavior by having the repo-host Pod suggest a larger size via annotations and having the controller apply the resized PVC request while persisting the latest suggestion in status.

Changes:

  • Add repo-host monitoring script that periodically computes and PATCHes suggested-pgbackrest-<repo>-pvc-size annotations when volume usage is high (feature-gated).
  • Teach the PostgresCluster controller to observe suggested sizes from repo-host Pods, cap them by configured limits, apply updated PVC requests, and store the latest suggestion in status.pgbackrest.repos[].desiredRepoVolume.
  • Extend repo-host RBAC and add unit + KUTTL E2E coverage for the new behavior.

Reviewed changes

Copilot reviewed 27 out of 28 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
pkg/apis/upstream.pgv2.percona.com/v1beta1/pgbackrest_types.go Adds DesiredRepoVolume to repo status to persist observed suggestions.
internal/pgbackrest/reconcile.go Threads autogrow-enabled repo names into pgBackRest config container script generation.
internal/pgbackrest/reconcile_test.go Unit test ensuring autogrow monitoring is only injected for eligible repos and only on repo-host.
internal/pgbackrest/rbac.go Grants repo-host ServiceAccount patch on pods to allow annotation updates.
internal/pgbackrest/rbac_test.go Updates RBAC test expectations for added patch verb.
internal/pgbackrest/config.go Injects monitor_volume function + per-repo calls into reload loop and PATCHes pod annotations.
internal/pgbackrest/config_test.go Expands shellcheck coverage for reload script with/without autogrow repos.
internal/naming/annotations.go Adds helpers for suggested-size annotation naming and parsing.
internal/controller/postgrescluster/watches.go Triggers reconcile when suggested repo-host volume annotations change.
internal/controller/postgrescluster/watches_test.go Verifies the new watch behavior enqueues reconciliation on suggestion changes.
internal/controller/postgrescluster/pgbackrest.go Observes desired repo volumes from Pods/status, applies capped PVC request changes, persists desired size in status.
internal/controller/postgrescluster/pgbackrest_test.go Adds targeted unit coverage for setRepoVolumeSize and repo-host SA/mount behavior.
e2e-tests/tests/repo-host-autogrow/00-deploy-operator.yaml Enables AutoGrowVolumes in E2E deployment.
e2e-tests/tests/repo-host-autogrow/00-assert.yaml Asserts operator is running before test proceeds.
e2e-tests/tests/repo-host-autogrow/01-create-cluster.yaml Creates cluster with repo PVC request+limit configured to exercise autogrow.
e2e-tests/tests/repo-host-autogrow/01-assert.yaml Waits for cluster + repo-host StatefulSet readiness.
e2e-tests/tests/repo-host-autogrow/02-grow-repo-volume.yaml Fills repo volume to exceed threshold and trigger suggestion logic.
e2e-tests/tests/repo-host-autogrow/02-assert.yaml Asserts suggestion annotation appears and is above expected floor; cleans up fill file.
e2e-tests/tests/repo-host-autogrow/03-assert.yaml Asserts controller persists suggestion into PostgresCluster status and PVC request is updated.
e2e-tests/tests/repo-host-autogrow/99-remove-cluster-gracefully.yaml Cleans up cluster/operator and checks for panics.
e2e-tests/run-release.csv Adds the new E2E test to release runs.
e2e-tests/run-pr.csv Adds the new E2E test to PR runs.
deploy/cw-bundle.yaml CRD bundle regeneration for new status field.
deploy/crd.yaml CRD regeneration for new status field.
deploy/bundle.yaml CRD bundle regeneration for new status field.
config/crd/bases/upstream.pgv2.percona.com_postgresclusters.yaml Base CRD updated with new status field schema.
config/crd/bases/pgv2.percona.com_perconapgclusters.yaml Base CRD updated with new status field schema.
build/crd/percona/generated/pgv2.percona.com_perconapgclusters.yaml Generated CRD updated with new status field schema.

Comment on lines +3021 to +3024
current, err := resource.ParseQuantity(value)
if err != nil {
return nil, errors.Wrap(err, "parse quantity")
}
Comment on lines +580 to +585
curl --fail --silent --show-error --cacert "${CACERT}" \
--header "Authorization: Bearer ${TOKEN}" \
--header "Content-Type: application/merge-patch+json" \
--request PATCH \
--data "${patch}" \
"${APISERVER}/api/v1/namespaces/${NAMESPACE}/pods/${HOSTNAME}"
) (map[string]string, error) {
desired := make(map[string]string)
if !feature.Enabled(ctx, feature.AutoGrowVolumes) {
return desired, nil

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what happens if the flag is disabled after a resize? It looks like we return nil here and later in setRepoVolumeSize we read the size from the spec (not actual pvc size), won't that cause a downscale and possibly break reconciliation?

@JNKPercona

Copy link
Copy Markdown
Collaborator
Test Name Result Time
backup-enable-disable passed 00:12:21
builtin-extensions failure 00:00:00
cert-manager-tls passed 00:10:25
custom-envs passed 00:20:41
custom-tls passed 00:06:37
database-init-sql passed 00:02:33
demand-backup passed 00:26:12
demand-backup-offline-snapshot failure 00:13:48
dynamic-configuration passed 00:03:47
finalizers passed 00:04:06
init-deploy passed 00:03:16
huge-pages passed 00:03:21
major-upgrade-14-to-15 passed 00:11:09
major-upgrade-15-to-16 passed 00:09:39
major-upgrade-16-to-17 passed 00:11:25
major-upgrade-17-to-18 passed 00:11:24
ldap passed 00:05:52
ldap-tls passed 00:09:22
monitoring passed 00:08:49
monitoring-pmm3 passed 00:08:31
one-pod failure 00:07:45
repo-host-autogrow passed 00:02:05
operator-self-healing passed 00:11:14
pgbouncer-mtls passed 00:04:02
pitr passed 00:12:14
scaling passed 00:05:23
scheduled-backup passed 00:29:50
self-healing passed 00:09:38
sidecars passed 00:02:56
standby-pgbackrest passed 00:17:12
standby-streaming passed 00:13:06
start-from-backup passed 00:12:27
tablespaces passed 00:07:49
telemetry-transfer passed 00:04:42
upgrade-consistency passed 00:07:03
upgrade-minor passed 00:08:21
users passed 00:05:17
migration-from-crunchy-standby passed 00:19:11
migration-from-crunchy-pv passed 00:08:40
migration-from-crunchy-backup-restore passed 00:20:40
Summary Value
Tests Run 40/40
Job Duration 03:01:11
Total Test Time 06:33:11

commit: f566c0d
image: perconalab/percona-postgresql-operator:PR-1689-f566c0d7d

@oksana-grishchenko oksana-grishchenko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree with Mayank's concern about disabling feature gate after resizing, that is a possible real life scenario, please consider it.
Other than that LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants