Skip to content

Improve backups/jobs reliability and observability #1776

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
Kajot-dev wants to merge 7 commits into
base: main
Choose a base branch
from
Draft

Improve backups/jobs reliability and observability #1776

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
1886ee7
Improve jobs observability
Kajot-dev Jun 23, 2026
222f154
Add E2E test and configurable backup
Kajot-dev Jul 17, 2026
cf2fe27
Fix nil pointer dereference
Kajot-dev Aug 25, 2026
c4a41a0
Fix keep job finalizers not removed on failed backups
Kajot-dev Aug 26, 2026
1bbe859
Attempt to fix e2e tests
Kajot-dev Aug 26, 2026
bef675a
Regenerate manifests
Kajot-dev Aug 26, 2026
9e49866
Correct test file naming
Kajot-dev Aug 31, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions build/crd/percona/generated/pgv2.percona.com_perconapgclusters.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -559,6 +559,10 @@ spec:
type: object
jobs:
properties:
activeDeadlineSeconds:
format: int64
minimum: 60
type: integer
affinity:
properties:
nodeAffinity:
Expand Down Expand Up @@ -1117,6 +1121,10 @@ spec:
type: string
type: object
type: object
startupDeadlineSeconds:
format: int64
minimum: 60
type: integer
tolerations:
items:
properties:
Expand Down
8 changes: 8 additions & 0 deletions config/crd/bases/pgv2.percona.com_perconapgclusters.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -973,6 +973,10 @@ spec:
type: object
jobs:
properties:
activeDeadlineSeconds:
format: int64
minimum: 60
type: integer
affinity:
properties:
nodeAffinity:
Expand Down Expand Up @@ -1531,6 +1535,10 @@ spec:
type: string
type: object
type: object
startupDeadlineSeconds:
format: int64
minimum: 60
type: integer
tolerations:
items:
properties:
Expand Down
8 changes: 8 additions & 0 deletions config/crd/bases/upstream.pgv2.percona.com_postgresclusters.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -468,6 +468,10 @@ spec:
type: object
jobs:
properties:
activeDeadlineSeconds:
format: int64
minimum: 60
type: integer
affinity:
properties:
nodeAffinity:
Expand Down Expand Up @@ -1026,6 +1030,10 @@ spec:
type: string
type: object
type: object
startupDeadlineSeconds:
format: int64
minimum: 60
type: integer
tolerations:
items:
properties:
Expand Down
16 changes: 16 additions & 0 deletions deploy/bundle.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1174,6 +1174,10 @@ spec:
type: object
jobs:
properties:
activeDeadlineSeconds:
format: int64
minimum: 60
type: integer
affinity:
properties:
nodeAffinity:
Expand Down Expand Up @@ -1732,6 +1736,10 @@ spec:
type: string
type: object
type: object
startupDeadlineSeconds:
format: int64
minimum: 60
type: integer
tolerations:
items:
properties:
Expand Down Expand Up @@ -21378,6 +21386,10 @@ spec:
type: object
jobs:
properties:
activeDeadlineSeconds:
format: int64
minimum: 60
type: integer
affinity:
properties:
nodeAffinity:
Expand Down Expand Up @@ -21936,6 +21948,10 @@ spec:
type: string
type: object
type: object
startupDeadlineSeconds:
format: int64
minimum: 60
type: integer
tolerations:
items:
properties:
Expand Down
16 changes: 16 additions & 0 deletions deploy/crd.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1174,6 +1174,10 @@ spec:
type: object
jobs:
properties:
activeDeadlineSeconds:
format: int64
minimum: 60
type: integer
affinity:
properties:
nodeAffinity:
Expand Down Expand Up @@ -1732,6 +1736,10 @@ spec:
type: string
type: object
type: object
startupDeadlineSeconds:
format: int64
minimum: 60
type: integer
tolerations:
items:
properties:
Expand Down Expand Up @@ -21378,6 +21386,10 @@ spec:
type: object
jobs:
properties:
activeDeadlineSeconds:
format: int64
minimum: 60
type: integer
affinity:
properties:
nodeAffinity:
Expand Down Expand Up @@ -21936,6 +21948,10 @@ spec:
type: string
type: object
type: object
startupDeadlineSeconds:
format: int64
minimum: 60
type: integer
tolerations:
items:
properties:
Expand Down
16 changes: 16 additions & 0 deletions deploy/cw-bundle.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1174,6 +1174,10 @@ spec:
type: object
jobs:
properties:
activeDeadlineSeconds:
format: int64
minimum: 60
type: integer
affinity:
properties:
nodeAffinity:
Expand Down Expand Up @@ -1732,6 +1736,10 @@ spec:
type: string
type: object
type: object
startupDeadlineSeconds:
format: int64
minimum: 60
type: integer
tolerations:
items:
properties:
Expand Down Expand Up @@ -21378,6 +21386,10 @@ spec:
type: object
jobs:
properties:
activeDeadlineSeconds:
format: int64
minimum: 60
type: integer
affinity:
properties:
nodeAffinity:
Expand Down Expand Up @@ -21936,6 +21948,10 @@ spec:
type: string
type: object
type: object
startupDeadlineSeconds:
format: int64
minimum: 60
type: integer
tolerations:
items:
properties:
Expand Down
11 changes: 11 additions & 0 deletions e2e-tests/tests/demand-backup/26-assert.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
apiVersion: kuttl.dev/v1beta1
kind: TestAssert
timeout: 30
commands:
- script: |-
set -o errexit
set -o xtrace

# Verify the startup deadline was set
kubectl get postgrescluster demand-backup -n "${NAMESPACE}" -o json | \
jq -e '.spec.backups.pgbackrest.jobs.startupDeadlineSeconds == 60' >/dev/null
21 changes: 21 additions & 0 deletions e2e-tests/tests/demand-backup/26-patch-cluster-short-startup-timeout.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
apiVersion: kuttl.dev/v1beta1
kind: TestStep
commands:
- script: |-
set -o errexit
set -o xtrace

# Patch the cluster to have a short startup deadline (60 seconds)
# This will cause backups with invalid specs to fail quickly
kubectl patch postgrescluster demand-backup -n "${NAMESPACE}" --type=merge -p '
{
"spec": {
"backups": {
"pgbackrest": {
"jobs": {
"startupDeadlineSeconds": 60
}
}
}
}
}'
17 changes: 17 additions & 0 deletions e2e-tests/tests/demand-backup/27-create-backup-invalid-resources.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
apiVersion: pgv2.percona.com/v2
kind: PerconaPGBackup
metadata:
name: demand-backup-invalid-resources
spec:
pgCluster: demand-backup
repoName: repo1
options:
- --type=full
containerOptions:
resources:
requests:
memory: "2Gi"
cpu: "2"
limits:
memory: "512Mi"
cpu: "500m"
22 changes: 22 additions & 0 deletions e2e-tests/tests/demand-backup/27-invalid-backup-assert.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
apiVersion: kuttl.dev/v1beta1
kind: TestAssert
timeout: 120
commands:
- script: |-
set -o errexit
set -o xtrace

# Wait for the backup to fail due to startup timeout
# The job won't start because limits < requests is invalid
kubectl wait --for=jsonpath='{.status.state}'=Failed --timeout=90s -n "${NAMESPACE}" pg-backup/demand-backup-invalid-resources

# Verify the error message mentions the timeout
ERROR_MSG=$(kubectl get pg-backup demand-backup-invalid-resources -n "${NAMESPACE}" -o jsonpath='{.status.error}')
echo "Backup error: $ERROR_MSG"

if [[ ! "$ERROR_MSG" =~ "timed out waiting for job to start" ]]; then
echo "ERROR: Expected timeout error but got: $ERROR_MSG"
exit 1
fi

echo "SUCCESS: Backup failed as expected due to startup timeout"
0 e2e-tests/tests/demand-backup/25-assert.yaml → ...ests/demand-backup/28-cleanup-assert.yaml
View file
Open in desktop
File renamed without changes.
0 ...kup/25-delete-cluster-with-finalizer.yaml → ...kup/28-delete-cluster-with-finalizer.yaml
View file
Open in desktop
File renamed without changes.
15 changes: 8 additions & 7 deletions e2e-tests/tests/scheduled-backup/20-assert.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
apiVersion: kuttl.dev/v1beta1
kind: TestAssert
timeout: 30
---
kind: ConfigMap
apiVersion: v1
metadata:
name: 19-pg-backup-objects
data:
data: "0"
commands:
- script: |-
set -o errexit
set -o xtrace

# Verify the startup deadline was set
kubectl get postgrescluster scheduled-backup -n "${NAMESPACE}" -o json | \
jq -e '.spec.backups.pgbackrest.jobs.startupDeadlineSeconds == 60' >/dev/null
21 changes: 21 additions & 0 deletions e2e-tests/tests/scheduled-backup/20-patch-cluster-short-startup-timeout.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
apiVersion: kuttl.dev/v1beta1
kind: TestStep
commands:
- script: |-
set -o errexit
set -o xtrace

# Patch the cluster to have a short startup deadline (60 seconds)
# This will cause scheduled backups that can't start to fail quickly
kubectl patch postgrescluster scheduled-backup -n "${NAMESPACE}" --type=merge -p '
{
"spec": {
"backups": {
"pgbackrest": {
"jobs": {
"startupDeadlineSeconds": 60
}
}
}
}
}'
38 changes: 38 additions & 0 deletions e2e-tests/tests/scheduled-backup/21-assert.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
apiVersion: kuttl.dev/v1beta1
kind: TestAssert
timeout: 120
commands:
- script: |-
set -o errexit
set -o xtrace

# Wait for the ScheduledBackupDegraded condition to be set to True
# The scheduled backup should fail because the ResourceQuota prevents job creation
kubectl wait --for=condition=ScheduledBackupDegraded=true --timeout=90s -n "${NAMESPACE}" postgrescluster/scheduled-backup || {
# If the condition wait times out, check the condition status manually
COND_STATUS=$(kubectl get postgrescluster scheduled-backup -n "${NAMESPACE}" -o json | \
jq -r '.status.conditions[] | select(.type == "ScheduledBackupDegraded") | .status' || echo "NotFound")

if [ "$COND_STATUS" != "True" ]; then
echo "ERROR: ScheduledBackupDegraded condition is not True. Current status: $COND_STATUS"
kubectl get postgrescluster scheduled-backup -n "${NAMESPACE}" -o json | jq '.status.conditions'
exit 1
fi
}

# Verify the condition reason and message
COND_REASON=$(kubectl get postgrescluster scheduled-backup -n "${NAMESPACE}" -o json | \
jq -r '.status.conditions[] | select(.type == "ScheduledBackupDegraded") | .reason')
COND_MSG=$(kubectl get postgrescluster scheduled-backup -n "${NAMESPACE}" -o json | \
jq -r '.status.conditions[] | select(.type == "ScheduledBackupDegraded") | .message')

echo "ScheduledBackupDegraded condition:"
echo " Reason: $COND_REASON"
echo " Message: $COND_MSG"

# The backup Job was blocked by the ResourceQuota and hit the startup timeout,
# so the ScheduledBackupDegraded condition is set with reason BackupFailed.
if [[ "$COND_REASON" != "BackupFailed" ]]; then
echo "ERROR: Expected reason BackupFailed, got: $COND_REASON"
exit 1
fi
Loading
Loading