K8SPG-786 use pg_stat_statements in upgrade test and additional test fixes

[valmiranogueira]

CHANGE DESCRIPTION

Description:

Use pg_stat_statements instead of pg_stat_monitor in the operator-upgrade-extensions test

Additionally:

• Fetch GKE versions exclusively from the STABLE release channel and keep one patch version per Kubernetes minor version.
• Filter cluster-wide operator error logs by the current KUTTL namespace.
  • Logs from other test namespaces are ignored.
  • Global errors without a namespace are still validated.
• Increase the builtin-extensions deployment timeout from 120 to 200 seconds to include MinIO startup.
• Replace finalizers instead of removing, in order to prevent conflicts with finalizers logic itself.

CHECKLIST

Jira

• Is the Jira ticket created and referenced properly?
• Does the Jira ticket have the proper statuses for documentation (Needs Doc) and QA (Needs QA)?
• Does the Jira ticket link to the proper milestone (Fix Version field)?

Tests

• Is an E2E test/test case added for the new feature/change?
• Are unit tests added where appropriate?

Config/Logging/Testability

• Are all needed new/changed options added to default YAML files?
• Are all needed new/changed options added to the Helm Chart?
• Did we add proper logging messages for operator actions?
• Did we ensure compatibility with the previous version or cluster upgrade process?
• Does the change support oldest and newest supported PG version?
• Does the change support oldest and newest supported Kubernetes version?

[Copilot]

🟡 Changes recommended

There are a couple of test-reliability issues to address (builtin-extensions step timeout still set to 120s at the TestStep level, and remove_all_finalizers uses a fragile JSONPatch replace instead of the repo’s more robust merge-patch pattern).

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Updates the E2E (KUTTL) suite for the PostgreSQL operator to improve upgrade/extension coverage and reduce test flakiness, including switching the operator-upgrade-extensions scenario to pg_stat_statements and tightening test infrastructure behaviors (GKE version selection, log filtering, cleanup finalizers, and timeouts).

Changes:

• Switch operator-upgrade-extensions E2E to use pg_stat_statements instead of pg_stat_monitor.
• Improve test infra robustness: select only STABLE GKE versions (one patch per minor) and filter operator error logs to the active KUTTL namespace.
• Adjust cleanup/finalizer handling and deployment timeouts to reduce upgrade/migration test flakes.

File summaries

File	Description
e2e-tests/tests/operator-upgrade-extensions/conf/operator-upgrade-extensions.yaml	Enables pg_stat_statements as a builtin extension for the upgrade test cluster.
e2e-tests/tests/operator-upgrade-extensions/03-enable-extensions-and-write-data.yaml	Creates/extensions workload using pg_stat_statements during the upgrade scenario.
e2e-tests/tests/operator-upgrade-extensions/01-create-cluster.yaml	Validates builtin extension fields weren’t pruned by the released CRD (now checks pg_stat_statements).
e2e-tests/tests/migration-from-crunchy-standby/99-cleanup.yaml	Uses replace to clear finalizers during cleanup.
e2e-tests/tests/migration-from-crunchy-pv/99-cleanup.yaml	Uses replace to clear finalizers during cleanup.
e2e-tests/tests/migration-from-crunchy-pv/03-migrate-pv.yaml	Uses replace to clear finalizers before deleting Crunchy source cluster.
e2e-tests/tests/migration-from-crunchy-backup-restore/99-cleanup.yaml	Uses replace to clear finalizers during cleanup.
e2e-tests/tests/builtin-extensions/00-deploy-operator.yaml	Moves/increases deployment timeout handling for the builtin-extensions test step.
e2e-tests/functions	Updates helper functions for finalizer removal, GKE version selection, and operator log error filtering.

Review details

• Files reviewed: 9/9 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[valmiranogueira]

this is incorrect, timeout for TestStep must be in script not in the class kuttl doc

[Copilot]

🔵 Needs a closer look

The new operator log namespace filter only handles JSON-style namespace fields and doesn’t reliably filter console-formatted logs as intended, which can cause unrelated namespace errors to fail tests.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

e2e-tests/functions:2645

• 1. Problem: The new namespace filter in check_logs_for_errors only recognizes JSON fields like "namespace":"...", but the comment above says operator logs are in console encoder format when LOG_STRUCTURED=false.

2. Why it matters: Errors from other KUTTL namespaces can still pass the filter in console-formatted logs, causing unrelated failures and defeating the intent of scoping operator errors to the current test namespace.
3. Fix: Extend the awk filter to also detect and filter common console field formats (e.g., namespace=<ns> / namespace="<ns>"), while still allowing lines with no namespace.

		| awk -v namespace="$NAMESPACE" '
			!match($0, /"namespace"[[:space:]]*:[[:space:]]*"[^"]+"/) ||
			index($0, "\"namespace\":\"" namespace "\"") ||
			index($0, "\"namespace\": \"" namespace "\"")
		' \

• Files reviewed: 9/9 changed files
• Comments generated: 0 new
• Review effort level: Lite

[JNKPercona]

Test Name	Result	Time
backup-enable-disable	passed	00:00:00
builtin-extensions	passed	00:00:00
custom-extensions	passed	00:00:00
cert-manager-tls	passed	00:00:00
cert-management-policy	passed	00:00:00
custom-envs	passed	00:00:00
custom-tls	passed	00:00:00
database-init-sql	passed	00:00:00
demand-backup	passed	00:00:00
demand-backup-offline-snapshot	passed	00:00:00
dynamic-configuration	passed	00:00:00
extra-volumes	passed	00:00:00
finalizers	passed	00:00:00
init-deploy	passed	00:00:00
huge-pages	passed	00:00:00
logical-replica	passed	00:00:00
major-upgrade-14-to-15	passed	00:00:00
major-upgrade-15-to-16	passed	00:00:00
major-upgrade-16-to-17	passed	00:00:00
major-upgrade-17-to-18	passed	00:00:00
major-upgrade-18-to-19	passed	00:00:00
major-upgrade-logical-replica	passed	00:00:00
ldap	passed	00:00:00
ldap-tls	passed	00:00:00
logcollection	passed	00:00:00
monitoring	passed	00:00:00
operator-upgrade-cert-manager	passed	00:00:00
operator-upgrade-extensions	passed	00:00:00
one-pod	passed	00:00:00
repo-host-autogrow	passed	00:00:00
operator-self-healing	passed	00:00:00
pgbouncer	passed	00:00:00
pg-tde	passed	00:00:00
pg-tde-wal-encrypt	passed	00:00:00
pitr	passed	00:00:00
scaling	passed	00:00:00
scheduled-backup	passed	00:00:00
self-healing	passed	00:00:00
sidecars	passed	00:00:00
standby-pgbackrest	passed	00:22:04
standby-streaming	passed	00:17:13
start-from-backup	passed	00:00:00
start-from-volumes	passed	00:00:00
tablespaces	passed	00:00:00
telemetry-transfer	passed	00:00:00
upgrade-consistency	passed	00:00:00
upgrade-minor	passed	00:00:00
users	passed	00:00:00
migration-from-crunchy-standby	passed	00:00:00
migration-from-crunchy-pv	passed	00:00:00
migration-from-crunchy-backup-restore	passed	00:00:00

Summary	Value
Tests Run	51/51
Job Duration	00:40:20
Total Test Time	00:39:17

commit: 0a0db5e
image: perconalab/percona-postgresql-operator:PR-1808-0a0db5e27