fix(e2e): fix runtime tests for operator deployments and CI reporting (#5027)

* refactor(e2e): move about:blank cleanup from try/finally to test.afterEach

Move the WebSocket teardown workaround (page.goto('about:blank')) from
per-test try/finally blocks into test.afterEach hooks. This is cleaner:
the cleanup is declared once per describe block and applies to all tests
that use the page fixture, without wrapping test bodies.

Assisted-by: OpenCode

* fix(e2e): fix external DB tests for operator deployments

For operator deployments, patching POSTGRES_* env vars directly onto the
deployment is reverted by operator reconciliation. The external DB tests
(Azure DB, RDS) configure postgres-cred secret with connection details,
but those values never reached the RHDH container as env vars.

Fix:
- Add postgres-cred to extraEnvs.secrets in the Backstage CR so the
  operator injects all its keys as env vars automatically
- Skip direct deployment env var patching in prepareForExternalDatabase
  for operator installs (same pattern as schema-mode setup)

Assisted-by: OpenCode

* fix(ci): write pessimistic test status before Playwright execution

When Prow kills a job (2h timeout) or Playwright hangs, the
mark_test_result call at the end of testing::run_tests never executes.
This leaves STATUS_TEST_FAILED.txt and STATUS_NUMBER_OF_TEST_FAILED.txt
with fewer entries than STATUS_DEPLOYMENT_NAMESPACE.txt, creating
misaligned arrays that break downstream reporting (Slack notifications).

Fix: write a pessimistic default (failed=true, count=N/A) immediately
after registering the test run. The real result overwrites it after
Playwright completes. If the job is killed mid-test, the STATUS files
still have entries for all registered test runs.

To support this, save_status_test_failed and save_status_number_of_test_failed
now regenerate the file from the in-memory array instead of appending,
making them safe to call multiple times for the same deployment ID.

Assisted-by: OpenCode

* docs(ci): document same-process requirement for _regenerate_status_file

The regenerate-from-array pattern requires all callers to run in the
same shell process because bash associative arrays are not inherited
by child processes. Add a comment documenting this constraint so it
travels with the code.

Also note the bash >= 4.3 requirement for local -n (nameref).

Assisted-by: OpenCode

* refactor(ci): unify unknown failure count sentinel value

Replace the two different sentinel strings ('N/A' and 'some') with a
single UNKNOWN_FAILURE_COUNT constant defined in test-run-tracker.sh.
Both meant 'failed but count unknown' — the downstream Slack consumer
now only has to handle one sentinel format.

Assisted-by: OpenCode

* refactor(e2e): extract BACKSTAGE_CR_EXTRA_ENV_SECRETS constant

The secrets list in the Backstage CR merge-patch (postgres-config.ts)
was hardcoded separately from generateBackstageCR (runtime-config.ts).
If a secret were added to one but not the other, the patch would
silently strip it mid-test.

Extract the base list into a shared constant so the two stay in sync.

Assisted-by: OpenCode

* refactor(e2e): add mergePatchCustomObject helper to KubeClient

Centralise the JSON merge-patch Content-Type header in a KubeClient
method, matching the existing jsonPatchDeployment and updateSecret
patterns. Simplifies the call site in addPostgresCredToBackstageCR
by removing the trailing positional undefined args.

Assisted-by: OpenCode

* docs(e2e): note intentional afterEach page fixture overhead

The suite-level afterEach requests the page fixture for every test,
including non-UI ones. This is an intentional tradeoff — the minor
overhead of an extra browser context is acceptable vs per-test
conditional logic.

Assisted-by: OpenCode

* docs(e2e): note local dist path for homepage plugin package

The runtime deployment uses a local ./dynamic-plugins/dist/ path for
the homepage plugin. Add a comment referencing #4909 so the OCI
migration sweep can find this spot.

Assisted-by: OpenCode

* refactor(e2e): remove redundant try/catch in config-map test

The try/catch only logged and rethrew — it existed to pair with the
finally block that was removed when about:blank cleanup moved to
afterEach. Playwright already reports errors with full stacks, so
the wrapper adds no value and costs an indent level.

Assisted-by: OpenCode

* docs(e2e): explain explicit form in getReleaseName

The verbose !== undefined && !== '' check is used instead of || because
oxlint's strict-boolean-expressions and prefer-nullish-coalescing rules
(pedantic category, set to error) reject || on string operands.

Assisted-by: OpenCode

Author: zdrapela

.ci/pipelines/lib/test-run-tracker.sh

@@ -9,6 +9,11 @@ readonly RHDH_TEST_RUN_TRACKER_LIB_SOURCED=1
 # shellcheck source=.ci/pipelines/reporting.sh
 source "$(dirname "${BASH_SOURCE[0]}")/../reporting.sh"
 
+# Sentinel value for STATUS_NUMBER_OF_TEST_FAILED when the real count
+# is unknown (deploy failed, Prow timeout, JUnit file missing, or
+# Playwright crashed without producing failure counts).
+UNKNOWN_FAILURE_COUNT="N/A"
+
 # Internal state
 _TEST_RUN_COUNTER=0
 
@@ -36,7 +41,7 @@ test_run_tracker::mark_deploy_failed() {
   test_run_tracker::register "$label"
   save_status_failed_to_deploy "${_TEST_RUN_COUNTER}" true
   save_status_test_failed "${_TEST_RUN_COUNTER}" true
-  save_status_number_of_test_failed "${_TEST_RUN_COUNTER}" "N/A"
+  save_status_number_of_test_failed "${_TEST_RUN_COUNTER}" "${UNKNOWN_FAILURE_COUNT}"
   save_overall_result 1
 }
 

.ci/pipelines/lib/testing.sh

@@ -52,6 +52,12 @@ testing::run_tests() {
   test_run_tracker::register "$artifacts_subdir"
   test_run_tracker::mark_deploy_success
 
+  # Pessimistic default: assume tests failed until Playwright proves otherwise.
+  # If the job is killed (Prow timeout) or Playwright hangs, the STATUS files
+  # still have entries for all registered test runs — preventing misaligned
+  # arrays that break downstream reporting (Slack notifications).
+  test_run_tracker::mark_test_result "false" "${UNKNOWN_FAILURE_COUNT}"
+
   BASE_URL="${url}"
   export BASE_URL
   log::info "BASE_URL: ${BASE_URL}"
@@ -183,15 +189,15 @@ testing::run_tests() {
     failed_tests=$((_junit_failures + _junit_errors))
     if [[ "${failed_tests}" -eq 0 ]]; then
       # Playwright exited non-zero but JUnit reports 0 failures and 0 errors —
-      # the process likely crashed or timed out globally. Report "some" so the
-      # Slack alert doesn't misleadingly say "0 tests failed".
-      failed_tests="some"
+      # the process likely crashed or timed out globally. Use the sentinel so
+      # the Slack alert doesn't misleadingly say "0 tests failed".
+      failed_tests="${UNKNOWN_FAILURE_COUNT}"
     fi
     echo "Number of failed tests: ${failed_tests}"
   else
     echo "JUnit results file not found: ${e2e_tests_dir}/${JUNIT_RESULTS}"
-    failed_tests="some"
-    echo "Number of failed tests unknown, saving as $failed_tests."
+    failed_tests="${UNKNOWN_FAILURE_COUNT}"
+    echo "Number of failed tests unknown, saving as ${failed_tests}."
   fi
   test_run_tracker::mark_test_result "$test_passed" "${failed_tests}"
   return "$test_result"

.ci/pipelines/reporting.sh

@@ -10,10 +10,11 @@ readonly REPORTING_LIB_SOURCED=1
 source "$(dirname "${BASH_SOURCE[0]}")"/lib/log.sh
 
 # Variables for reporting
-export STATUS_DEPLOYMENT_NAMESPACE # Array that holds the namespaces of deployments.
-export STATUS_FAILED_TO_DEPLOY     # Array that indicates if deployment failed. false = success, true = failure
-export STATUS_TEST_FAILED          # Array that indicates if test run failed. false = success, true = failure
-export OVERALL_RESULT              # Overall result of the test run. 0 = success, 1 = failure
+export STATUS_DEPLOYMENT_NAMESPACE  # Array that holds the namespaces of deployments.
+export STATUS_FAILED_TO_DEPLOY      # Array that indicates if deployment failed. false = success, true = failure
+export STATUS_TEST_FAILED           # Array that indicates if test run failed. false = success, true = failure
+export STATUS_NUMBER_OF_TEST_FAILED # Array that holds the number of test failures per deployment.
+export OVERALL_RESULT               # Overall result of the test run. 0 = success, 1 = failure
 
 mkdir -p "$ARTIFACT_DIR/reporting"
 
@@ -22,35 +23,54 @@ save_status_deployment_namespace() {
   local current_namespace=$2
   log::debug "Saving STATUS_DEPLOYMENT_NAMESPACE[\"${current_deployment}\"]=${current_namespace}"
   STATUS_DEPLOYMENT_NAMESPACE["${current_deployment}"]="${current_namespace}"
-  printf "%s\n" "${STATUS_DEPLOYMENT_NAMESPACE["${current_deployment}"]}" >> "$SHARED_DIR/STATUS_DEPLOYMENT_NAMESPACE.txt"
-  cp "$SHARED_DIR/STATUS_DEPLOYMENT_NAMESPACE.txt" "$ARTIFACT_DIR/reporting/STATUS_DEPLOYMENT_NAMESPACE.txt"
+  _regenerate_status_file "STATUS_DEPLOYMENT_NAMESPACE"
 }
 
 save_status_failed_to_deploy() {
   local current_deployment=$1
   local status=$2
   log::debug "Saving STATUS_FAILED_TO_DEPLOY[\"${current_deployment}\"]=${status}"
   STATUS_FAILED_TO_DEPLOY["${current_deployment}"]="${status}"
-  printf "%s\n" "${STATUS_FAILED_TO_DEPLOY["${current_deployment}"]}" >> "$SHARED_DIR/STATUS_FAILED_TO_DEPLOY.txt"
-  cp "$SHARED_DIR/STATUS_FAILED_TO_DEPLOY.txt" "$ARTIFACT_DIR/reporting/STATUS_FAILED_TO_DEPLOY.txt"
+  _regenerate_status_file "STATUS_FAILED_TO_DEPLOY"
 }
 
 save_status_test_failed() {
   local current_deployment=$1
   local status=$2
   log::debug "Saving STATUS_TEST_FAILED[\"${current_deployment}\"]=${status}"
   STATUS_TEST_FAILED["${current_deployment}"]="${status}"
-  printf "%s\n" "${STATUS_TEST_FAILED["${current_deployment}"]}" >> "$SHARED_DIR/STATUS_TEST_FAILED.txt"
-  cp "$SHARED_DIR/STATUS_TEST_FAILED.txt" "$ARTIFACT_DIR/reporting/STATUS_TEST_FAILED.txt"
+  _regenerate_status_file "STATUS_TEST_FAILED"
 }
 
 save_status_number_of_test_failed() {
   local current_deployment=$1
   local number=$2
   log::debug "Saving STATUS_NUMBER_OF_TEST_FAILED[\"${current_deployment}\"]=${number}"
   STATUS_NUMBER_OF_TEST_FAILED["${current_deployment}"]="${number}"
-  printf "%s\n" "${STATUS_NUMBER_OF_TEST_FAILED["${current_deployment}"]}" >> "$SHARED_DIR/STATUS_NUMBER_OF_TEST_FAILED.txt"
-  cp "$SHARED_DIR/STATUS_NUMBER_OF_TEST_FAILED.txt" "$ARTIFACT_DIR/reporting/STATUS_NUMBER_OF_TEST_FAILED.txt"
+  _regenerate_status_file "STATUS_NUMBER_OF_TEST_FAILED"
+}
+
+# Regenerate a STATUS file from its in-memory associative array.
+# Writes the file from scratch each time so that the same deployment ID
+# can be safely updated multiple times (e.g. pessimistic default written
+# before Playwright runs, then overwritten with the real result).
+#
+# IMPORTANT: All callers must run in the same shell process.
+# Associative arrays are not inherited by child processes (export -f
+# only exports function definitions, not array contents). If this
+# function runs in a subshell, it will see an empty array and truncate
+# the file. This is fine today — all test_run_tracker calls are
+# sequential in the main shell.
+_regenerate_status_file() {
+  local var_name=$1
+  local -n _arr="${var_name}"
+  local file="$SHARED_DIR/${var_name}.txt"
+  : > "$file"
+  local key
+  for key in $(printf '%s\n' "${!_arr[@]}" | sort -n); do
+    printf '%s\n' "${_arr[$key]}" >> "$file"
+  done
+  cp "$file" "$ARTIFACT_DIR/reporting/${var_name}.txt"
 }
 
 save_overall_result() {

e2e-tests/playwright/e2e/configuration-test/config-map.spec.ts

@@ -26,6 +26,15 @@ test.describe("Change app-config at e2e test runtime", () => {
     await ensureRuntimeDeployed();
   });
 
+  // RHDH's frontend opens an SSE (EventSource) connection for live
+  // updates. With tracing enabled, Playwright's fixture teardown hangs
+  // waiting for network idle, which never resolves while SSE stays open
+  // (microsoft/playwright#41513, fixed in v1.62). Navigating away drops
+  // the connection so teardown completes immediately.
+  test.afterEach(async ({ page }) => {
+    await page.goto("about:blank").catch(() => {});
+  });
+
   test("Verify title change after ConfigMap modification", async ({ page }) => {
     test.setTimeout(300000);
 
@@ -34,38 +43,29 @@ test.describe("Change app-config at e2e test runtime", () => {
 
     const kubeUtils = new KubeClient();
     const dynamicTitle = generateDynamicTitle();
-    try {
-      console.log("Updating app-config ConfigMap with new title.");
-      await kubeUtils.patchAppConfig(namespace, (appConfig: Record<string, unknown>) => {
-        if (!isRecord(appConfig.app)) {
-          throw new Error("Invalid app-config structure: expected 'app' section not found.");
-        }
-        console.log(`Current title: ${String(appConfig.app.title)}`);
-        appConfig.app.title = dynamicTitle;
-        console.log(`New title: ${dynamicTitle}`);
-      });
 
-      console.log(`Restarting deployment '${deploymentName}' to apply ConfigMap changes.`);
-      await kubeUtils.restartDeployment(deploymentName, namespace);
+    console.log("Updating app-config ConfigMap with new title.");
+    await kubeUtils.patchAppConfig(namespace, (appConfig: Record<string, unknown>) => {
+      if (!isRecord(appConfig.app)) {
+        throw new Error("Invalid app-config structure: expected 'app' section not found.");
+      }
+      console.log(`Current title: ${String(appConfig.app.title)}`);
+      appConfig.app.title = dynamicTitle;
+      console.log(`New title: ${dynamicTitle}`);
+    });
+
+    console.log(`Restarting deployment '${deploymentName}' to apply ConfigMap changes.`);
+    await kubeUtils.restartDeployment(deploymentName, namespace);
 
-      const common = new Common(page);
-      await page.context().clearCookies();
-      await page.context().clearPermissions();
-      await page.reload({ waitUntil: "domcontentloaded" });
-      await common.loginAsGuest();
-      await new UIhelper(page).openSidebar("Home");
-      console.log("Verifying new title in the UI... ");
-      expect(await page.title()).toContain(dynamicTitle);
-      console.log("Title successfully verified in the UI.");
-    } catch (error) {
-      console.log(`Test failed during ConfigMap update or deployment restart:`, error);
-      throw error;
-    } finally {
-      // Navigate away from RHDH to close WebSocket connections before
-      // Playwright tears down the page — prevents a long hang during
-      // context/trace cleanup.
-      await page.goto("about:blank").catch(() => {});
-    }
+    const common = new Common(page);
+    await page.context().clearCookies();
+    await page.context().clearPermissions();
+    await page.reload({ waitUntil: "domcontentloaded" });
+    await common.loginAsGuest();
+    await new UIhelper(page).openSidebar("Home");
+    console.log("Verifying new title in the UI... ");
+    expect(await page.title()).toContain(dynamicTitle);
+    console.log("Title successfully verified in the UI.");
   });
 });
 

e2e-tests/playwright/e2e/external-database/verify-tls-config-with-external-azure-db.spec.ts

@@ -89,6 +89,12 @@ test.describe("Verify TLS configuration with Azure Database for PostgreSQL healt
         });
       });
 
+      // Drop RHDH SSE connection so Playwright trace teardown doesn't hang
+      // (microsoft/playwright#41513, fixed in v1.62).
+      test.afterEach(async ({ page }) => {
+        await page.goto("about:blank").catch(() => {});
+      });
+
       test("Configure and restart deployment", async ({}, testInfo) => {
         if (!config.host) {
           testInfo.skip(true, `AZURE_DB_*_HOST not set for ${config.name}`);
@@ -105,14 +111,10 @@ test.describe("Verify TLS configuration with Azure Database for PostgreSQL healt
       });
 
       test("Verify successful DB connection", async ({ page }) => {
-        try {
-          const uiHelper = new UIhelper(page);
-          const common = new Common(page);
-          await common.loginAsGuest();
-          await uiHelper.verifyHeading("Welcome back!");
-        } finally {
-          await page.goto("about:blank").catch(() => {});
-        }
+        const uiHelper = new UIhelper(page);
+        const common = new Common(page);
+        await common.loginAsGuest();
+        await uiHelper.verifyHeading("Welcome back!");
       });
     });
   }

e2e-tests/playwright/e2e/external-database/verify-tls-config-with-external-rds.spec.ts

@@ -89,6 +89,12 @@ test.describe("Verify TLS configuration with RDS PostgreSQL health check", () =>
         });
       });
 
+      // Drop RHDH SSE connection so Playwright trace teardown doesn't hang
+      // (microsoft/playwright#41513, fixed in v1.62).
+      test.afterEach(async ({ page }) => {
+        await page.goto("about:blank").catch(() => {});
+      });
+
       test("Configure and restart deployment", async ({}, testInfo) => {
         if (!config.host) {
           testInfo.skip(true, `RDS_*_HOST not set for ${config.name}`);
@@ -105,14 +111,10 @@ test.describe("Verify TLS configuration with RDS PostgreSQL health check", () =>
       });
 
       test("Verify successful DB connection", async ({ page }) => {
-        try {
-          const uiHelper = new UIhelper(page);
-          const common = new Common(page);
-          await common.loginAsGuest();
-          await uiHelper.verifyHeading("Welcome back!");
-        } finally {
-          await page.goto("about:blank").catch(() => {});
-        }
+        const uiHelper = new UIhelper(page);
+        const common = new Common(page);
+        await common.loginAsGuest();
+        await uiHelper.verifyHeading("Welcome back!");
       });
     });
   }

e2e-tests/playwright/e2e/plugin-division-mode-schema/verify-schema-mode.spec.ts

@@ -12,7 +12,7 @@ import { ChildProcessWithoutNullStreams, spawn } from "child_process";
 import { test, expect } from "@support/coverage/test";
 
 import { Common } from "../../utils/common";
-import { resolveInstallMethod } from "../../utils/helper";
+import { getReleaseName, resolveInstallMethod } from "../../utils/helper";
 import { KubeClient } from "../../utils/kube-client";
 import { ensureRuntimeDeployed } from "../../utils/runtime-deploy";
 import { setPortForwardRestarter } from "./schema-mode-db";
@@ -80,7 +80,7 @@ function killPortForward(proc: ChildProcessWithoutNullStreams | undefined): Prom
 
 test.describe("Verify pluginDivisionMode: schema", () => {
   const namespace = process.env.NAME_SPACE_RUNTIME ?? "showcase-runtime";
-  const releaseName = process.env.RELEASE_NAME ?? "rhdh";
+  const releaseName = getReleaseName();
   const installMethod = resolveInstallMethod();
 
   let portForwardProcess: ChildProcessWithoutNullStreams | undefined;
@@ -156,6 +156,17 @@ test.describe("Verify pluginDivisionMode: schema", () => {
     await killPortForward(portForwardProcess);
   });
 
+  // RHDH's frontend opens an SSE (EventSource) connection for live
+  // updates. With tracing enabled, Playwright's fixture teardown hangs
+  // waiting for network idle, which never resolves while SSE stays open
+  // (microsoft/playwright#41513, fixed in v1.62). Navigating away drops
+  // the connection so teardown completes immediately.
+  // Requesting `page` creates a context for every test, including non-UI
+  // ones — acceptable overhead vs per-test conditional logic.
+  test.afterEach(async ({ page }) => {
+    await page.goto("about:blank").catch(() => {});
+  });
+
   test("Verify database user has restricted permissions", async () => {
     const hasRestrictedPerms = await testSetup.verifyRestrictedDatabasePermissions();
     expect(hasRestrictedPerms).toBe(true);
@@ -181,17 +192,10 @@ test.describe("Verify pluginDivisionMode: schema", () => {
     }
 
     const common = new Common(page);
-    try {
-      await common.loginAsGuest();
+    await common.loginAsGuest();
 
-      await expect(page.getByRole("heading", { level: 1 })).toBeVisible();
+    await expect(page.getByRole("heading", { level: 1 })).toBeVisible();
 
-      console.log("RHDH is accessible - plugins successfully created schemas in schema mode");
-    } finally {
-      // Navigate away from RHDH to close WebSocket connections before
-      // Playwright tears down the page — prevents a long hang during
-      // context/trace cleanup.
-      await page.goto("about:blank").catch(() => {});
-    }
+    console.log("RHDH is accessible - plugins successfully created schemas in schema mode");
   });
 });

e2e-tests/playwright/utils/helper.ts

@@ -131,6 +131,20 @@ export function resolveInstallMethod(): "helm" | "operator" {
   return job.includes("operator") ? "operator" : "helm";
 }
 
+/**
+ * Canonical release name resolution. Returns the RELEASE_NAME env var if set
+ * and non-empty, otherwise defaults to "rhdh".
+ *
+ * Note: the explicit check is used instead of `||` because oxlint's
+ * strict-boolean-expressions and prefer-nullish-coalescing rules
+ * (pedantic category) reject `||` on string operands.
+ */
+export function getReleaseName(): string {
+  return process.env.RELEASE_NAME !== undefined && process.env.RELEASE_NAME !== ""
+    ? process.env.RELEASE_NAME
+    : "rhdh";
+}
+
 /** Base64-encode a string. */
 export function base64Encode(value: string): string {
   return Buffer.from(value).toString("base64");

e2e-tests/playwright/utils/kube-client/helpers.ts

@@ -1,7 +1,7 @@
 import * as k8s from "@kubernetes/client-node";
 
 import { getErrorMessage, hasErrorResponse, hasStatusCode } from "../errors";
-import { resolveInstallMethod } from "../helper";
+import { getReleaseName, resolveInstallMethod } from "../helper";
 
 export function isRecord(value: unknown): value is Record<string, unknown> {
   return typeof value === "object" && value !== null && !Array.isArray(value);
@@ -135,10 +135,7 @@ export function getKubeApiErrorMessage(error: unknown): string {
  * then falls back to JOB_NAME pattern matching.
  */
 export function getRhdhDeploymentName(): string {
-  const releaseName =
-    process.env.RELEASE_NAME !== undefined && process.env.RELEASE_NAME !== ""
-      ? process.env.RELEASE_NAME
-      : "rhdh";
+  const releaseName = getReleaseName();
   return resolveInstallMethod() === "operator"
     ? `backstage-${releaseName}`
     : `${releaseName}-developer-hub`;