feat(e2e): cluster-free local E2E harness (legacy app, Tier B)

[gustavolira]

Summary

Adds a cluster-free E2E harness for RHDH: run the existing Playwright specs against a real RHDH instance without an OpenShift/Kubernetes cluster or container image build. Playwright boots the backend and the legacy app (packages/app) dev servers in-process, installs dynamic plugins from the public OCI registry (ghcr, via skopeo — no source build), and drives the browser against them.

Part of RHIDP-13501 (E2E Test Optimization), Layer 4a spike RHIDP-15075.

Status: 4 tests green in CI ✅

The GitHub Actions job (.github/workflows/e2e-cluster-free.yaml) runs cluster-free and passes in ~3.5 min:

✓ guest-signin-happy-path › Homepage renders with Search Bar, Quick Access and Starred Entities
✓ guest-signin-happy-path › Verify Profile is Guest in the Settings page
✓ guest-signin-happy-path › Sign Out and Verify that you return to the Sign-in page
✓ learning-path-page      › Links in Learning Paths open in a new tab
  4 passed (1.5m)

Install (cached): ~90s · Populate plugins from OCI: ~11s · Boot + run: ~2m

The full guest-signin-happy-path spec passes unmodified — including Settings/Sign-out, which navigate through the global-header dynamic plugin's profile dropdown.

How it works

1. Install deps + skopeo (Playwright container v1.61.1-noble — browsers preinstalled).
2. Populate dynamic-plugins-root via ./e2e-tests/local-harness/populate.sh — the single source of truth for the populate step (CI, docs, and the fail-fast globalSetup message all point to it). It installs the plugin set from e2e-tests/local-harness/dynamic-plugins.yaml (dynamic-home-page + global-header, from ghcr) with the CLI pinned to the version CI uses.
3. yarn e2e:legacy-local — boots backend + legacy app with the config overlay, guest auth, and runs the specs.

Test scoping uses a @cluster-free tag + a testMatch file allowlist: enabling a validated spec is tagging the test and adding its file to testMatch.

The global-header fix (why Settings/Sign-out work now)

The repo's static app-config.dynamic-plugins.yaml only mounts the bare GlobalHeader container with no children, so off-cluster the header rendered empty. In-cluster, the full mount points (ProfileDropdown, Settings/Sign-out items, search) come from the plugin's pluginConfig in the catalog index. The harness now carries that canonical pluginConfig (copied from rhdh-plugins workspaces/global-header/.../app-config.dynamic.yaml) on the OCI entry, and the webServer loads the generated dynamic-plugins-root/app-config.dynamic-plugins.yaml last — the same file and merge order the production container uses. This pattern generalizes to any plugin whose config is not in the repo's static file.

Why OCI instead of the catalog index

The catalog index's dynamic-plugins.default.yaml references core plugins via local ./dynamic-plugins/dist/… paths that only exist after a source build (which CI doesn't do). The overlays repo publishes those plugins as OCI artifacts, which install-dynamic-plugins can pull with skopeo directly — no build.

Scope / follow-ups

• Runs 4 tests today (full guest-signin + learning-paths). Next candidates mapped in docs/e2e-tests/local-e2e-harness.md (Known issues): settings.spec.ts needs the CI catalog test entities, home-page-customization.spec.ts needs the CI home-card customization — follow-up PRs (spec inventory tracked as RHIDP-15076).
• OCI tags are backstage 1.49.4 (RHDH is on 1.52.0); they load fine, but bump when the overlays repo publishes matching tags (noted in the config file).
• Overlay plugin tests are out of scope here — they get their own PR in the overlay repo.
• Not wired into any required check yet; it's an additive cluster-free job.

🤖 Generated with Claude Code

[rhdh-qodo-merge]

PR Summary by Qodo

Add cluster-free local Playwright E2E harnesses (legacy Tier B + app-next)
✨ Enhancement 🧪 Tests 📝 Documentation ⚙️ Configuration changes 🕐 40+ Minutes

Description

• Add Playwright harnesses that boot backend + frontend dev servers in-process, off-cluster.
• Provide legacy Tier B harness to run existing specs unchanged with Scalprum dynamic plugins.
• Document setup, limitations, and add guest-auth + in-memory SQLite config overlay.

Diagram

graph TD
  PW(["Playwright runner"]) --> CH{"Harness"} -->|"Legacy Tier B"| LEG(["Legacy app dev server (janus-cli)"]) --> BR["Browser-driven E2E"]
  CH -->|"app-next"| NEXT(["App-next dev server (backstage-cli)"]) --> BR
  PW --> BE(["Backend dev server (backstage-cli)"]) --> BR
  CFG[["app-config.local-e2e.yaml"]] --> BE
  CFG --> LEG
  CFG --> NEXT
  DPR[("dynamic-plugins-root")] --> LEG
  subgraph Legend
    direction LR
    _svc(["Dev server"]) ~~~ _file[["Config file"]] ~~~ _db[("Plugins root")] ~~~ _dec{"Decision"} ~~~ _box["Test target"]
  end

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Single Playwright config with projects (legacy + app-next)

• ➕ Eliminates duplication of reporters/timeouts/webServer wiring
• ➕ Allows running one or both harnesses via Playwright project selection
• ➕ Centralizes env handling (PATH, CI reuseExistingServer)
• ➖ Slightly more complex conditional webServer setup
• ➖ Harder to keep legacy testMatch constraints isolated without care

2. Use container-based rhdh-local for local E2E

• ➕ Closest to production image runtime; fewer dev-server quirks
• ➕ Avoids needing local workspace CLIs on PATH
• ➖ Requires container runtime and image pulls
• ➖ Heavier/slower for automated E2E; contradicts ‘cluster-free/in-process’ goal

Recommendation: Keep the PR’s approach (in-process dev servers via Playwright webServer) because it directly optimizes for fast, cluster-free automated E2E and preserves existing legacy specs unchanged. If this becomes long-lived, consider consolidating common Playwright config into a shared base and using Playwright projects to reduce config duplication while keeping distinct legacy vs app-next server commands.

Files changed (6) +381 / -1

Tests (3) +248 / -0

playwright.app-next-local.config.tsAdd Playwright config to boot backend + app-next dev server +94/-0

Add Playwright config to boot backend + app-next dev server

• Defines a Playwright configuration that starts the backend and packages/app-next dev server via webServer, reusing servers locally and forcing fresh starts in CI. Ensures repo-root .bin CLIs are discoverable via PATH and sets reporting/timeout defaults for the harness.

e2e-tests/playwright.app-next-local.config.ts

playwright.legacy-local.config.tsAdd Tier B Playwright config for legacy app with dynamic plugins +105/-0

Add Tier B Playwright config for legacy app with dynamic plugins

• Creates a legacy harness config that runs a curated subset of existing specs off-cluster. Boots backend and packages/app via backstage-cli/janus-cli with layered configs including dynamic plugins and the local E2E overlay, and reuses servers outside CI.

e2e-tests/playwright.legacy-local.config.ts

guest-identity.spec.tsAdd app-next local guest sign-in + identity smoke test +49/-0

Add app-next local guest sign-in + identity smoke test

• Introduces a minimal E2E spec validating the app-next harness: guest sign-in succeeds, core navigation is present, and the Settings page renders the guest Backstage identity. Assertions intentionally match app-next UI behavior rather than legacy home-page expectations.

e2e-tests/playwright/app-next-local/guest-identity.spec.ts

Documentation (1) +105 / -0

local-e2e-harness.mdDocument cluster-free local E2E harness usage and constraints +105/-0

Document cluster-free local E2E harness usage and constraints

• Adds a dedicated guide describing the legacy Tier B and app-next harnesses, dynamic-plugins-root population paths, and what’s currently verified. Documents key limitations (global-header mounting, app-next dynamic plugin loading, and external-service-dependent specs) and explains why this approach differs from rhdh-local.

docs/e2e-tests/local-e2e-harness.md

Other (2) +28 / -1

app-config.local-e2e.yamlAdd local E2E config overlay (guest auth + in-memory DB) +25/-0

Add local E2E config overlay (guest auth + in-memory DB)

• Introduces a shared config overlay for cluster-free E2E runs. Enables the guest auth provider (including CI-safe allowance) and forces an in-memory better-sqlite3 backend database for fully self-contained runs.

app-config.local-e2e.yaml

package.jsonAdd yarn scripts to run legacy-local and app-next-local harnesses +3/-1

Add yarn scripts to run legacy-local and app-next-local harnesses

• Adds two Playwright entrypoint scripts for the new harness configs. Keeps existing tooling scripts intact while enabling one-command local harness execution.

e2e-tests/package.json

[rhdh-qodo-merge]

Code Review by Qodo

🐞 Bugs (2) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. Wrong server reused 🐞 Bug ☼ Reliability

Description

With reuseExistingServer enabled outside CI, Playwright will reuse any process responding on the
configured localhost URLs, even if it's not the intended RHDH harness. Because the URLs are
hard-coded to localhost:3000 and localhost:7007, this can lead to tests running against
stale/foreign servers and producing misleading results.

Code

e2e-tests/playwright.legacy-local.config.ts[R88-100]

+      url: backendReadiness,
+      reuseExistingServer: !process.env.CI,
+      timeout: 180 * 1000,
+      stdout: "pipe",
+      stderr: "pipe",
+    },
+    {
+      command: `janus-cli package start ${sharedConfigArgs}`,
+      cwd: "../packages/app",
+      env: { ...process.env, PATH: `${repoRootBin}:${process.env.PATH}` },
+      url: frontendUrl,
+      reuseExistingServer: !process.env.CI,
+      timeout: 240 * 1000,

Relevance

⭐⭐ Medium

No prior review evidence on reuseExistingServer risks; pattern likely tolerated for local dev but
could be tightened.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

Both configs hard-code the target URLs/ports and enable reuse of any existing server when CI is
not set, which makes it possible to run tests against an unintended process bound to those ports.

e2e-tests/playwright.legacy-local.config.ts[31-33]
e2e-tests/playwright.legacy-local.config.ts[88-100]
e2e-tests/playwright.app-next-local.config.ts[24-26]
e2e-tests/playwright.app-next-local.config.ts[75-89]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The harness sets `reuseExistingServer: !process.env.CI` while targeting fixed `http://localhost:3000` and `http://localhost:7007/...`. If anything else is listening on those ports (or a previous run is still alive), Playwright may attach to the wrong process.

### Issue Context
This affects both `playwright.legacy-local.config.ts` and `playwright.app-next-local.config.ts`.

### Fix
Pick one:
1) Make reuse explicit (opt-in), e.g. `reuseExistingServer: process.env.REUSE_EXISTING_SERVER === 'true'`.
2) Use per-run ports (random/free port) and pass them into backend/frontend via config/env overrides, and derive `baseURL`/readiness URLs from those ports.
3) Add a lightweight identity check before reusing (e.g., fetch a known endpoint and validate a header/body) and fail if it doesn’t match.

### Fix Focus Areas
- e2e-tests/playwright.legacy-local.config.ts[31-33]
- e2e-tests/playwright.legacy-local.config.ts[88-100]
- e2e-tests/playwright.app-next-local.config.ts[24-26]
- e2e-tests/playwright.app-next-local.config.ts[75-89]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. NODE_OPTIONS clobbered 🐞 Bug ☼ Reliability

Description

The local harness webServer env sets NODE_OPTIONS to a fixed value, overwriting any pre-existing
NODE_OPTIONS from the environment. This can break harness startup in CI/dev setups that rely on
other Node flags (e.g., memory limits or preload hooks).

Code

e2e-tests/playwright.legacy-local.config.ts[R83-87]

+      env: {
+        ...process.env,
+        PATH: `${repoRootBin}:${process.env.PATH}`,
+        NODE_OPTIONS: "--no-node-snapshot",
+      },

Relevance

⭐ Low

Repo already sets NODE_OPTIONS="--no-node-snapshot" unconditionally in Docker/CI; clobbering appears
accepted.

PR-#2297

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

Both harness configs spread ...process.env but then unconditionally override NODE_OPTIONS, which
discards any upstream flags.

e2e-tests/playwright.legacy-local.config.ts[83-87]
e2e-tests/playwright.app-next-local.config.ts[70-74]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The Playwright `webServer.env` blocks replace `NODE_OPTIONS` with `"--no-node-snapshot"`, which drops any existing `NODE_OPTIONS` settings inherited from the parent environment.

### Issue Context
This affects both the legacy and app-next local harness configs.

### Fix
Construct `NODE_OPTIONS` by appending `--no-node-snapshot` to any existing value, e.g.:

```ts
const nodeOptions = [process.env.NODE_OPTIONS, '--no-node-snapshot']
 .filter(Boolean)
 .join(' ');

NODE_OPTIONS: nodeOptions,
```

### Fix Focus Areas
- e2e-tests/playwright.legacy-local.config.ts[83-87]
- e2e-tests/playwright.app-next-local.config.ts[70-74]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[github-actions]

The container image build workflow finished with status: cancelled.

[github-actions]

Image was built and published successfully. It is available at:

• quay.io/rhdh-community/rhdh:pr-5005
• quay.io/rhdh-community/rhdh:pr-5005-bfbe9dc6

[github-actions]

The container image build workflow finished with status: cancelled.

[github-actions]

Image was built and published successfully. It is available at:

• quay.io/rhdh-community/rhdh:pr-5005
• quay.io/rhdh-community/rhdh:pr-5005-ebddd2cc

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 54.77%. Comparing base (5c9e678) to head (ee3120b).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5005      +/-   ##
==========================================
- Coverage   55.39%   54.77%   -0.62%     
==========================================
  Files         122      110      -12     
  Lines        2365     2147     -218     
  Branches      563      538      -25     
==========================================
- Hits         1310     1176     -134     
+ Misses       1049      969      -80     
+ Partials        6        2       -4     

Flag	Coverage Δ
rhdh	54.77% <ø> (-0.62%)	⬇️

---

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5c9e678...ee3120b. Read the comment docs.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

Image was built and published successfully. It is available at:

• quay.io/rhdh-community/rhdh:pr-5005
• quay.io/rhdh-community/rhdh:pr-5005-1bf02b56

[gustavolira]

/test e2e-ocp-helm-nightly

[github-actions]

Image was built and published successfully. It is available at:

• quay.io/rhdh-community/rhdh:pr-5005
• quay.io/rhdh-community/rhdh:pr-5005-ab72af06

[github-actions]

Image was built and published successfully. It is available at:

• quay.io/rhdh-community/rhdh:pr-5005
• quay.io/rhdh-community/rhdh:pr-5005-f2019bbc

[github-actions]

Image was built and published successfully. It is available at:

• quay.io/rhdh-community/rhdh:pr-5005
• quay.io/rhdh-community/rhdh:pr-5005-c85e002a

[github-actions]

Image was built and published successfully. It is available at:

• quay.io/rhdh-community/rhdh:pr-5005
• quay.io/rhdh-community/rhdh:pr-5005-3d62adfc

[github-actions]

The container image build workflow finished with status: cancelled.

[github-actions]

The container image build workflow finished with status: cancelled.

[github-actions]

Image was built and published successfully. It is available at:

• quay.io/rhdh-community/rhdh:pr-5005
• quay.io/rhdh-community/rhdh:pr-5005-9c56ab12

[github-actions]

The container image build workflow finished with status: cancelled.

[github-actions]

Image was built and published successfully. It is available at:

• quay.io/rhdh-community/rhdh:pr-5005
• quay.io/rhdh-community/rhdh:pr-5005-faa8069b

[openshift-ci]

@gustavolira: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-ocp-helm-nightly	1bf02b5	link	false	/test e2e-ocp-helm-nightly

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[github-actions]

The container image build workflow finished with status: cancelled.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[github-actions]

Image was built and published successfully. It is available at:

• quay.io/rhdh-community/rhdh:pr-5005
• quay.io/rhdh-community/rhdh:pr-5005-ee3120b4