feat: swap to instrumented __coverage image in dedicated nightly coverage runs

docs/changelog.md

@@ -2,7 +2,13 @@
 
 All notable changes to this project will be documented in this file.
 
-## [2.1.0] - Current
+## [2.2.0] - Current
+
+### Added
+
+- **Nightly coverage image swap** (`E2E_NIGHTLY_COVERAGE`): In nightly plugin resolution, an explicit opt-in points a rolled-out frontend plugin (workspace with a `coverage-anchors/` directory) at the overlay's instrumented `__coverage` ghcr build so a coverage-dedicated nightly run can collect browser coverage. This includes DPDY plugins: in a coverage run they bypass `{{inherit}}` (the Konflux catalog image, which can't be instrumented) and use the ghcr build of the same source. Gated separately from the ambient `E2E_COLLECT_COVERAGE`, so the functional nightly's resolution is unchanged — it still deploys the shipped `{{inherit}}`/Konflux builds.
+
+## [2.1.0]
 
 ### Added
 

docs/guide/configuration/environment-variables.md

@@ -48,6 +48,7 @@ These control automatic plugin configuration injection from metadata files.
 | ------------------------------------- | ------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
 | `GIT_PR_NUMBER`                       | PR number (set by OpenShift CI)                         | Enables OCI URL generation for PR builds                                                                                                                                                 |
 | `E2E_NIGHTLY_MODE`                    | When `"true"`, activates nightly mode                   | Plugins in `default.packages.yaml` with OCI metadata use `{{inherit}}` (RHDH resolves both OCI tag and config from DPDY); other OCI plugins use full metadata refs with config injection |
+| `E2E_NIGHTLY_COVERAGE`                | When `"true"`, opt into the nightly coverage image swap | In nightly resolution, a rolled-out frontend plugin (workspace has `coverage-anchors/`) gets its released OCI tag swapped to the instrumented `__coverage` variant so the run can collect browser coverage. Off by default — the functional nightly is unaffected; `{{inherit}}` plugins are never swapped |
 | `RHDH_SKIP_PLUGIN_METADATA_INJECTION` | When `"true"`, disables metadata injection              | Local-only opt-out (ignored when `CI=true`)                                                                                                                                              |
 | `RELEASE_BRANCH_NAME`                 | Release branch (set by OpenShift CI step registry)      | Used to fetch `default.packages.yaml` for DPDY resolution in nightly mode. Required in CI, defaults to `main` locally                                                                    |
 | `NIGHTLY_DPDY_OCI_REGISTRY`           | OCI registry for `{{inherit}}` refs                     | Overrides default `registry.access.redhat.com/rhdh` for all plugins using `{{inherit}}` in nightly mode                                                                                  |

docs/overlay/reference/plugin-metadata-resolution.md

@@ -106,17 +106,33 @@ For each plugin, the resolver checks in order:
    No  ↓
 
 3. Is nightly mode AND plugin is in default.packages.yaml AND metadata spec.dynamicArtifact is OCI?
-   Yes → use {{inherit}} tag:  oci://{registry}/plugin:{{inherit}}
+   Yes ↓
+       In a coverage run (E2E_NIGHTLY_COVERAGE=true), a rolled-out frontend plugin
+       bypasses {{inherit}} and uses the ghcr __coverage build instead (see step 4),
+       because the {{inherit}}/Konflux catalog image can't be instrumented.
+       Otherwise → use {{inherit}} tag:  oci://{registry}/plugin:{{inherit}}
          RHDH resolves both the OCI tag (version) and default config from its built-in DPDY.
          Registry: NIGHTLY_DPDY_OCI_REGISTRY_MAP > NIGHTLY_DPDY_OCI_REGISTRY > default registry.access.redhat.com/rhdh
    No  ↓
 
 4. Use metadata's dynamicArtifact as-is
    (OCI ref → OCI ref, wrapper path → wrapper path)
+   In a coverage run (E2E_NIGHTLY_COVERAGE=true), a rolled-out frontend plugin
+   (workspace has a coverage-anchors/ directory) gets its OCI tag swapped to the
+   instrumented __coverage variant:
+     oci://ghcr.io/.../plugin:bs_X__Y!alias → ...:bs_X__Y__coverage!alias
 ```
 
 Metadata is the source of truth for the package reference, except for plugins in `default.packages.yaml` with OCI metadata in nightly mode — these use `{{inherit}}` so RHDH resolves both the OCI tag and config from its built-in DPDY, testing the exact versions and configuration shipped in the RC.
 
+### Coverage image swap (`E2E_NIGHTLY_COVERAGE`)
+
+A nightly run can only collect browser coverage if RHDH deploys the **instrumented** `__coverage` plugin image (built by the overlay release publish). When `E2E_NIGHTLY_COVERAGE=true`, step 4 above swaps a rolled-out frontend plugin's released OCI tag to its `__coverage` variant.
+
+This is a separate flag from the ambient `E2E_COLLECT_COVERAGE` (which only toggles the collector fixture) on purpose: the functional nightly runs with `E2E_COLLECT_COVERAGE=true` by default, and the `__coverage` variant is built non-fatally, so swapping there could point at a tag that doesn't exist and break the deployment. The explicit `E2E_NIGHTLY_COVERAGE` opt-in keeps the functional nightly's resolution unchanged; only a coverage-dedicated run (which ensures the images exist) sets it.
+
+In a coverage run this also applies to **DPDY plugins**: instead of resolving to `{{inherit}}` (the Konflux catalog image at `registry.access.redhat.com/rhdh`, which can't be instrumented), a rolled-out frontend DPDY plugin is pointed at the overlay's instrumented ghcr `__coverage` build of the same plugin source. The functional nightly still uses `{{inherit}}` and tests the shipped Konflux build — the coverage run is a separate measurement that deliberately deploys the instrumentable ghcr build instead. No downstream/Konflux pipeline changes are needed.
+
 ## Resolution Scenarios
 
 The tables below show what happens to each plugin type in PR check and nightly modes. Local dev behaves the same as PR check (metadata refs + full config injection).

package.json

@@ -1,6 +1,6 @@
 {
   "name": "@red-hat-developer-hub/e2e-test-utils",
-  "version": "2.1.0",
+  "version": "2.2.0",
   "description": "Test utilities for RHDH E2E tests",
   "license": "Apache-2.0",
   "repository": {

src/utils/plugin-metadata.ts

@@ -206,6 +206,14 @@ function toDisplayName(packageName: string): string {
   return packageName.replace(/^@/, "").replace(/\//g, "-");
 }
 
+// Append the __coverage suffix to an OCI image tag, before the optional
+// !<extractPath> — the instrumented variant the overlay's release publish
+// builds. The tag group is greedy up to the first `!`, so the suffix always
+// lands at the end of the tag.
+function toCoverageImageRef(ref: string): string {
+  return ref.replace(/(:[^!]+)/, "$1__coverage");
+}
+
 // ── Metadata Loading ──────────────────────────────────────────────────────────
 
 export const DEFAULT_METADATA_PATH = "../metadata";
@@ -464,17 +472,34 @@ async function resolvePluginPackages(
   metadataPath: string,
   dpdyPackages: Set<string> | null = null,
 ): Promise<PluginEntry[]> {
+  const workspaceRoot = path.resolve(metadataPath, "..");
+
   // Build PR OCI URLs if applicable
   const prNumber = process.env.GIT_PR_NUMBER;
   let prOciUrls: Map<string, string> | null = null;
   if (prNumber) {
     console.log(
       `[PluginMetadata] PR build detected (PR #${prNumber}), fetching OCI URLs...`,
     );
-    const workspacePath = path.resolve(metadataPath, "..");
-    prOciUrls = await getOCIUrlsForPR(workspacePath, prNumber);
+    prOciUrls = await getOCIUrlsForPR(workspaceRoot, prNumber);
   }
 
+  // A dedicated coverage run swaps rolled-out frontend plugins to their
+  // instrumented __coverage image so the browser exposes window.__coverage__.
+  //
+  // This is gated on E2E_NIGHTLY_COVERAGE (an explicit opt-in), NOT on the
+  // ambient E2E_COLLECT_COVERAGE: the functional nightly runs with coverage
+  // collection on by default but deploys RELEASED images, and the __coverage
+  // variant is built non-fatally by the overlay release publish — so swapping
+  // there could point at a tag that doesn't exist and break the deployment.
+  // Requiring the explicit opt-in keeps the functional nightly's resolution
+  // identical to today; only a coverage-dedicated run (which ensures the
+  // images exist) sets the flag. The coverage-anchors/ check further restricts
+  // the swap to rolled-out workspaces.
+  const coverageSwap =
+    process.env.E2E_NIGHTLY_COVERAGE === "true" &&
+    fs.existsSync(path.join(workspaceRoot, "coverage-anchors"));
+
   return plugins.map((plugin) => {
     const pkg = plugin.package;
     const pluginName = extractPluginName(pkg);
@@ -491,9 +516,7 @@ async function resolvePluginPackages(
           const usesCoverage =
             process.env.E2E_COLLECT_COVERAGE === "true" &&
             metadata.role === "frontend-plugin";
-          const resolved = usesCoverage
-            ? prUrl.replace(/(:[^!]+)/, "$1__coverage")
-            : prUrl;
+          const resolved = usesCoverage ? toCoverageImageRef(prUrl) : prUrl;
           console.log(`[PluginMetadata] PR: ${pkg} → ${resolved}`);
           return { ...plugin, package: resolved };
         }
@@ -507,16 +530,33 @@ async function resolvePluginPackages(
         dpdyPackages?.has(metadata.packageName) &&
         metadata.packagePath.startsWith("oci://")
       ) {
+        // In a coverage run, a {{inherit}} ref would deploy the Konflux catalog
+        // image (registry.access.redhat.com/rhdh), which we can't instrument.
+        // Bypass it and deploy the overlay's instrumented __coverage build from
+        // ghcr (metadata.packagePath) — same plugin source, just built by us, so
+        // the run can collect coverage. The functional nightly (no coverage
+        // opt-in) still uses {{inherit}}, i.e. the shipped Konflux build.
+        if (coverageSwap && metadata.role === "frontend-plugin") {
+          const resolved = toCoverageImageRef(metadata.packagePath);
+          console.log(`[PluginMetadata] DPDY coverage: ${pkg} → ${resolved}`);
+          return { ...plugin, package: resolved };
+        }
         const registry = getDpdyRegistry(metadata.packageName);
         const inheritRef = `oci://${registry}/${displayName}:{{inherit}}`;
         console.log(`[PluginMetadata] DPDY inherit: ${pkg} → ${inheritRef}`);
         return { ...plugin, package: inheritRef };
       }
 
       // OCI: use metadata's dynamicArtifact directly (not in default.packages.yaml, or not nightly).
+      // For a rolled-out frontend plugin in a coverage run, swap to the
+      // instrumented __coverage image so the nightly can collect coverage.
       if (metadata.packagePath.startsWith("oci://")) {
-        console.log(`[PluginMetadata] ${pkg} → ${metadata.packagePath}`);
-        return { ...plugin, package: metadata.packagePath };
+        const resolved =
+          coverageSwap && metadata.role === "frontend-plugin"
+            ? toCoverageImageRef(metadata.packagePath)
+            : metadata.packagePath;
+        console.log(`[PluginMetadata] ${pkg} → ${resolved}`);
+        return { ...plugin, package: resolved };
       }
 
       // Wrapper (local path): metadata is the source of truth.

src/utils/tests/plugin-metadata.nightly.test.ts

@@ -5,6 +5,9 @@
 import { describe, it, beforeEach, afterEach } from "node:test";
 import assert from "node:assert";
 import fs from "fs-extra";
+import path from "path";
+import os from "os";
+import yaml from "js-yaml";
 import {
   isNightlyJob,
   processPluginsForDeployment,
@@ -17,8 +20,8 @@
 
 describe("isNightlyJob", () => {
   const env = withCleanEnv();
   beforeEach(() => env.save());
   afterEach(() => env.restore());
 
   it("returns false with no env vars set", () => {
     delete process.env.E2E_NIGHTLY_MODE;
@@ -102,12 +105,12 @@
 
 describe("processPluginsForDeployment — nightly mode", () => {
   const env = withCleanEnv();
   beforeEach(() => {
     env.save();
     delete process.env.GIT_PR_NUMBER;
     process.env.E2E_NIGHTLY_MODE = "true";
   });
   afterEach(() => env.restore());
 
   it("skips metadata injection for wrapper plugins in nightly mode", async () => {
     const metadataDir = await createMetadataFixture([
@@ -313,7 +316,7 @@
 
 describe("processPluginsForDeployment — nightly {{inherit}}", () => {
   const env = withCleanEnv();
   beforeEach(() => {
     env.save();
     delete process.env.GIT_PR_NUMBER;
     process.env.E2E_NIGHTLY_MODE = "true";
@@ -951,3 +954,174 @@
     );
   });
 });
+
+// ── Nightly coverage image swap ──────────────────────────────────────────────
+
+describe("processPluginsForDeployment — nightly coverage swap", () => {
+  const env = withCleanEnv();
+  beforeEach(() => {
+    env.save();
+    delete process.env.GIT_PR_NUMBER;
+    process.env.E2E_NIGHTLY_MODE = "true";
+  });
+  afterEach(() => env.restore());
+
+  const OCI_REF =
+    "oci://ghcr.io/redhat-developer/rhdh-plugin-export-overlays/red-hat-developer-hub-backstage-plugin-theme:bs_1.49.4__0.14.5!red-hat-developer-hub-backstage-plugin-theme";
+
+  // Builds a workspace layout (<ws>/metadata + optional <ws>/coverage-anchors)
+  // and returns the metadata dir to pass as metadataPath. The resolver derives
+  // the workspace root as the parent of the metadata dir.
+  async function createCoverageWorkspace(opts: {
+    rolledOut: boolean;
+    role?: string;
+  }): Promise<string> {
+    const ws = await fs.mkdtemp(path.join(os.tmpdir(), "cov-ws-"));
+    const metadataDir = path.join(ws, "metadata");
+    await fs.ensureDir(metadataDir);
+    if (opts.rolledOut) {
+      await fs.ensureDir(path.join(ws, "coverage-anchors"));
+    }
+    await fs.writeFile(
+      path.join(metadataDir, "theme.yaml"),
+      yaml.dump({
+        apiVersion: "extensions.backstage.io/v1alpha1",
+        kind: "Package",
+        metadata: { name: "theme" },
+        spec: {
+          packageName: "@red-hat-developer-hub/backstage-plugin-theme",
+          dynamicArtifact: OCI_REF,
+          ...(opts.role ? { backstage: { role: opts.role } } : {}),
+        },
+      }),
+    );
+    return metadataDir;
+  }
+
+  const config: DynamicPluginsConfig = {
+    plugins: [{ package: OCI_REF, disabled: false }],
+  };
+
+  async function resolveWith(metadataDir: string): Promise<string> {
+    const result = await processPluginsForDeployment(
+      config,
+      metadataDir,
+      new Set(), // empty DPDY → OCI-direct branch, not {{inherit}}
+    );
+    return result.plugins![0].package;
+  }
+
+  it("swaps to the __coverage image for a rolled-out frontend plugin when opted in", async () => {
+    process.env.E2E_NIGHTLY_COVERAGE = "true";
+    const metadataDir = await createCoverageWorkspace({
+      rolledOut: true,
+      role: "frontend-plugin",
+    });
+    try {
+      const pkg = await resolveWith(metadataDir);
+      assert.strictEqual(
+        pkg,
+        "oci://ghcr.io/redhat-developer/rhdh-plugin-export-overlays/red-hat-developer-hub-backstage-plugin-theme:bs_1.49.4__0.14.5__coverage!red-hat-developer-hub-backstage-plugin-theme",
+        "tag must get the __coverage suffix, the !path must be preserved",
+      );
+    } finally {
+      await fs.remove(path.resolve(metadataDir, ".."));
+    }
+  });
+
+  it("does NOT swap in the functional nightly (E2E_NIGHTLY_COVERAGE unset), even with E2E_COLLECT_COVERAGE on", async () => {
+    // The functional nightly runs with E2E_COLLECT_COVERAGE=true by default but
+    // must keep deploying the released image — swapping there could point at a
+    // __coverage tag that doesn't exist and break the deployment.
+    process.env.E2E_COLLECT_COVERAGE = "true";
+    delete process.env.E2E_NIGHTLY_COVERAGE;
+    const metadataDir = await createCoverageWorkspace({
+      rolledOut: true,
+      role: "frontend-plugin",
+    });
+    try {
+      assert.strictEqual(
+        await resolveWith(metadataDir),
+        OCI_REF,
+        "functional nightly resolution must be unchanged (no swap)",
+      );
+    } finally {
+      await fs.remove(path.resolve(metadataDir, ".."));
+    }
+  });
+
+  it("does not swap when the workspace has no coverage-anchors (not rolled out)", async () => {
+    process.env.E2E_NIGHTLY_COVERAGE = "true";
+    const metadataDir = await createCoverageWorkspace({
+      rolledOut: false,
+      role: "frontend-plugin",
+    });
+    try {
+      assert.strictEqual(await resolveWith(metadataDir), OCI_REF);
+    } finally {
+      await fs.remove(path.resolve(metadataDir, ".."));
+    }
+  });
+
+  it("does not swap a non-frontend plugin even when rolled out and opted in", async () => {
+    process.env.E2E_NIGHTLY_COVERAGE = "true";
+    const metadataDir = await createCoverageWorkspace({
+      rolledOut: true,
+      role: "backend-plugin",
+    });
+    try {
+      assert.strictEqual(await resolveWith(metadataDir), OCI_REF);
+    } finally {
+      await fs.remove(path.resolve(metadataDir, ".."));
+    }
+  });
+
+  it("swaps a DPDY plugin to the ghcr __coverage build (bypassing {{inherit}}) when opted in", async () => {
+    // A {{inherit}} ref would deploy the Konflux catalog image, which can't be
+    // instrumented. In a coverage run, a rolled-out frontend DPDY plugin must
+    // instead use the overlay's instrumented ghcr build of the same source.
+    process.env.E2E_NIGHTLY_COVERAGE = "true";
+    const metadataDir = await createCoverageWorkspace({
+      rolledOut: true,
+      role: "frontend-plugin",
+    });
+    try {
+      const result = await processPluginsForDeployment(
+        config,
+        metadataDir,
+        new Set(["@red-hat-developer-hub/backstage-plugin-theme"]), // in DPDY
+      );
+      assert.strictEqual(
+        result.plugins![0].package,
+        "oci://ghcr.io/redhat-developer/rhdh-plugin-export-overlays/red-hat-developer-hub-backstage-plugin-theme:bs_1.49.4__0.14.5__coverage!red-hat-developer-hub-backstage-plugin-theme",
+        "DPDY plugin in a coverage run must use the ghcr __coverage build, not {{inherit}}",
+      );
+    } finally {
+      await fs.remove(path.resolve(metadataDir, ".."));
+    }
+  });
+
+  it("keeps a DPDY plugin on {{inherit}} in the functional nightly (no opt-in)", async () => {
+    // The functional nightly (no E2E_NIGHTLY_COVERAGE) must still deploy the
+    // shipped Konflux build via {{inherit}} — unchanged from today.
+    delete process.env.E2E_NIGHTLY_COVERAGE;
+    const metadataDir = await createCoverageWorkspace({
+      rolledOut: true,
+      role: "frontend-plugin",
+    });
+    try {
+      const result = await processPluginsForDeployment(
+        config,
+        metadataDir,
+        new Set(["@red-hat-developer-hub/backstage-plugin-theme"]), // in DPDY
+      );
+      assert.strictEqual(
+        result.plugins![0].package,
+        "oci://registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-theme:{{inherit}}",
+        "functional nightly must keep DPDY plugins on {{inherit}}",
+      );
+    } finally {
+      await fs.remove(path.resolve(metadataDir, ".."));
+    }
+  });
+});