feat: Docker-free in-process smoke harness for backend dynamic plugins

.github/workflows/native-smoke.yaml

@@ -0,0 +1,111 @@
+# Copyright Red Hat, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+
+# Run the Docker-free, in-process smoke harness: validates that a dynamic backend
+# plugin installs (via the install-dynamic-plugins CLI) and boots (via startTestBackend)
+# with no container and no cluster.
+#
+# - pull_request: validates the harness itself when smoke-tests-native/ changes.
+# - workflow_dispatch: run on demand against any plugin ref.
+name: Native Smoke Harness
+
+on:
+  pull_request:
+    branches:
+      - main
+      - "release-*"
+    paths:
+      - "smoke-tests-native/**"
+      - ".github/workflows/native-smoke.yaml"
+  workflow_dispatch:
+    inputs:
+      plugin_ref:
+        description: >-
+          OCI package ref to validate (oci://…:tag!name). Leave empty to use the
+          known-good pure-backend default (env.DEFAULT_PLUGIN_REF below).
+          Catalog-extending modules are not supported yet.
+        type: string
+        required: false
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.number || github.ref }}
+  cancel-in-progress: true
+
+permissions:
+  contents: read
+  packages: read
+
+env:
+  # Single source of truth for the plugin validated on pull_request and when the
+  # dispatch input is left empty.
+  DEFAULT_PLUGIN_REF: "oci://ghcr.io/redhat-developer/rhdh-plugin-export-overlays/roadiehq-scaffolder-backend-module-http-request:bs_1.49.4__5.6.0!roadiehq-scaffolder-backend-module-http-request"
+
+jobs:
+  smoke:
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    defaults:
+      run:
+        working-directory: smoke-tests-native
+    steps:
+      - name: Checkout
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        with:
+          persist-credentials: false
+
+      - name: Enable Corepack
+        # Before setup-node so its yarn-cache detection resolves Yarn 4 via corepack.
+        run: corepack enable
+
+      - name: Setup Node.js
+        uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
+        with:
+          node-version: 24
+          cache: yarn
+          cache-dependency-path: smoke-tests-native/yarn.lock
+
+      - name: Install skopeo
+        # The install-dynamic-plugins CLI shells out to skopeo to pull OCI plugins.
+        run: sudo apt-get update && sudo apt-get install -y skopeo
+
+      - name: Log in to ghcr.io
+        # Public plugin images pull anonymously; authenticating only avoids rate limits.
+        # Non-fatal: fork PRs get a restricted token, and anonymous pulls still work.
+        continue-on-error: true
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          GH_ACTOR: ${{ github.actor }}
+        run: echo "$GH_TOKEN" | skopeo login ghcr.io -u "$GH_ACTOR" --password-stdin
+
+      - name: Install dependencies
+        run: yarn install --immutable
+
+      - name: Resolve plugin ref
+        # Dispatch input wins; otherwise the default. Encapsulated in env vars so the
+        # value never lands in a shell script via ${{ }} interpolation.
+        env:
+          INPUT_REF: ${{ inputs.plugin_ref }}
+        run: |
+          REF="${INPUT_REF:-$DEFAULT_PLUGIN_REF}"
+          printf 'plugins:\n  - package: %s\n' "$REF" > dp.yaml
+          echo "dynamic-plugins.yaml:"
+          cat dp.yaml
+
+      - name: Run native smoke harness
+        # yarn smoke builds with esbuild then runs `node dist/native-smoke.mjs`.
+        # Exits non-zero on any failure, so the job fails on a bad plugin.
+        run: yarn smoke --dynamic-plugins dp.yaml --out results.json
+
+      - name: Show result
+        if: always()
+        run: cat results.json || echo "no results.json produced"
+
+      - name: Upload results
+        if: always()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        with:
+          name: native-smoke-results
+          path: smoke-tests-native/results.json
+          if-no-files-found: warn

smoke-tests-native/.gitignore

@@ -0,0 +1,7 @@
+node_modules/
+.yarn/
+.pnp.*
+results.json
+dp.yaml
+*.log
+dist/

smoke-tests-native/.yarnrc.yml

@@ -0,0 +1,3 @@
+# A real node_modules tree is required so extracted OCI plugins can resolve their
+# @backstage/* peers via patchModuleResolution() — PnP would have no node_modules dir.
+nodeLinker: node-modules

smoke-tests-native/README.md

@@ -0,0 +1,126 @@
+# Native (Docker-free) smoke harness
+
+Validates RHDH dynamic **backend** plugins in-process — install via the published
+`install-dynamic-plugins` CLI, then boot with `startTestBackend` — with **no Docker
+container and no cluster**. About **20x faster** than the per-workspace `docker run rhdh`
+smoke-test it replaces for that scope.
+
+**Tickets:** RHIDP-15075, RHIDP-15076, RHIDP-13530 (epic RHIDP-13501).
+
+## Why
+
+The repo's container smoke-test boots RHDH in Docker (`docker run rhdh …`) just to check a
+plugin loads. An earlier native attempt (PR #2231) was closed because it was 694 lines of
+bespoke OCI parsing and predated the npm CLI. Now that `install-dynamic-plugins` is
+published — and RHDH already uses it in-process in `plugin-dynamic-loading.spec.ts`
+(PR #4967) — that extraction collapses to one CLI call, so the smoke validation runs
+in-process with no container.
+
+## What it does
+
+```
+install CLI (extract OCI → dynamic-plugins-root, run with cwd=root)
+  → discoverPlugins()         # scan install dirs, classify by package.json backstage.role
+  → loadBackendPlugins()      # require() each, assert default BackendFeature
+  → startTestBackend()        # boot core + loaded features in-process (+ rootConfig)
+  → validateFrontendBundle()  # scalprum/remoteEntry present (presence check, not executed)
+  → results.json + exit code
+```
+
+`src/loader.ts` and `src/{module-resolution,plugin-config}.ts` are ported from RHDH
+PR #4967; `discoverPlugins()` replaces RHDH's `loadManifest()` because this CLI version
+lays out one dir per plugin instead of emitting a `manifest.json`.
+
+## What it deliberately does NOT do
+
+It does **not render frontend UI**. `startTestBackend` is backend-only. UI-behaviour tests
+(the 24 overlay `e2e-tests`, which are ~all Playwright `uiHelper`-driven) need a real
+frontend — that is the **NFS / app-next** path (RHIDP-15082), intentionally out of scope.
+
+## Run
+
+Requires Node 24 and Yarn 4 (matching the repo's `versions.json` and the sibling
+`workspaces/*/e2e-tests`), plus registry access to pull the OCI plugin images.
+
+```bash
+yarn install
+
+cat > dp.yaml <<'YAML'
+plugins:
+  - package: oci://ghcr.io/redhat-developer/rhdh-plugin-export-overlays/<plugin>:<tag>!<name>
+YAML
+yarn smoke --dynamic-plugins dp.yaml
+```
+
+`yarn check` runs `tsc --noEmit`. This is a standalone tool dir, not a
+`workspaces/*/e2e-tests` one, so it is outside `e2e-code-quality.yaml` (which only scans
+`workspaces/*/e2e-tests/**`).
+
+### CI
+
+`.github/workflows/native-smoke.yaml` runs the harness two ways:
+
+- **`pull_request`** (paths `smoke-tests-native/**`): validates the harness itself on every
+  change here, against a known-good pure-backend plugin.
+- **`workflow_dispatch`**: Actions → "Native Smoke Harness" → Run workflow, with an optional
+  `plugin_ref` to validate any plugin on demand.
+
+It installs skopeo, builds, runs `yarn smoke`, uploads `results.json`, and fails the job on
+a non-passing plugin.
+
+Exit code `0` = pass; non-zero with `results.json` detailing `fail-load` / `fail-start` /
+`fail-bundle`.
+
+## Best fit (from the 64-workspace analysis, RHIDP-15076)
+
+- **12 pure-backend workspaces** → fully covered here (load + backend start):
+  `3scale, ai-integrations, apiconnect, github-notifications, keycloak,
+  mcp-integrations, pingidentity, scaffolder-backend-module-{kubernetes,regex,servicenow,sonarqube},
+  scaffolder-relation-processor`.
+- **32 smoke-tests** → replace the Docker container with this harness (backend start +
+  frontend bundle/registration check).
+- **24 UI e2e-tests** → NOT this harness; need the NFS/app-next render harness.
+
+## Status of validation
+
+- ✅ Install CLI interface confirmed: `@red-hat-developer-hub/cli-module-install-dynamic-plugins@0.3.0`
+  (`install <dynamic-plugins-root>`), fetchable via `npx`.
+- ✅ Harness logic ported from the **already-green** RHDH nightly test (PR #4967).
+- ✅ Builds clean (esbuild → `dist/native-smoke.mjs`, run with plain `node`); `tsc --noEmit` passes.
+- ✅ `patchModuleResolution()` ported (`src/module-resolution.ts`) so extracted plugins
+  resolve their `@backstage/*` peers against this harness's `node_modules`. Requires a
+  node-modules linker — see `.yarnrc.yml`.
+- ✅ End-to-end run done locally (Node 24) against a real catalog-index plugin: `pass`,
+  backend loaded 1/1, `startTestBackend` booted — see the Benchmark section below.
+
+## Module resolution
+
+Extracted plugins live under a temp dir with no `node_modules` of their own, so their bare
+`@backstage/*` imports must resolve against this harness. `patchModuleResolution()` (ported
+from RHDH PR #4967) extends `Module._nodeModulePaths` to append `HARNESS_NODE_MODULES`
+before any plugin is `require`d. This is why the package uses `nodeLinker: node-modules`
+(`.yarnrc.yml`) rather than Yarn PnP — the patch needs a real `node_modules` directory to
+point at.
+
+## Benchmark: native vs Docker (real run)
+
+Same plugin both ways: `roadiehq-scaffolder-backend-module-http-request`
+(`bs_1.49.4__5.6.0`), from the real catalog index
+`quay.io/rhdh-community/plugin-catalog-index:1.11-bs_1.49.4`. Same minimal app-config
+(sqlite `:memory:` + guest). Node 24. The RHDH base image (`quay.io/rhdh-community/rhdh:next`,
+6.55 GB) was pre-pulled and is excluded from the Docker timing (one-time infra, amortized
+across all workspaces in a CI run).
+
+| Approach | What it does | Wall-clock |
+|----------|--------------|------------|
+| **Native (this harness)** | skopeo pull plugin → load → `startTestBackend` boot | **5 s cold, 3–4 s warm** |
+| **Docker smoke** (`run-workspace-smoke-tests.yaml`) | container start → in-container `install-dynamic-plugins` (pulls same plugin) → full `node packages/backend` boot → `/healthcheck` 200 | **104 s** |
+
+Roughly **20× faster cold, ~25–35× warm.** Both confirm the plugin loads; the Docker run
+additionally boots the entire RHDH backend (that extra work is exactly the overhead the
+in-process approach removes). Note the comparison is per-workspace — the Docker smoke boots
+one container per workspace, which is the unit this harness replaces.
+
+Caveat: the native harness currently boots a minimal backend scoped to the plugin's needs
+(e.g. scaffolder for scaffolder modules). Catalog-extending modules need the catalog core,
+which does not yet boot cleanly standalone — see the coreFeatures note in `src/native-smoke.ts`.

smoke-tests-native/package.json

@@ -0,0 +1,29 @@
+{
+  "name": "native-smoke-tests",
+  "version": "1.0.0",
+  "private": true,
+  "type": "module",
+  "description": "Docker-free, in-process smoke harness for RHDH dynamic backend plugins: install via the published install-dynamic-plugins CLI, then boot with startTestBackend — no container, no cluster.",
+  "engines": {
+    "node": ">=24",
+    "yarn": ">=4"
+  },
+  "packageManager": "yarn@4.12.0",
+  "scripts": {
+    "build": "esbuild src/native-smoke.ts --bundle --platform=node --format=esm --packages=external --outfile=dist/native-smoke.mjs",
+    "smoke": "yarn build && node dist/native-smoke.mjs",
+    "tsc:check": "tsc --noEmit",
+    "check": "yarn tsc:check"
+  },
+  "dependencies": {
+    "@backstage/backend-plugin-api": "1.9.2",
+    "@backstage/backend-test-utils": "1.11.4",
+    "@backstage/plugin-scaffolder-backend": "3.3.0"
+  },
+  "devDependencies": {
+    "@red-hat-developer-hub/cli-module-install-dynamic-plugins": "0.3.0",
+    "@types/node": "24.13.2",
+    "esbuild": "0.28.1",
+    "typescript": "6.0.3"
+  }
+}