Skip to content

Commit 5eb2451

Browse files
gustavoliraclaude
gustavolira
and
claude
committed
feat: Docker-free in-process smoke harness for backend dynamic plugins
Adds smoke-tests-native/: validates RHDH dynamic backend plugins in-process via the published install-dynamic-plugins CLI + startTestBackend — no Docker, no cluster (~20x faster than the per-workspace docker run smoke for that scope). Runs in a dedicated native-smoke.yaml workflow (pull_request + workflow_dispatch). Loader, module-resolution and plugin-config are ported from RHDH PR #4967. Scope: backend non-catalog plugins. Catalog-extending modules and frontend stay on the Docker smoke (catalog wiring + NFS are follow-ups). Additive — removes nothing. Tickets: RHIDP-15075, RHIDP-15076, RHIDP-13530 (epic RHIDP-13501). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 1dce107 commit 5eb2451

11 files changed

Lines changed: 10357 additions & 0 deletions

File tree

‎.github/workflows/native-smoke.yaml‎

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
# Copyright Red Hat, Inc.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
6+
# Run the Docker-free, in-process smoke harness: validates that a dynamic backend
7+
# plugin installs (via the install-dynamic-plugins CLI) and boots (via startTestBackend)
8+
# with no container and no cluster.
9+
#
10+
# - pull_request: validates the harness itself when smoke-tests-native/ changes.
11+
# - workflow_dispatch: run on demand against any plugin ref.
12+
name: Native Smoke Harness
13+
14+
on:
15+
pull_request:
16+
branches:
17+
- main
18+
- "release-*"
19+
paths:
20+
- "smoke-tests-native/**"
21+
- ".github/workflows/native-smoke.yaml"
22+
workflow_dispatch:
23+
inputs:
24+
plugin_ref:
25+
description: >-
26+
OCI package ref to validate (oci://…:tag!name). Defaults to a known-good
27+
pure-backend plugin. Catalog-extending modules are not supported yet.
28+
type: string
29+
required: false
30+
default: "oci://ghcr.io/redhat-developer/rhdh-plugin-export-overlays/roadiehq-scaffolder-backend-module-http-request:bs_1.49.4__5.6.0!roadiehq-scaffolder-backend-module-http-request"
31+
32+
concurrency:
33+
group: ${{ github.workflow }}-${{ github.event.number || github.ref }}
34+
cancel-in-progress: true
35+
36+
permissions:
37+
contents: read
38+
packages: read
39+
40+
env:
41+
# Default plugin validated on pull_request; overridable via the dispatch input.
42+
DEFAULT_PLUGIN_REF: "oci://ghcr.io/redhat-developer/rhdh-plugin-export-overlays/roadiehq-scaffolder-backend-module-http-request:bs_1.49.4__5.6.0!roadiehq-scaffolder-backend-module-http-request"
43+
44+
jobs:
45+
smoke:
46+
runs-on: ubuntu-latest
47+
timeout-minutes: 15
48+
defaults:
49+
run:
50+
working-directory: smoke-tests-native
51+
steps:
52+
- name: Checkout
53+
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
54+
with:
55+
persist-credentials: false
56+
57+
- name: Setup Node.js
58+
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
59+
with:
60+
node-version: 24
61+
62+
- name: Enable Corepack
63+
run: corepack enable
64+
65+
- name: Install skopeo
66+
# The install-dynamic-plugins CLI shells out to skopeo to pull OCI plugins.
67+
run: sudo apt-get update && sudo apt-get install -y skopeo
68+
69+
- name: Log in to ghcr.io
70+
# Public plugin images pull anonymously; authenticating only avoids rate limits.
71+
# Non-fatal: fork PRs get a restricted token, and anonymous pulls still work.
72+
continue-on-error: true
73+
env:
74+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
75+
GH_ACTOR: ${{ github.actor }}
76+
run: echo "$GH_TOKEN" | skopeo login ghcr.io -u "$GH_ACTOR" --password-stdin
77+
78+
- name: Install dependencies
79+
run: yarn install --immutable
80+
81+
- name: Resolve plugin ref
82+
# Dispatch input wins; otherwise the default. Encapsulated in env vars so the
83+
# value never lands in a shell script via ${{ }} interpolation.
84+
env:
85+
INPUT_REF: ${{ inputs.plugin_ref }}
86+
run: |
87+
REF="${INPUT_REF:-$DEFAULT_PLUGIN_REF}"
88+
printf 'plugins:\n - package: %s\n' "$REF" > dp.yaml
89+
echo "dynamic-plugins.yaml:"
90+
cat dp.yaml
91+
92+
- name: Run native smoke harness
93+
# yarn smoke builds with esbuild then runs `node dist/native-smoke.mjs`.
94+
# Exits non-zero on any failure, so the job fails on a bad plugin.
95+
run: yarn smoke --dynamic-plugins dp.yaml --out results.json
96+
97+
- name: Show result
98+
if: always()
99+
run: cat results.json || echo "no results.json produced"
100+
101+
- name: Upload results
102+
if: always()
103+
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
104+
with:
105+
name: native-smoke-results
106+
path: smoke-tests-native/results.json
107+
if-no-files-found: warn

‎smoke-tests-native/.gitignore‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
node_modules/
2+
.yarn/
3+
.pnp.*
4+
results.json
5+
dp.yaml
6+
*.log
7+
dist/

‎smoke-tests-native/.yarnrc.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# A real node_modules tree is required so extracted OCI plugins can resolve their
2+
# @backstage/* peers via patchModuleResolution() — PnP would have no node_modules dir.
3+
nodeLinker: node-modules

‎smoke-tests-native/README.md‎

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
# Native (Docker-free) smoke harness
2+
3+
Validates RHDH dynamic **backend** plugins in-process — install via the published
4+
`install-dynamic-plugins` CLI, then boot with `startTestBackend` — with **no Docker
5+
container and no cluster**. About **20x faster** than the per-workspace `docker run rhdh`
6+
smoke-test it replaces for that scope.
7+
8+
**Tickets:** RHIDP-15075, RHIDP-15076, RHIDP-13530 (epic RHIDP-13501).
9+
10+
## Why
11+
12+
The repo's container smoke-test boots RHDH in Docker (`docker run rhdh …`) just to check a
13+
plugin loads. An earlier native attempt (PR #2231) was closed because it was 694 lines of
14+
bespoke OCI parsing and predated the npm CLI. Now that `install-dynamic-plugins` is
15+
published — and RHDH already uses it in-process in `plugin-dynamic-loading.spec.ts`
16+
(PR #4967) — that extraction collapses to one CLI call, so the smoke validation runs
17+
in-process with no container.
18+
19+
## What it does
20+
21+
```
22+
install CLI (extract OCI → dynamic-plugins-root, run with cwd=root)
23+
→ discoverPlugins() # scan install dirs, classify by package.json backstage.role
24+
→ loadBackendPlugins() # require() each, assert default BackendFeature
25+
→ startTestBackend() # boot core + loaded features in-process (+ rootConfig)
26+
→ validateFrontendBundle() # scalprum/remoteEntry present (presence check, not executed)
27+
→ results.json + exit code
28+
```
29+
30+
`src/loader.ts` and `src/{module-resolution,plugin-config}.ts` are ported from RHDH
31+
PR #4967; `discoverPlugins()` replaces RHDH's `loadManifest()` because this CLI version
32+
lays out one dir per plugin instead of emitting a `manifest.json`.
33+
34+
## What it deliberately does NOT do
35+
36+
It does **not render frontend UI**. `startTestBackend` is backend-only. UI-behaviour tests
37+
(the 24 overlay `e2e-tests`, which are ~all Playwright `uiHelper`-driven) need a real
38+
frontend — that is the **NFS / app-next** path (RHIDP-15082), intentionally out of scope.
39+
40+
## Run
41+
42+
Requires Node 24 and Yarn 4 (matching the repo's `versions.json` and the sibling
43+
`workspaces/*/e2e-tests`), plus registry access to pull the OCI plugin images.
44+
45+
```bash
46+
yarn install
47+
48+
cat > dp.yaml <<'YAML'
49+
plugins:
50+
- package: oci://ghcr.io/redhat-developer/rhdh-plugin-export-overlays/<plugin>:<tag>!<name>
51+
YAML
52+
yarn smoke --dynamic-plugins dp.yaml
53+
```
54+
55+
`yarn check` runs `tsc --noEmit`. This is a standalone tool dir, not a
56+
`workspaces/*/e2e-tests` one, so it is outside `e2e-code-quality.yaml` (which only scans
57+
`workspaces/*/e2e-tests/**`).
58+
59+
### CI
60+
61+
`.github/workflows/native-smoke.yaml` runs the harness two ways:
62+
63+
- **`pull_request`** (paths `smoke-tests-native/**`): validates the harness itself on every
64+
change here, against a known-good pure-backend plugin.
65+
- **`workflow_dispatch`**: Actions → "Native Smoke Harness" → Run workflow, with an optional
66+
`plugin_ref` to validate any plugin on demand.
67+
68+
It installs skopeo, builds, runs `yarn smoke`, uploads `results.json`, and fails the job on
69+
a non-passing plugin.
70+
71+
Exit code `0` = pass; non-zero with `results.json` detailing `fail-load` / `fail-start` /
72+
`fail-bundle`.
73+
74+
## Best fit (from the 64-workspace analysis, RHIDP-15076)
75+
76+
- **12 pure-backend workspaces** → fully covered here (load + backend start):
77+
`3scale, ai-integrations, apiconnect, github-notifications, keycloak,
78+
mcp-integrations, pingidentity, scaffolder-backend-module-{kubernetes,regex,servicenow,sonarqube},
79+
scaffolder-relation-processor`.
80+
- **32 smoke-tests** → replace the Docker container with this harness (backend start +
81+
frontend bundle/registration check).
82+
- **24 UI e2e-tests** → NOT this harness; need the NFS/app-next render harness.
83+
84+
## Status of validation
85+
86+
- ✅ Install CLI interface confirmed: `@red-hat-developer-hub/cli-module-install-dynamic-plugins@0.3.0`
87+
(`install <dynamic-plugins-root>`), fetchable via `npx`.
88+
- ✅ Harness logic ported from the **already-green** RHDH nightly test (PR #4967).
89+
- ✅ Builds clean (esbuild → `dist/native-smoke.mjs`, run with plain `node`); `tsc --noEmit` passes.
90+
- ✅ `patchModuleResolution()` ported (`src/module-resolution.ts`) so extracted plugins
91+
resolve their `@backstage/*` peers against this harness's `node_modules`. Requires a
92+
node-modules linker — see `.yarnrc.yml`.
93+
- ✅ End-to-end run done locally (Node 24) against a real catalog-index plugin: `pass`,
94+
backend loaded 1/1, `startTestBackend` booted — see the Benchmark section below.
95+
96+
## Module resolution
97+
98+
Extracted plugins live under a temp dir with no `node_modules` of their own, so their bare
99+
`@backstage/*` imports must resolve against this harness. `patchModuleResolution()` (ported
100+
from RHDH PR #4967) extends `Module._nodeModulePaths` to append `HARNESS_NODE_MODULES`
101+
before any plugin is `require`d. This is why the package uses `nodeLinker: node-modules`
102+
(`.yarnrc.yml`) rather than Yarn PnP — the patch needs a real `node_modules` directory to
103+
point at.
104+
105+
## Benchmark: native vs Docker (real run)
106+
107+
Same plugin both ways: `roadiehq-scaffolder-backend-module-http-request`
108+
(`bs_1.49.4__5.6.0`), from the real catalog index
109+
`quay.io/rhdh-community/plugin-catalog-index:1.11-bs_1.49.4`. Same minimal app-config
110+
(sqlite `:memory:` + guest). Node 24. The RHDH base image (`quay.io/rhdh-community/rhdh:next`,
111+
6.55 GB) was pre-pulled and is excluded from the Docker timing (one-time infra, amortized
112+
across all workspaces in a CI run).
113+
114+
| Approach | What it does | Wall-clock |
115+
|----------|--------------|------------|
116+
| **Native (this harness)** | skopeo pull plugin → load → `startTestBackend` boot | **5 s cold, 3–4 s warm** |
117+
| **Docker smoke** (`run-workspace-smoke-tests.yaml`) | container start → in-container `install-dynamic-plugins` (pulls same plugin) → full `node packages/backend` boot → `/healthcheck` 200 | **104 s** |
118+
119+
Roughly **20× faster cold, ~25–35× warm.** Both confirm the plugin loads; the Docker run
120+
additionally boots the entire RHDH backend (that extra work is exactly the overhead the
121+
in-process approach removes). Note the comparison is per-workspace — the Docker smoke boots
122+
one container per workspace, which is the unit this harness replaces.
123+
124+
Caveat: the native harness currently boots a minimal backend scoped to the plugin's needs
125+
(e.g. scaffolder for scaffolder modules). Catalog-extending modules need the catalog core,
126+
which does not yet boot cleanly standalone — see the coreFeatures note in `src/native-smoke.ts`.

‎smoke-tests-native/package.json‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
{
2+
"name": "native-smoke-tests",
3+
"version": "1.0.0",
4+
"private": true,
5+
"type": "module",
6+
"description": "Docker-free, in-process smoke harness for RHDH dynamic backend plugins: install via the published install-dynamic-plugins CLI, then boot with startTestBackend — no container, no cluster.",
7+
"engines": {
8+
"node": ">=24",
9+
"yarn": ">=3"
10+
},
11+
"packageManager": "yarn@4.12.0",
12+
"scripts": {
13+
"build": "esbuild src/native-smoke.ts --bundle --platform=node --format=esm --packages=external --outfile=dist/native-smoke.mjs",
14+
"smoke": "yarn build && node dist/native-smoke.mjs",
15+
"tsc:check": "tsc --noEmit",
16+
"check": "yarn tsc:check"
17+
},
18+
"dependencies": {
19+
"@backstage/backend-plugin-api": "1.9.2",
20+
"@backstage/backend-test-utils": "1.11.4",
21+
"@backstage/plugin-scaffolder-backend": "3.3.0"
22+
},
23+
"devDependencies": {
24+
"@red-hat-developer-hub/cli-module-install-dynamic-plugins": "0.3.0",
25+
"@types/node": "24.13.2",
26+
"esbuild": "0.24.2",
27+
"typescript": "6.0.2"
28+
}
29+
}

0 commit comments

Comments
 (0)