diff --git a/.claude/skills/test-placement/SKILL.md b/.claude/skills/test-placement/SKILL.md new file mode 100644 index 0000000000..ce40d34f25 --- /dev/null +++ b/.claude/skills/test-placement/SKILL.md @@ -0,0 +1,118 @@ +--- +name: test-placement +description: >- + Propose where to test a change in the RHDH dynamic-plugin ecosystem: which + repo (rhdh-plugins, rhdh-plugin-export-overlays, rhdh), which test layer + (unit, integration, component, cluster-free E2E, cluster E2E), where the test + lives, and how to create it. Use when a developer asks where to test + something, whether a test needs a cluster, or which repo a test belongs in +--- +# Test Placement Advisor + +Given the context of a change, bug, or new feature in the RHDH dynamic-plugin ecosystem, propose **where** it should be tested: which repository, which test layer, where the test lives, and how to create it. The guiding rule: **pick the cheapest environment that can actually catch the bug — most plugin validation does not need a cluster, and an increasing part doesn't need Docker either.** + +Conventions in this skill: paths are prefixed with the repo they live in — `rhdh:`, `overlays:` (= rhdh-plugin-export-overlays), `plugins:` (= rhdh-plugins). Some harnesses referenced here are **still in review** (see References for PR status). Before recommending a harness or script, verify its path exists on the target repo's `main`; if it doesn't, tell the developer it is pending in the corresponding PR instead of asserting it exists. + +## When to Use + +- A developer asks "where should I test this?", "should this be an e2e test?", "does this need a cluster?", or "which repo does this test belong in?" +- A new plugin, plugin version bump, or plugin config change needs test coverage. +- A bug escaped to a cluster e2e run and the team wants a cheaper regression test. +- Reviewing a PR that adds a test at the wrong layer (e.g. a cluster e2e for pure UI logic). + +## Step 1 — Gather context (ask if missing) + +Before recommending, establish: + +1. **What is being validated?** Plugin logic/UI component · packaging/published artifact · plugin loading/rendering inside RHDH · platform behavior (Helm/Operator/ingress/auth). +2. **Where did the change happen?** Plugin source (`rhdh-plugins` or another source repo) · packaging metadata (`rhdh-plugin-export-overlays`) · the RHDH app itself (`rhdh`). +3. **Does verifying it require a rendered UI?** (clicks, headings, navigation) +4. **Does it require real external services?** (GitHub, Keycloak, LDAP, managed DBs) +5. **Does it require cluster infrastructure?** (ConfigMap reload, routes, operator, pod logs, port-forward) + +## Step 2 — Decision table + +| The dev wants to verify… | Repo | Test type / harness | Cluster? | Docker? | +| --- | --- | --- | --- | --- | +| Plugin logic / components / API | `rhdh-plugins` (or the plugin's source repo) | unit + component tests, dev app (`yarn start`) | no | no | +| The plugin still builds as a dynamic plugin | `rhdh-plugins` | `npx @red-hat-developer-hub/cli plugin export` | no | no | +| The **published OCI artifact** installs and the **backend plugin boots** | `rhdh-plugin-export-overlays` | native smoke harness (`overlays: smoke-tests-native/`) | no | **no** | +| All plugins of a **workspace** boot together | `rhdh-plugin-export-overlays` | native smoke with a `dynamic-plugins.yaml` listing the workspace's `oci://` refs | no | **no** | +| A frontend artifact ships its bundle (`dist-scalprum/`, `plugin-manifest.json`) | `rhdh-plugin-export-overlays` | native smoke presence check | no | **no** | +| The plugin **loads inside the real RHDH app** and the **UI renders** | `rhdh` | cluster-free E2E harness (`rhdh: e2e-tests/playwright.legacy-local.config.ts`, script `e2e:legacy-local`) | no | no | +| Every plugin in the **catalog index** is sane | `rhdh` | catalog-index plugin sanity check (nightly) | no | no | +| RHDH **container** behavior (image entrypoint, install script inside the image) | `rhdh-plugin-export-overlays` (artifacts) / any (manual) | Docker smoke (`overlays: smoke-tests/` + workflow `run-workspace-smoke-tests.yaml`) · manual: [rhdh-local](https://github.com/redhat-developer/rhdh-local) | no | yes | +| Helm chart / Operator / ingress / real Keycloak / RBAC on OCP | `rhdh` e2e or `overlays: workspaces/*/e2e-tests` | cluster e2e | **yes** | n/a (cluster) | + +Rule of thumb: **source bugs → rhdh-plugins · artifact bugs → overlays · integration/render bugs → rhdh · platform bugs → cluster e2e.** If a bug is catchable in more than one place, test it in the cheapest one and don't duplicate downstream. + +## Step 3 — Layer ladder (cheapest that catches the bug wins) + +| Layer | Scope | Tooling | Cluster | Typical time | +| --- | --- | --- | --- | --- | +| **L1** Unit | Pure functions / logic | Jest/Vitest | no | ms–s | +| **L2** Integration | Backend module + plugin API, mocked external deps | `startTestBackend` + supertest | no | s | +| **L3** Component | React component/page with a test harness | RTL + dev server | no | s–min | +| **L4a** E2E cluster-free | Full app, no managed infra | Playwright + local harness | no | min | +| **L4b** E2E full | Real OCP/K8s, managed DBs, real IdPs | Playwright + cluster | **yes** | min–h | + +The full per-spec classification of the RHDH e2e suite lives in `rhdh: docs/e2e-tests/layer-migration-matrix.md` (see References) — consult it before adding or migrating an e2e spec. + +## Step 4 — How to create the test (per placement) + +### `rhdh-plugins` (or other source repo) — plugin correctness + +- **Where:** `plugins: workspaces//plugins//src/**` next to the code, following that workspace's existing test setup (Jest/RTL; MSW for API mocks). +- **How:** copy the pattern of a neighboring `*.test.ts(x)`; run with the workspace's `yarn test`. Manual verification via the workspace dev app (`yarn start`). +- Also validate packaging when the plugin's build config changed: `npx @red-hat-developer-hub/cli plugin export` must produce `dist-dynamic/` (+ `dist-scalprum/` for frontend). +- **Cannot do here:** validate the *published* OCI artifact, or integration with RHDH's app shell/shared deps. + +### `rhdh-plugin-export-overlays` — the published artifact + +- **Where:** the native smoke harness, `overlays: smoke-tests-native/` (in review — see References). It installs plugins from OCI with the real `install-dynamic-plugins` CLI and boots backend plugins via `startTestBackend`. +- **How:** write a `dynamic-plugins.yaml` listing the `oci://` artifact refs to validate (a workspace's refs are the `spec.dynamicArtifact` values in `overlays: workspaces//metadata/*.yaml`), then run `yarn smoke --dynamic-plugins [--out results.json]`. Exit code 0 = pass; non-zero writes `results.json` detailing `fail-load` / `fail-start`. CI: `overlays: .github/workflows/native-smoke.yaml`. +- **Known gaps (don't fight them):** catalog-extending backend modules don't boot in a minimal `startTestBackend` (upstream `catalog-backend` registration issue — stay on the Docker smoke until fixed); plugins whose `dynamicArtifact` is a local `./dynamic-plugins/dist/...` path ship inside the RHDH image and have no OCI artifact to validate here. +- **Never add UI-render tests here** — this repo has no app to render into. Delegate rendering to the `rhdh` cluster-free harness. +- The Docker smoke it replaces (backend scope) lives at `overlays: smoke-tests/` + workflow `run-workspace-smoke-tests.yaml`. +- `overlays: workspaces/*/e2e-tests` (cluster, Playwright + `e2e-test-utils`) exists for plugins whose value *is* cluster integration (topology, tekton, argocd…). Don't add one for UI-only behavior. + +### `rhdh` — the real app + +**L4a cluster-free harness** (in review — see References; config `rhdh: e2e-tests/playwright.legacy-local.config.ts`, docs `rhdh: docs/e2e-tests/local-e2e-harness.md`) — the only cheap place a frontend dynamic plugin can be *rendered*. To enable a spec/test: + +1. Plugin not yet installed by the harness? Add its OCI entry to `rhdh: e2e-tests/local-harness/dynamic-plugins.yaml` (tags on `ghcr.io/redhat-developer/rhdh-plugin-export-overlays/`). If its mount-point config is not in the repo's static `app-config.dynamic-plugins.yaml`, attach the plugin's **canonical `pluginConfig`** (source of truth: `plugins: workspaces//plugins//app-config.dynamic.yaml`) — the harness loads the generated `dynamic-plugins-root/app-config.dynamic-plugins.yaml` last, exactly like the production container. +2. Config that only exists in CI config maps (`rhdh: .ci/pipelines/resources/config_map/*`)? Mirror just the needed keys into `rhdh: app-config.local-e2e.yaml` (objects deep-merge; arrays replace). +3. Tag the test `{ tag: "@cluster-free" }` and add its spec file to the config's `testMatch` allowlist. +4. Repopulate + validate: `./e2e-tests/local-harness/populate.sh`, then `yarn --cwd e2e-tests e2e:legacy-local` (CI runs this in ~4 min). Requires skopeo — preinstalled in CI; on macOS `brew install skopeo`. + +**L4b cluster e2e** (`rhdh: e2e-tests/playwright/e2e/**`) — only when the subject *is* cluster/platform behavior or a real external service. Requirements: `component` annotation in `beforeAll` (see the repo's `ci-e2e-testing` rule), correct config map choice (RBAC vs non-RBAC), project registration in `e2e-tests/playwright/projects.json` if a new project is needed. + +**L3 component tests** — page-level RTL compositions, pattern proposed under RHIDP-13235 in [rhdh#4864](https://github.com/redhat-developer/rhdh/pull/4864) (closed, not merged). Check for merged examples before recommending a concrete template; prefer L3 over L4a when no dynamic-plugin loading is involved. + +**Catalog-index-wide sanity** (nightly, in review — see References) — nothing to write per plugin; it sweeps the whole index. + +## Not possible today (researched — don't burn time) + +- **Rendering a frontend dynamic plugin without an RHDH app** — the artifact is a legacy-frontend Scalprum bundle; no standalone host exists, and building one means maintaining a version-coupled "mini RHDH". +- **`@backstage/frontend-dynamic-feature-loader` for current plugins** — targets the new frontend system (alpha); our exported plugins are legacy-system bundles. Revisit when app-next matures (RHIDP-15082). +- **Catalog-extending modules in a minimal `startTestBackend`** — upstream `catalog-backend` issue (see overlays harness notes). + +## Output format + +Answer with a concrete recommendation: + +- **Repo:** which of the three (or the plugin's own source repo). +- **Layer / harness:** L1–L4b + the specific harness or suite. +- **Location:** the directory/file where the test goes. +- **Scaffolding:** the minimal steps or files to create, referencing an existing neighbor as template. +- **Why not elsewhere:** one line on the layers you rejected (especially if the dev proposed a more expensive one). +- **Cost:** rough feedback time (seconds / ~4 min cluster-free / cluster job). + +## References (status as of 2026-07-02 — verify before citing as merged) + +- Epic **RHIDP-13501** (E2E Test Optimization) — the per-repo responsibility split lives in the epic's comments and its Jira attachment `rhdh-dynamic-plugin-testing-guideline.md` (Jira-only; if unreachable, the decision table above is the summary). +- Layer matrix: — in review in [rhdh#5044](https://github.com/redhat-developer/rhdh/pull/5044) (RHIDP-15076). +- Cluster-free harness + docs: in review in [rhdh#5005](https://github.com/redhat-developer/rhdh/pull/5005) (RHIDP-15075). +- Overlays native smoke: in review in [overlays#2714](https://github.com/redhat-developer/rhdh-plugin-export-overlays/pull/2714). +- Catalog-index sanity check: in review in [rhdh#4967](https://github.com/redhat-developer/rhdh/pull/4967). +- L3 pattern proposal: [rhdh#4864](https://github.com/redhat-developer/rhdh/pull/4864) (closed) under RHIDP-13235. diff --git a/.cursor/skills/test-placement/SKILL.md b/.cursor/skills/test-placement/SKILL.md new file mode 100644 index 0000000000..ae76c96f60 --- /dev/null +++ b/.cursor/skills/test-placement/SKILL.md @@ -0,0 +1,113 @@ +--- +name: test-placement +description: 'Propose where to test a change in the RHDH dynamic-plugin ecosystem: which repo (rhdh-plugins, rhdh-plugin-export-overlays, rhdh), which test layer (unit, integration, component, cluster-free E2E, cluster E2E), where the test lives, and how to create it. Use when a developer asks where to test something, whether a test needs a cluster, or which repo a test belongs in' +--- +# Test Placement Advisor + +Given the context of a change, bug, or new feature in the RHDH dynamic-plugin ecosystem, propose **where** it should be tested: which repository, which test layer, where the test lives, and how to create it. The guiding rule: **pick the cheapest environment that can actually catch the bug — most plugin validation does not need a cluster, and an increasing part doesn't need Docker either.** + +Conventions in this skill: paths are prefixed with the repo they live in — `rhdh:`, `overlays:` (= rhdh-plugin-export-overlays), `plugins:` (= rhdh-plugins). Some harnesses referenced here are **still in review** (see References for PR status). Before recommending a harness or script, verify its path exists on the target repo's `main`; if it doesn't, tell the developer it is pending in the corresponding PR instead of asserting it exists. + +## When to Use + +- A developer asks "where should I test this?", "should this be an e2e test?", "does this need a cluster?", or "which repo does this test belong in?" +- A new plugin, plugin version bump, or plugin config change needs test coverage. +- A bug escaped to a cluster e2e run and the team wants a cheaper regression test. +- Reviewing a PR that adds a test at the wrong layer (e.g. a cluster e2e for pure UI logic). + +## Step 1 — Gather context (ask if missing) + +Before recommending, establish: + +1. **What is being validated?** Plugin logic/UI component · packaging/published artifact · plugin loading/rendering inside RHDH · platform behavior (Helm/Operator/ingress/auth). +2. **Where did the change happen?** Plugin source (`rhdh-plugins` or another source repo) · packaging metadata (`rhdh-plugin-export-overlays`) · the RHDH app itself (`rhdh`). +3. **Does verifying it require a rendered UI?** (clicks, headings, navigation) +4. **Does it require real external services?** (GitHub, Keycloak, LDAP, managed DBs) +5. **Does it require cluster infrastructure?** (ConfigMap reload, routes, operator, pod logs, port-forward) + +## Step 2 — Decision table + +| The dev wants to verify… | Repo | Test type / harness | Cluster? | Docker? | +| --- | --- | --- | --- | --- | +| Plugin logic / components / API | `rhdh-plugins` (or the plugin's source repo) | unit + component tests, dev app (`yarn start`) | no | no | +| The plugin still builds as a dynamic plugin | `rhdh-plugins` | `npx @red-hat-developer-hub/cli plugin export` | no | no | +| The **published OCI artifact** installs and the **backend plugin boots** | `rhdh-plugin-export-overlays` | native smoke harness (`overlays: smoke-tests-native/`) | no | **no** | +| All plugins of a **workspace** boot together | `rhdh-plugin-export-overlays` | native smoke with a `dynamic-plugins.yaml` listing the workspace's `oci://` refs | no | **no** | +| A frontend artifact ships its bundle (`dist-scalprum/`, `plugin-manifest.json`) | `rhdh-plugin-export-overlays` | native smoke presence check | no | **no** | +| The plugin **loads inside the real RHDH app** and the **UI renders** | `rhdh` | cluster-free E2E harness (`rhdh: e2e-tests/playwright.legacy-local.config.ts`, script `e2e:legacy-local`) | no | no | +| Every plugin in the **catalog index** is sane | `rhdh` | catalog-index plugin sanity check (nightly) | no | no | +| RHDH **container** behavior (image entrypoint, install script inside the image) | `rhdh-plugin-export-overlays` (artifacts) / any (manual) | Docker smoke (`overlays: smoke-tests/` + workflow `run-workspace-smoke-tests.yaml`) · manual: [rhdh-local](https://github.com/redhat-developer/rhdh-local) | no | yes | +| Helm chart / Operator / ingress / real Keycloak / RBAC on OCP | `rhdh` e2e or `overlays: workspaces/*/e2e-tests` | cluster e2e | **yes** | n/a (cluster) | + +Rule of thumb: **source bugs → rhdh-plugins · artifact bugs → overlays · integration/render bugs → rhdh · platform bugs → cluster e2e.** If a bug is catchable in more than one place, test it in the cheapest one and don't duplicate downstream. + +## Step 3 — Layer ladder (cheapest that catches the bug wins) + +| Layer | Scope | Tooling | Cluster | Typical time | +| --- | --- | --- | --- | --- | +| **L1** Unit | Pure functions / logic | Jest/Vitest | no | ms–s | +| **L2** Integration | Backend module + plugin API, mocked external deps | `startTestBackend` + supertest | no | s | +| **L3** Component | React component/page with a test harness | RTL + dev server | no | s–min | +| **L4a** E2E cluster-free | Full app, no managed infra | Playwright + local harness | no | min | +| **L4b** E2E full | Real OCP/K8s, managed DBs, real IdPs | Playwright + cluster | **yes** | min–h | + +The full per-spec classification of the RHDH e2e suite lives in `rhdh: docs/e2e-tests/layer-migration-matrix.md` (see References) — consult it before adding or migrating an e2e spec. + +## Step 4 — How to create the test (per placement) + +### `rhdh-plugins` (or other source repo) — plugin correctness + +- **Where:** `plugins: workspaces//plugins//src/**` next to the code, following that workspace's existing test setup (Jest/RTL; MSW for API mocks). +- **How:** copy the pattern of a neighboring `*.test.ts(x)`; run with the workspace's `yarn test`. Manual verification via the workspace dev app (`yarn start`). +- Also validate packaging when the plugin's build config changed: `npx @red-hat-developer-hub/cli plugin export` must produce `dist-dynamic/` (+ `dist-scalprum/` for frontend). +- **Cannot do here:** validate the *published* OCI artifact, or integration with RHDH's app shell/shared deps. + +### `rhdh-plugin-export-overlays` — the published artifact + +- **Where:** the native smoke harness, `overlays: smoke-tests-native/` (in review — see References). It installs plugins from OCI with the real `install-dynamic-plugins` CLI and boots backend plugins via `startTestBackend`. +- **How:** write a `dynamic-plugins.yaml` listing the `oci://` artifact refs to validate (a workspace's refs are the `spec.dynamicArtifact` values in `overlays: workspaces//metadata/*.yaml`), then run `yarn smoke --dynamic-plugins [--out results.json]`. Exit code 0 = pass; non-zero writes `results.json` detailing `fail-load` / `fail-start`. CI: `overlays: .github/workflows/native-smoke.yaml`. +- **Known gaps (don't fight them):** catalog-extending backend modules don't boot in a minimal `startTestBackend` (upstream `catalog-backend` registration issue — stay on the Docker smoke until fixed); plugins whose `dynamicArtifact` is a local `./dynamic-plugins/dist/...` path ship inside the RHDH image and have no OCI artifact to validate here. +- **Never add UI-render tests here** — this repo has no app to render into. Delegate rendering to the `rhdh` cluster-free harness. +- The Docker smoke it replaces (backend scope) lives at `overlays: smoke-tests/` + workflow `run-workspace-smoke-tests.yaml`. +- `overlays: workspaces/*/e2e-tests` (cluster, Playwright + `e2e-test-utils`) exists for plugins whose value *is* cluster integration (topology, tekton, argocd…). Don't add one for UI-only behavior. + +### `rhdh` — the real app + +**L4a cluster-free harness** (in review — see References; config `rhdh: e2e-tests/playwright.legacy-local.config.ts`, docs `rhdh: docs/e2e-tests/local-e2e-harness.md`) — the only cheap place a frontend dynamic plugin can be *rendered*. To enable a spec/test: + +1. Plugin not yet installed by the harness? Add its OCI entry to `rhdh: e2e-tests/local-harness/dynamic-plugins.yaml` (tags on `ghcr.io/redhat-developer/rhdh-plugin-export-overlays/`). If its mount-point config is not in the repo's static `app-config.dynamic-plugins.yaml`, attach the plugin's **canonical `pluginConfig`** (source of truth: `plugins: workspaces//plugins//app-config.dynamic.yaml`) — the harness loads the generated `dynamic-plugins-root/app-config.dynamic-plugins.yaml` last, exactly like the production container. +2. Config that only exists in CI config maps (`rhdh: .ci/pipelines/resources/config_map/*`)? Mirror just the needed keys into `rhdh: app-config.local-e2e.yaml` (objects deep-merge; arrays replace). +3. Tag the test `{ tag: "@cluster-free" }` and add its spec file to the config's `testMatch` allowlist. +4. Repopulate + validate: `./e2e-tests/local-harness/populate.sh`, then `yarn --cwd e2e-tests e2e:legacy-local` (CI runs this in ~4 min). Requires skopeo — preinstalled in CI; on macOS `brew install skopeo`. + +**L4b cluster e2e** (`rhdh: e2e-tests/playwright/e2e/**`) — only when the subject *is* cluster/platform behavior or a real external service. Requirements: `component` annotation in `beforeAll` (see the repo's `ci-e2e-testing` rule), correct config map choice (RBAC vs non-RBAC), project registration in `e2e-tests/playwright/projects.json` if a new project is needed. + +**L3 component tests** — page-level RTL compositions, pattern proposed under RHIDP-13235 in [rhdh#4864](https://github.com/redhat-developer/rhdh/pull/4864) (closed, not merged). Check for merged examples before recommending a concrete template; prefer L3 over L4a when no dynamic-plugin loading is involved. + +**Catalog-index-wide sanity** (nightly, in review — see References) — nothing to write per plugin; it sweeps the whole index. + +## Not possible today (researched — don't burn time) + +- **Rendering a frontend dynamic plugin without an RHDH app** — the artifact is a legacy-frontend Scalprum bundle; no standalone host exists, and building one means maintaining a version-coupled "mini RHDH". +- **`@backstage/frontend-dynamic-feature-loader` for current plugins** — targets the new frontend system (alpha); our exported plugins are legacy-system bundles. Revisit when app-next matures (RHIDP-15082). +- **Catalog-extending modules in a minimal `startTestBackend`** — upstream `catalog-backend` issue (see overlays harness notes). + +## Output format + +Answer with a concrete recommendation: + +- **Repo:** which of the three (or the plugin's own source repo). +- **Layer / harness:** L1–L4b + the specific harness or suite. +- **Location:** the directory/file where the test goes. +- **Scaffolding:** the minimal steps or files to create, referencing an existing neighbor as template. +- **Why not elsewhere:** one line on the layers you rejected (especially if the dev proposed a more expensive one). +- **Cost:** rough feedback time (seconds / ~4 min cluster-free / cluster job). + +## References (status as of 2026-07-02 — verify before citing as merged) + +- Epic **RHIDP-13501** (E2E Test Optimization) — the per-repo responsibility split lives in the epic's comments and its Jira attachment `rhdh-dynamic-plugin-testing-guideline.md` (Jira-only; if unreachable, the decision table above is the summary). +- Layer matrix: — in review in [rhdh#5044](https://github.com/redhat-developer/rhdh/pull/5044) (RHIDP-15076). +- Cluster-free harness + docs: in review in [rhdh#5005](https://github.com/redhat-developer/rhdh/pull/5005) (RHIDP-15075). +- Overlays native smoke: in review in [overlays#2714](https://github.com/redhat-developer/rhdh-plugin-export-overlays/pull/2714). +- Catalog-index sanity check: in review in [rhdh#4967](https://github.com/redhat-developer/rhdh/pull/4967). +- L3 pattern proposal: [rhdh#4864](https://github.com/redhat-developer/rhdh/pull/4864) (closed) under RHIDP-13235. diff --git a/.opencode/skill/test-placement/SKILL.md b/.opencode/skill/test-placement/SKILL.md new file mode 100644 index 0000000000..ce40d34f25 --- /dev/null +++ b/.opencode/skill/test-placement/SKILL.md @@ -0,0 +1,118 @@ +--- +name: test-placement +description: >- + Propose where to test a change in the RHDH dynamic-plugin ecosystem: which + repo (rhdh-plugins, rhdh-plugin-export-overlays, rhdh), which test layer + (unit, integration, component, cluster-free E2E, cluster E2E), where the test + lives, and how to create it. Use when a developer asks where to test + something, whether a test needs a cluster, or which repo a test belongs in +--- +# Test Placement Advisor + +Given the context of a change, bug, or new feature in the RHDH dynamic-plugin ecosystem, propose **where** it should be tested: which repository, which test layer, where the test lives, and how to create it. The guiding rule: **pick the cheapest environment that can actually catch the bug — most plugin validation does not need a cluster, and an increasing part doesn't need Docker either.** + +Conventions in this skill: paths are prefixed with the repo they live in — `rhdh:`, `overlays:` (= rhdh-plugin-export-overlays), `plugins:` (= rhdh-plugins). Some harnesses referenced here are **still in review** (see References for PR status). Before recommending a harness or script, verify its path exists on the target repo's `main`; if it doesn't, tell the developer it is pending in the corresponding PR instead of asserting it exists. + +## When to Use + +- A developer asks "where should I test this?", "should this be an e2e test?", "does this need a cluster?", or "which repo does this test belong in?" +- A new plugin, plugin version bump, or plugin config change needs test coverage. +- A bug escaped to a cluster e2e run and the team wants a cheaper regression test. +- Reviewing a PR that adds a test at the wrong layer (e.g. a cluster e2e for pure UI logic). + +## Step 1 — Gather context (ask if missing) + +Before recommending, establish: + +1. **What is being validated?** Plugin logic/UI component · packaging/published artifact · plugin loading/rendering inside RHDH · platform behavior (Helm/Operator/ingress/auth). +2. **Where did the change happen?** Plugin source (`rhdh-plugins` or another source repo) · packaging metadata (`rhdh-plugin-export-overlays`) · the RHDH app itself (`rhdh`). +3. **Does verifying it require a rendered UI?** (clicks, headings, navigation) +4. **Does it require real external services?** (GitHub, Keycloak, LDAP, managed DBs) +5. **Does it require cluster infrastructure?** (ConfigMap reload, routes, operator, pod logs, port-forward) + +## Step 2 — Decision table + +| The dev wants to verify… | Repo | Test type / harness | Cluster? | Docker? | +| --- | --- | --- | --- | --- | +| Plugin logic / components / API | `rhdh-plugins` (or the plugin's source repo) | unit + component tests, dev app (`yarn start`) | no | no | +| The plugin still builds as a dynamic plugin | `rhdh-plugins` | `npx @red-hat-developer-hub/cli plugin export` | no | no | +| The **published OCI artifact** installs and the **backend plugin boots** | `rhdh-plugin-export-overlays` | native smoke harness (`overlays: smoke-tests-native/`) | no | **no** | +| All plugins of a **workspace** boot together | `rhdh-plugin-export-overlays` | native smoke with a `dynamic-plugins.yaml` listing the workspace's `oci://` refs | no | **no** | +| A frontend artifact ships its bundle (`dist-scalprum/`, `plugin-manifest.json`) | `rhdh-plugin-export-overlays` | native smoke presence check | no | **no** | +| The plugin **loads inside the real RHDH app** and the **UI renders** | `rhdh` | cluster-free E2E harness (`rhdh: e2e-tests/playwright.legacy-local.config.ts`, script `e2e:legacy-local`) | no | no | +| Every plugin in the **catalog index** is sane | `rhdh` | catalog-index plugin sanity check (nightly) | no | no | +| RHDH **container** behavior (image entrypoint, install script inside the image) | `rhdh-plugin-export-overlays` (artifacts) / any (manual) | Docker smoke (`overlays: smoke-tests/` + workflow `run-workspace-smoke-tests.yaml`) · manual: [rhdh-local](https://github.com/redhat-developer/rhdh-local) | no | yes | +| Helm chart / Operator / ingress / real Keycloak / RBAC on OCP | `rhdh` e2e or `overlays: workspaces/*/e2e-tests` | cluster e2e | **yes** | n/a (cluster) | + +Rule of thumb: **source bugs → rhdh-plugins · artifact bugs → overlays · integration/render bugs → rhdh · platform bugs → cluster e2e.** If a bug is catchable in more than one place, test it in the cheapest one and don't duplicate downstream. + +## Step 3 — Layer ladder (cheapest that catches the bug wins) + +| Layer | Scope | Tooling | Cluster | Typical time | +| --- | --- | --- | --- | --- | +| **L1** Unit | Pure functions / logic | Jest/Vitest | no | ms–s | +| **L2** Integration | Backend module + plugin API, mocked external deps | `startTestBackend` + supertest | no | s | +| **L3** Component | React component/page with a test harness | RTL + dev server | no | s–min | +| **L4a** E2E cluster-free | Full app, no managed infra | Playwright + local harness | no | min | +| **L4b** E2E full | Real OCP/K8s, managed DBs, real IdPs | Playwright + cluster | **yes** | min–h | + +The full per-spec classification of the RHDH e2e suite lives in `rhdh: docs/e2e-tests/layer-migration-matrix.md` (see References) — consult it before adding or migrating an e2e spec. + +## Step 4 — How to create the test (per placement) + +### `rhdh-plugins` (or other source repo) — plugin correctness + +- **Where:** `plugins: workspaces//plugins//src/**` next to the code, following that workspace's existing test setup (Jest/RTL; MSW for API mocks). +- **How:** copy the pattern of a neighboring `*.test.ts(x)`; run with the workspace's `yarn test`. Manual verification via the workspace dev app (`yarn start`). +- Also validate packaging when the plugin's build config changed: `npx @red-hat-developer-hub/cli plugin export` must produce `dist-dynamic/` (+ `dist-scalprum/` for frontend). +- **Cannot do here:** validate the *published* OCI artifact, or integration with RHDH's app shell/shared deps. + +### `rhdh-plugin-export-overlays` — the published artifact + +- **Where:** the native smoke harness, `overlays: smoke-tests-native/` (in review — see References). It installs plugins from OCI with the real `install-dynamic-plugins` CLI and boots backend plugins via `startTestBackend`. +- **How:** write a `dynamic-plugins.yaml` listing the `oci://` artifact refs to validate (a workspace's refs are the `spec.dynamicArtifact` values in `overlays: workspaces//metadata/*.yaml`), then run `yarn smoke --dynamic-plugins [--out results.json]`. Exit code 0 = pass; non-zero writes `results.json` detailing `fail-load` / `fail-start`. CI: `overlays: .github/workflows/native-smoke.yaml`. +- **Known gaps (don't fight them):** catalog-extending backend modules don't boot in a minimal `startTestBackend` (upstream `catalog-backend` registration issue — stay on the Docker smoke until fixed); plugins whose `dynamicArtifact` is a local `./dynamic-plugins/dist/...` path ship inside the RHDH image and have no OCI artifact to validate here. +- **Never add UI-render tests here** — this repo has no app to render into. Delegate rendering to the `rhdh` cluster-free harness. +- The Docker smoke it replaces (backend scope) lives at `overlays: smoke-tests/` + workflow `run-workspace-smoke-tests.yaml`. +- `overlays: workspaces/*/e2e-tests` (cluster, Playwright + `e2e-test-utils`) exists for plugins whose value *is* cluster integration (topology, tekton, argocd…). Don't add one for UI-only behavior. + +### `rhdh` — the real app + +**L4a cluster-free harness** (in review — see References; config `rhdh: e2e-tests/playwright.legacy-local.config.ts`, docs `rhdh: docs/e2e-tests/local-e2e-harness.md`) — the only cheap place a frontend dynamic plugin can be *rendered*. To enable a spec/test: + +1. Plugin not yet installed by the harness? Add its OCI entry to `rhdh: e2e-tests/local-harness/dynamic-plugins.yaml` (tags on `ghcr.io/redhat-developer/rhdh-plugin-export-overlays/`). If its mount-point config is not in the repo's static `app-config.dynamic-plugins.yaml`, attach the plugin's **canonical `pluginConfig`** (source of truth: `plugins: workspaces//plugins//app-config.dynamic.yaml`) — the harness loads the generated `dynamic-plugins-root/app-config.dynamic-plugins.yaml` last, exactly like the production container. +2. Config that only exists in CI config maps (`rhdh: .ci/pipelines/resources/config_map/*`)? Mirror just the needed keys into `rhdh: app-config.local-e2e.yaml` (objects deep-merge; arrays replace). +3. Tag the test `{ tag: "@cluster-free" }` and add its spec file to the config's `testMatch` allowlist. +4. Repopulate + validate: `./e2e-tests/local-harness/populate.sh`, then `yarn --cwd e2e-tests e2e:legacy-local` (CI runs this in ~4 min). Requires skopeo — preinstalled in CI; on macOS `brew install skopeo`. + +**L4b cluster e2e** (`rhdh: e2e-tests/playwright/e2e/**`) — only when the subject *is* cluster/platform behavior or a real external service. Requirements: `component` annotation in `beforeAll` (see the repo's `ci-e2e-testing` rule), correct config map choice (RBAC vs non-RBAC), project registration in `e2e-tests/playwright/projects.json` if a new project is needed. + +**L3 component tests** — page-level RTL compositions, pattern proposed under RHIDP-13235 in [rhdh#4864](https://github.com/redhat-developer/rhdh/pull/4864) (closed, not merged). Check for merged examples before recommending a concrete template; prefer L3 over L4a when no dynamic-plugin loading is involved. + +**Catalog-index-wide sanity** (nightly, in review — see References) — nothing to write per plugin; it sweeps the whole index. + +## Not possible today (researched — don't burn time) + +- **Rendering a frontend dynamic plugin without an RHDH app** — the artifact is a legacy-frontend Scalprum bundle; no standalone host exists, and building one means maintaining a version-coupled "mini RHDH". +- **`@backstage/frontend-dynamic-feature-loader` for current plugins** — targets the new frontend system (alpha); our exported plugins are legacy-system bundles. Revisit when app-next matures (RHIDP-15082). +- **Catalog-extending modules in a minimal `startTestBackend`** — upstream `catalog-backend` issue (see overlays harness notes). + +## Output format + +Answer with a concrete recommendation: + +- **Repo:** which of the three (or the plugin's own source repo). +- **Layer / harness:** L1–L4b + the specific harness or suite. +- **Location:** the directory/file where the test goes. +- **Scaffolding:** the minimal steps or files to create, referencing an existing neighbor as template. +- **Why not elsewhere:** one line on the layers you rejected (especially if the dev proposed a more expensive one). +- **Cost:** rough feedback time (seconds / ~4 min cluster-free / cluster job). + +## References (status as of 2026-07-02 — verify before citing as merged) + +- Epic **RHIDP-13501** (E2E Test Optimization) — the per-repo responsibility split lives in the epic's comments and its Jira attachment `rhdh-dynamic-plugin-testing-guideline.md` (Jira-only; if unreachable, the decision table above is the summary). +- Layer matrix: — in review in [rhdh#5044](https://github.com/redhat-developer/rhdh/pull/5044) (RHIDP-15076). +- Cluster-free harness + docs: in review in [rhdh#5005](https://github.com/redhat-developer/rhdh/pull/5005) (RHIDP-15075). +- Overlays native smoke: in review in [overlays#2714](https://github.com/redhat-developer/rhdh-plugin-export-overlays/pull/2714). +- Catalog-index sanity check: in review in [rhdh#4967](https://github.com/redhat-developer/rhdh/pull/4967). +- L3 pattern proposal: [rhdh#4864](https://github.com/redhat-developer/rhdh/pull/4864) (closed) under RHIDP-13235. diff --git a/.rulesync/skills/test-placement/SKILL.md b/.rulesync/skills/test-placement/SKILL.md new file mode 100644 index 0000000000..332bc33d23 --- /dev/null +++ b/.rulesync/skills/test-placement/SKILL.md @@ -0,0 +1,120 @@ +--- +name: test-placement +description: >- + Propose where to test a change in the RHDH dynamic-plugin ecosystem: which + repo (rhdh-plugins, rhdh-plugin-export-overlays, rhdh), which test layer + (unit, integration, component, cluster-free E2E, cluster E2E), where the + test lives, and how to create it. Use when a developer asks where to test + something, whether a test needs a cluster, or which repo a test belongs in +targets: + - '*' +--- +# Test Placement Advisor + +Given the context of a change, bug, or new feature in the RHDH dynamic-plugin ecosystem, propose **where** it should be tested: which repository, which test layer, where the test lives, and how to create it. The guiding rule: **pick the cheapest environment that can actually catch the bug — most plugin validation does not need a cluster, and an increasing part doesn't need Docker either.** + +Conventions in this skill: paths are prefixed with the repo they live in — `rhdh:`, `overlays:` (= rhdh-plugin-export-overlays), `plugins:` (= rhdh-plugins). Some harnesses referenced here are **still in review** (see References for PR status). Before recommending a harness or script, verify its path exists on the target repo's `main`; if it doesn't, tell the developer it is pending in the corresponding PR instead of asserting it exists. + +## When to Use + +- A developer asks "where should I test this?", "should this be an e2e test?", "does this need a cluster?", or "which repo does this test belong in?" +- A new plugin, plugin version bump, or plugin config change needs test coverage. +- A bug escaped to a cluster e2e run and the team wants a cheaper regression test. +- Reviewing a PR that adds a test at the wrong layer (e.g. a cluster e2e for pure UI logic). + +## Step 1 — Gather context (ask if missing) + +Before recommending, establish: + +1. **What is being validated?** Plugin logic/UI component · packaging/published artifact · plugin loading/rendering inside RHDH · platform behavior (Helm/Operator/ingress/auth). +2. **Where did the change happen?** Plugin source (`rhdh-plugins` or another source repo) · packaging metadata (`rhdh-plugin-export-overlays`) · the RHDH app itself (`rhdh`). +3. **Does verifying it require a rendered UI?** (clicks, headings, navigation) +4. **Does it require real external services?** (GitHub, Keycloak, LDAP, managed DBs) +5. **Does it require cluster infrastructure?** (ConfigMap reload, routes, operator, pod logs, port-forward) + +## Step 2 — Decision table + +| The dev wants to verify… | Repo | Test type / harness | Cluster? | Docker? | +| --- | --- | --- | --- | --- | +| Plugin logic / components / API | `rhdh-plugins` (or the plugin's source repo) | unit + component tests, dev app (`yarn start`) | no | no | +| The plugin still builds as a dynamic plugin | `rhdh-plugins` | `npx @red-hat-developer-hub/cli plugin export` | no | no | +| The **published OCI artifact** installs and the **backend plugin boots** | `rhdh-plugin-export-overlays` | native smoke harness (`overlays: smoke-tests-native/`) | no | **no** | +| All plugins of a **workspace** boot together | `rhdh-plugin-export-overlays` | native smoke with a `dynamic-plugins.yaml` listing the workspace's `oci://` refs | no | **no** | +| A frontend artifact ships its bundle (`dist-scalprum/`, `plugin-manifest.json`) | `rhdh-plugin-export-overlays` | native smoke presence check | no | **no** | +| The plugin **loads inside the real RHDH app** and the **UI renders** | `rhdh` | cluster-free E2E harness (`rhdh: e2e-tests/playwright.legacy-local.config.ts`, script `e2e:legacy-local`) | no | no | +| Every plugin in the **catalog index** is sane | `rhdh` | catalog-index plugin sanity check (nightly) | no | no | +| RHDH **container** behavior (image entrypoint, install script inside the image) | `rhdh-plugin-export-overlays` (artifacts) / any (manual) | Docker smoke (`overlays: smoke-tests/` + workflow `run-workspace-smoke-tests.yaml`) · manual: [rhdh-local](https://github.com/redhat-developer/rhdh-local) | no | yes | +| Helm chart / Operator / ingress / real Keycloak / RBAC on OCP | `rhdh` e2e or `overlays: workspaces/*/e2e-tests` | cluster e2e | **yes** | n/a (cluster) | + +Rule of thumb: **source bugs → rhdh-plugins · artifact bugs → overlays · integration/render bugs → rhdh · platform bugs → cluster e2e.** If a bug is catchable in more than one place, test it in the cheapest one and don't duplicate downstream. + +## Step 3 — Layer ladder (cheapest that catches the bug wins) + +| Layer | Scope | Tooling | Cluster | Typical time | +| --- | --- | --- | --- | --- | +| **L1** Unit | Pure functions / logic | Jest/Vitest | no | ms–s | +| **L2** Integration | Backend module + plugin API, mocked external deps | `startTestBackend` + supertest | no | s | +| **L3** Component | React component/page with a test harness | RTL + dev server | no | s–min | +| **L4a** E2E cluster-free | Full app, no managed infra | Playwright + local harness | no | min | +| **L4b** E2E full | Real OCP/K8s, managed DBs, real IdPs | Playwright + cluster | **yes** | min–h | + +The full per-spec classification of the RHDH e2e suite lives in `rhdh: docs/e2e-tests/layer-migration-matrix.md` (see References) — consult it before adding or migrating an e2e spec. + +## Step 4 — How to create the test (per placement) + +### `rhdh-plugins` (or other source repo) — plugin correctness + +- **Where:** `plugins: workspaces//plugins//src/**` next to the code, following that workspace's existing test setup (Jest/RTL; MSW for API mocks). +- **How:** copy the pattern of a neighboring `*.test.ts(x)`; run with the workspace's `yarn test`. Manual verification via the workspace dev app (`yarn start`). +- Also validate packaging when the plugin's build config changed: `npx @red-hat-developer-hub/cli plugin export` must produce `dist-dynamic/` (+ `dist-scalprum/` for frontend). +- **Cannot do here:** validate the *published* OCI artifact, or integration with RHDH's app shell/shared deps. + +### `rhdh-plugin-export-overlays` — the published artifact + +- **Where:** the native smoke harness, `overlays: smoke-tests-native/` (in review — see References). It installs plugins from OCI with the real `install-dynamic-plugins` CLI and boots backend plugins via `startTestBackend`. +- **How:** write a `dynamic-plugins.yaml` listing the `oci://` artifact refs to validate (a workspace's refs are the `spec.dynamicArtifact` values in `overlays: workspaces//metadata/*.yaml`), then run `yarn smoke --dynamic-plugins [--out results.json]`. Exit code 0 = pass; non-zero writes `results.json` detailing `fail-load` / `fail-start`. CI: `overlays: .github/workflows/native-smoke.yaml`. +- **Known gaps (don't fight them):** catalog-extending backend modules don't boot in a minimal `startTestBackend` (upstream `catalog-backend` registration issue — stay on the Docker smoke until fixed); plugins whose `dynamicArtifact` is a local `./dynamic-plugins/dist/...` path ship inside the RHDH image and have no OCI artifact to validate here. +- **Never add UI-render tests here** — this repo has no app to render into. Delegate rendering to the `rhdh` cluster-free harness. +- The Docker smoke it replaces (backend scope) lives at `overlays: smoke-tests/` + workflow `run-workspace-smoke-tests.yaml`. +- `overlays: workspaces/*/e2e-tests` (cluster, Playwright + `e2e-test-utils`) exists for plugins whose value *is* cluster integration (topology, tekton, argocd…). Don't add one for UI-only behavior. + +### `rhdh` — the real app + +**L4a cluster-free harness** (in review — see References; config `rhdh: e2e-tests/playwright.legacy-local.config.ts`, docs `rhdh: docs/e2e-tests/local-e2e-harness.md`) — the only cheap place a frontend dynamic plugin can be *rendered*. To enable a spec/test: + +1. Plugin not yet installed by the harness? Add its OCI entry to `rhdh: e2e-tests/local-harness/dynamic-plugins.yaml` (tags on `ghcr.io/redhat-developer/rhdh-plugin-export-overlays/`). If its mount-point config is not in the repo's static `app-config.dynamic-plugins.yaml`, attach the plugin's **canonical `pluginConfig`** (source of truth: `plugins: workspaces//plugins//app-config.dynamic.yaml`) — the harness loads the generated `dynamic-plugins-root/app-config.dynamic-plugins.yaml` last, exactly like the production container. +2. Config that only exists in CI config maps (`rhdh: .ci/pipelines/resources/config_map/*`)? Mirror just the needed keys into `rhdh: app-config.local-e2e.yaml` (objects deep-merge; arrays replace). +3. Tag the test `{ tag: "@cluster-free" }` and add its spec file to the config's `testMatch` allowlist. +4. Repopulate + validate: `./e2e-tests/local-harness/populate.sh`, then `yarn --cwd e2e-tests e2e:legacy-local` (CI runs this in ~4 min). Requires skopeo — preinstalled in CI; on macOS `brew install skopeo`. + +**L4b cluster e2e** (`rhdh: e2e-tests/playwright/e2e/**`) — only when the subject *is* cluster/platform behavior or a real external service. Requirements: `component` annotation in `beforeAll` (see the repo's `ci-e2e-testing` rule), correct config map choice (RBAC vs non-RBAC), project registration in `e2e-tests/playwright/projects.json` if a new project is needed. + +**L3 component tests** — page-level RTL compositions, pattern proposed under RHIDP-13235 in [rhdh#4864](https://github.com/redhat-developer/rhdh/pull/4864) (closed, not merged). Check for merged examples before recommending a concrete template; prefer L3 over L4a when no dynamic-plugin loading is involved. + +**Catalog-index-wide sanity** (nightly, in review — see References) — nothing to write per plugin; it sweeps the whole index. + +## Not possible today (researched — don't burn time) + +- **Rendering a frontend dynamic plugin without an RHDH app** — the artifact is a legacy-frontend Scalprum bundle; no standalone host exists, and building one means maintaining a version-coupled "mini RHDH". +- **`@backstage/frontend-dynamic-feature-loader` for current plugins** — targets the new frontend system (alpha); our exported plugins are legacy-system bundles. Revisit when app-next matures (RHIDP-15082). +- **Catalog-extending modules in a minimal `startTestBackend`** — upstream `catalog-backend` issue (see overlays harness notes). + +## Output format + +Answer with a concrete recommendation: + +- **Repo:** which of the three (or the plugin's own source repo). +- **Layer / harness:** L1–L4b + the specific harness or suite. +- **Location:** the directory/file where the test goes. +- **Scaffolding:** the minimal steps or files to create, referencing an existing neighbor as template. +- **Why not elsewhere:** one line on the layers you rejected (especially if the dev proposed a more expensive one). +- **Cost:** rough feedback time (seconds / ~4 min cluster-free / cluster job). + +## References (status as of 2026-07-02 — verify before citing as merged) + +- Epic **RHIDP-13501** (E2E Test Optimization) — the per-repo responsibility split lives in the epic's comments and its Jira attachment `rhdh-dynamic-plugin-testing-guideline.md` (Jira-only; if unreachable, the decision table above is the summary). +- Layer matrix: — in review in [rhdh#5044](https://github.com/redhat-developer/rhdh/pull/5044) (RHIDP-15076). +- Cluster-free harness + docs: in review in [rhdh#5005](https://github.com/redhat-developer/rhdh/pull/5005) (RHIDP-15075). +- Overlays native smoke: in review in [overlays#2714](https://github.com/redhat-developer/rhdh-plugin-export-overlays/pull/2714). +- Catalog-index sanity check: in review in [rhdh#4967](https://github.com/redhat-developer/rhdh/pull/4967). +- L3 pattern proposal: [rhdh#4864](https://github.com/redhat-developer/rhdh/pull/4864) (closed) under RHIDP-13235.