From 230423f1e106fe9cd35cbc84caf9d22a503b8944 Mon Sep 17 00:00:00 2001 From: Gustavo Lira Date: Thu, 2 Jul 2026 16:39:57 -0300 Subject: [PATCH 1/2] =?UTF-8?q?docs(ai):=20add=20test-placement=20skill=20?= =?UTF-8?q?=E2=80=94=20where=20to=20test=20what=20in=20the=20plugin=20ecos?= =?UTF-8?q?ystem?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds an AI-assistant skill that, given the context of a change or bug, proposes the right repository (rhdh-plugins / rhdh-plugin-export-overlays / rhdh), the right test layer (L1 unit -> L4b cluster e2e), the location, and how to scaffold the test. Encodes the per-repo testing responsibility split consolidated on epic RHIDP-13501 (decision table, capability ladder, known-impossible list) so the cheapest environment that catches the bug is picked by default. Source in .rulesync/skills; .claude/.cursor/.opencode generated via rulesync 7.30.0. Co-Authored-By: Claude Opus 4.8 (1M context) --- .claude/skills/test-placement/SKILL.md | 112 ++++++++++++++++++++++ .cursor/skills/test-placement/SKILL.md | 108 +++++++++++++++++++++ .opencode/skill/test-placement/SKILL.md | 112 ++++++++++++++++++++++ .rulesync/skills/test-placement/SKILL.md | 114 +++++++++++++++++++++++ 4 files changed, 446 insertions(+) create mode 100644 .claude/skills/test-placement/SKILL.md create mode 100644 .cursor/skills/test-placement/SKILL.md create mode 100644 .opencode/skill/test-placement/SKILL.md create mode 100644 .rulesync/skills/test-placement/SKILL.md diff --git a/.claude/skills/test-placement/SKILL.md b/.claude/skills/test-placement/SKILL.md new file mode 100644 index 0000000000..1bb4ee54da --- /dev/null +++ b/.claude/skills/test-placement/SKILL.md @@ -0,0 +1,112 @@ +--- +name: test-placement +description: >- + Propose where to test a change in the RHDH dynamic-plugin ecosystem: which + repo (rhdh-plugins, rhdh-plugin-export-overlays, rhdh), which test layer + (unit, integration, component, cluster-free E2E, cluster E2E), where the test + lives, and how to create it +--- +# Test Placement Advisor + +Given the context of a change, bug, or new feature in the RHDH dynamic-plugin ecosystem, propose **where** it should be tested: which repository, which test layer, where the test lives, and how to create it. The guiding rule: **pick the cheapest environment that can actually catch the bug — most plugin validation does not need a cluster, and an increasing part doesn't need Docker either.** + +## When to Use + +- A developer asks "where should I test this?", "should this be an e2e test?", "does this need a cluster?", or "which repo does this test belong in?" +- A new plugin, plugin version bump, or plugin config change needs test coverage. +- A bug escaped to a cluster e2e run and the team wants a cheaper regression test. +- Reviewing a PR that adds a test at the wrong layer (e.g. a cluster e2e for pure UI logic). + +## Step 1 — Gather context (ask if missing) + +Before recommending, establish: + +1. **What is being validated?** Plugin logic/UI component · packaging/published artifact · plugin loading/rendering inside RHDH · platform behavior (Helm/Operator/ingress/auth). +2. **Where did the change happen?** Plugin source (`rhdh-plugins` or another source repo) · packaging metadata (`rhdh-plugin-export-overlays`) · the RHDH app itself (`rhdh`). +3. **Does verifying it require a rendered UI?** (clicks, headings, navigation) +4. **Does it require real external services?** (GitHub, Keycloak, LDAP, managed DBs) +5. **Does it require cluster infrastructure?** (ConfigMap reload, routes, operator, pod logs, port-forward) + +## Step 2 — Decision table + +| The dev wants to verify… | Repo | Test type / harness | Cluster? | Docker? | +| --- | --- | --- | --- | --- | +| Plugin logic / components / API | `rhdh-plugins` (or the plugin's source repo) | unit + component tests, dev app (`yarn start`) | no | no | +| The plugin still builds as a dynamic plugin | `rhdh-plugins` | `export-dynamic` / `npx @red-hat-developer-hub/cli plugin export` | no | no | +| The **published OCI artifact** installs and the **backend plugin boots** | `rhdh-plugin-export-overlays` | native smoke harness (`smoke-tests-native/`, overlays PR #2714) | no | **no** | +| All plugins of a workspace boot together | `rhdh-plugin-export-overlays` | native smoke `--workspace` mode | no | **no** | +| A frontend artifact ships its bundle (`dist-scalprum/`, `plugin-manifest.json`) | `rhdh-plugin-export-overlays` | native smoke presence check | no | **no** | +| The plugin **loads inside the real RHDH app** and the **UI renders** | `rhdh` | cluster-free E2E harness (`e2e:legacy-local`, rhdh PR #5005) | no | no | +| Every plugin in the **catalog index** is sane | `rhdh` | plugin sanity check (rhdh PR #4967, nightly) | no | no | +| RHDH **container** behavior (image entrypoint, install script inside the image) | any | Docker smoke / [rhdh-local](https://github.com/redhat-developer/rhdh-local) | no | yes | +| Helm chart / Operator / ingress / real Keycloak / RBAC on OCP | `rhdh` e2e or `rhdh-plugin-export-overlays` `workspaces/*/e2e-tests` | cluster e2e | **yes** | — | + +Rule of thumb: **source bugs → rhdh-plugins · artifact bugs → overlays · integration/render bugs → rhdh · platform bugs → cluster e2e.** If a bug is catchable in more than one place, test it in the cheapest one and don't duplicate downstream. + +## Step 3 — Layer ladder (cheapest that catches the bug wins) + +| Layer | Scope | Tooling | Cluster | Typical time | +| --- | --- | --- | --- | --- | +| **L1** Unit | Pure functions / logic | Jest/Vitest | no | ms–s | +| **L2** Integration | Backend module + plugin API, mocked external deps | `startTestBackend` + supertest | no | s | +| **L3** Component | React component/page with a test harness | RTL + dev server | no | s–min | +| **L4a** E2E cluster-free | Full app, no managed infra | Playwright + local harness | no | min | +| **L4b** E2E full | Real OCP/K8s, managed DBs, real IdPs | Playwright + cluster | **yes** | min–h | + +The full per-spec classification of the RHDH e2e suite lives in `rhdh/docs/e2e-tests/layer-migration-matrix.md` (RHIDP-15076) — consult it before adding or migrating an e2e spec. + +## Step 4 — How to create the test (per placement) + +### `rhdh-plugins` (or other source repo) — plugin correctness + +- **Where:** `workspaces//plugins//src/**` next to the code, following that workspace's existing test setup (Jest/RTL; MSW for API mocks). +- **How:** copy the pattern of a neighboring `*.test.ts(x)`; run with the workspace's `yarn test`. Manual verification via the workspace dev app (`yarn start`). +- Also validate packaging when the plugin's build config changed: `npx @red-hat-developer-hub/cli plugin export` must produce `dist-dynamic/` (+ `dist-scalprum/` for frontend). +- **Cannot do here:** validate the *published* OCI artifact, or integration with RHDH's app shell/shared deps. + +### `rhdh-plugin-export-overlays` — the published artifact + +- **Where:** the native smoke harness (`smoke-tests-native/`, overlays PR #2714). Per-plugin: install from OCI with the real `install-dynamic-plugins` CLI and boot backend plugins via `startTestBackend`. Workspace mode: `yarn smoke --workspace ` boots every `oci://` artifact of the workspace together. +- New workspaces usually need **no new test code** — the harness discovers artifacts from `workspaces//metadata/*.yaml`. +- **Known gaps (don't fight them):** catalog-extending backend modules don't boot in a minimal `startTestBackend` (upstream `catalog-backend` registration issue — stay on Docker smoke until fixed); plugins whose `dynamicArtifact` is a local `./dynamic-plugins/dist/...` path ship inside the RHDH image and have no OCI artifact to validate here. +- **Never add UI-render tests here** — this repo has no app to render into. Delegate rendering to the `rhdh` cluster-free harness. +- `workspaces/*/e2e-tests` (cluster, Playwright + `e2e-test-utils`) exists for plugins whose value *is* cluster integration (topology, tekton, argocd…). Don't add one for UI-only behavior. + +### `rhdh` — the real app + +**L4a cluster-free harness** (`e2e-tests/playwright.legacy-local.config.ts`, docs at `docs/e2e-tests/local-e2e-harness.md`) — the only cheap place a frontend dynamic plugin can be *rendered*. To enable a spec/test: + +1. Plugin not yet installed by the harness? Add its OCI entry to `e2e-tests/local-harness/dynamic-plugins.yaml` (tags on `ghcr.io/redhat-developer/rhdh-plugin-export-overlays/`). If its mount-point config is not in the repo's static `app-config.dynamic-plugins.yaml`, attach the plugin's **canonical `pluginConfig`** (source of truth: `rhdh-plugins` → `workspaces//plugins//app-config.dynamic.yaml`) — the harness loads the generated `dynamic-plugins-root/app-config.dynamic-plugins.yaml` last, exactly like the production container. +2. Config that only exists in CI config maps (`.ci/pipelines/resources/config_map/*`)? Mirror just the needed keys into `app-config.local-e2e.yaml` (objects deep-merge; arrays replace). +3. Tag the test `{ tag: "@cluster-free" }` and add its spec file to the config's `testMatch` allowlist. +4. Repopulate + validate: `./e2e-tests/local-harness/populate.sh`, then `yarn --cwd e2e-tests e2e:legacy-local` (CI does this in ~4 min; skopeo is Linux/CI-only). + +**L4b cluster e2e** (`e2e-tests/playwright/e2e/**`) — only when the subject *is* cluster/platform behavior or a real external service. Requirements: `component` annotation in `beforeAll` (see `ci-e2e-testing` rule), correct config map choice (RBAC vs non-RBAC), project registration in `e2e-tests/playwright/projects.json` if a new project is needed. + +**L3 component tests** — pattern established on branch `RHIDP-13235-layer3-component-tests` (page-level RTL compositions). Prefer this over L4a when no dynamic-plugin loading is involved. + +**Catalog-index-wide sanity** (rhdh PR #4967, nightly) — nothing to write per plugin; it sweeps the whole index. + +## Not possible today (researched — don't burn time) + +- **Rendering a frontend dynamic plugin without an RHDH app** — the artifact is a legacy-frontend Scalprum bundle; no standalone host exists, and building one means maintaining a version-coupled "mini RHDH". +- **`@backstage/frontend-dynamic-feature-loader` for current plugins** — targets the new frontend system (alpha); our exported plugins are legacy-system bundles. Revisit when app-next matures (RHIDP-15082). +- **Catalog-extending modules in a minimal `startTestBackend`** — upstream `catalog-backend` issue (see overlays harness notes). + +## Output format + +Answer with a concrete recommendation: + +- **Repo:** which of the three (or the plugin's own source repo). +- **Layer / harness:** L1–L4b + the specific harness or suite. +- **Location:** the directory/file where the test goes. +- **Scaffolding:** the minimal steps or files to create, referencing an existing neighbor as template. +- **Why not elsewhere:** one line on the layers you rejected (especially if the dev proposed a more expensive one). +- **Cost:** rough feedback time (seconds / ~4 min cluster-free / cluster job). + +## References + +- Epic RHIDP-13501 (E2E Test Optimization) — per-repo responsibility split in the epic comments and the attached `rhdh-dynamic-plugin-testing-guideline.md`. +- `rhdh/docs/e2e-tests/layer-migration-matrix.md` — per-spec layer classification (RHIDP-15076). +- `rhdh/docs/e2e-tests/local-e2e-harness.md` — cluster-free harness usage. +- Reference PRs: overlays#2714 (native smoke), rhdh#4967 (catalog-index sanity), rhdh#5005 (cluster-free E2E harness). diff --git a/.cursor/skills/test-placement/SKILL.md b/.cursor/skills/test-placement/SKILL.md new file mode 100644 index 0000000000..64d14eb430 --- /dev/null +++ b/.cursor/skills/test-placement/SKILL.md @@ -0,0 +1,108 @@ +--- +name: test-placement +description: 'Propose where to test a change in the RHDH dynamic-plugin ecosystem: which repo (rhdh-plugins, rhdh-plugin-export-overlays, rhdh), which test layer (unit, integration, component, cluster-free E2E, cluster E2E), where the test lives, and how to create it' +--- +# Test Placement Advisor + +Given the context of a change, bug, or new feature in the RHDH dynamic-plugin ecosystem, propose **where** it should be tested: which repository, which test layer, where the test lives, and how to create it. The guiding rule: **pick the cheapest environment that can actually catch the bug — most plugin validation does not need a cluster, and an increasing part doesn't need Docker either.** + +## When to Use + +- A developer asks "where should I test this?", "should this be an e2e test?", "does this need a cluster?", or "which repo does this test belong in?" +- A new plugin, plugin version bump, or plugin config change needs test coverage. +- A bug escaped to a cluster e2e run and the team wants a cheaper regression test. +- Reviewing a PR that adds a test at the wrong layer (e.g. a cluster e2e for pure UI logic). + +## Step 1 — Gather context (ask if missing) + +Before recommending, establish: + +1. **What is being validated?** Plugin logic/UI component · packaging/published artifact · plugin loading/rendering inside RHDH · platform behavior (Helm/Operator/ingress/auth). +2. **Where did the change happen?** Plugin source (`rhdh-plugins` or another source repo) · packaging metadata (`rhdh-plugin-export-overlays`) · the RHDH app itself (`rhdh`). +3. **Does verifying it require a rendered UI?** (clicks, headings, navigation) +4. **Does it require real external services?** (GitHub, Keycloak, LDAP, managed DBs) +5. **Does it require cluster infrastructure?** (ConfigMap reload, routes, operator, pod logs, port-forward) + +## Step 2 — Decision table + +| The dev wants to verify… | Repo | Test type / harness | Cluster? | Docker? | +| --- | --- | --- | --- | --- | +| Plugin logic / components / API | `rhdh-plugins` (or the plugin's source repo) | unit + component tests, dev app (`yarn start`) | no | no | +| The plugin still builds as a dynamic plugin | `rhdh-plugins` | `export-dynamic` / `npx @red-hat-developer-hub/cli plugin export` | no | no | +| The **published OCI artifact** installs and the **backend plugin boots** | `rhdh-plugin-export-overlays` | native smoke harness (`smoke-tests-native/`, overlays PR #2714) | no | **no** | +| All plugins of a workspace boot together | `rhdh-plugin-export-overlays` | native smoke `--workspace` mode | no | **no** | +| A frontend artifact ships its bundle (`dist-scalprum/`, `plugin-manifest.json`) | `rhdh-plugin-export-overlays` | native smoke presence check | no | **no** | +| The plugin **loads inside the real RHDH app** and the **UI renders** | `rhdh` | cluster-free E2E harness (`e2e:legacy-local`, rhdh PR #5005) | no | no | +| Every plugin in the **catalog index** is sane | `rhdh` | plugin sanity check (rhdh PR #4967, nightly) | no | no | +| RHDH **container** behavior (image entrypoint, install script inside the image) | any | Docker smoke / [rhdh-local](https://github.com/redhat-developer/rhdh-local) | no | yes | +| Helm chart / Operator / ingress / real Keycloak / RBAC on OCP | `rhdh` e2e or `rhdh-plugin-export-overlays` `workspaces/*/e2e-tests` | cluster e2e | **yes** | — | + +Rule of thumb: **source bugs → rhdh-plugins · artifact bugs → overlays · integration/render bugs → rhdh · platform bugs → cluster e2e.** If a bug is catchable in more than one place, test it in the cheapest one and don't duplicate downstream. + +## Step 3 — Layer ladder (cheapest that catches the bug wins) + +| Layer | Scope | Tooling | Cluster | Typical time | +| --- | --- | --- | --- | --- | +| **L1** Unit | Pure functions / logic | Jest/Vitest | no | ms–s | +| **L2** Integration | Backend module + plugin API, mocked external deps | `startTestBackend` + supertest | no | s | +| **L3** Component | React component/page with a test harness | RTL + dev server | no | s–min | +| **L4a** E2E cluster-free | Full app, no managed infra | Playwright + local harness | no | min | +| **L4b** E2E full | Real OCP/K8s, managed DBs, real IdPs | Playwright + cluster | **yes** | min–h | + +The full per-spec classification of the RHDH e2e suite lives in `rhdh/docs/e2e-tests/layer-migration-matrix.md` (RHIDP-15076) — consult it before adding or migrating an e2e spec. + +## Step 4 — How to create the test (per placement) + +### `rhdh-plugins` (or other source repo) — plugin correctness + +- **Where:** `workspaces//plugins//src/**` next to the code, following that workspace's existing test setup (Jest/RTL; MSW for API mocks). +- **How:** copy the pattern of a neighboring `*.test.ts(x)`; run with the workspace's `yarn test`. Manual verification via the workspace dev app (`yarn start`). +- Also validate packaging when the plugin's build config changed: `npx @red-hat-developer-hub/cli plugin export` must produce `dist-dynamic/` (+ `dist-scalprum/` for frontend). +- **Cannot do here:** validate the *published* OCI artifact, or integration with RHDH's app shell/shared deps. + +### `rhdh-plugin-export-overlays` — the published artifact + +- **Where:** the native smoke harness (`smoke-tests-native/`, overlays PR #2714). Per-plugin: install from OCI with the real `install-dynamic-plugins` CLI and boot backend plugins via `startTestBackend`. Workspace mode: `yarn smoke --workspace ` boots every `oci://` artifact of the workspace together. +- New workspaces usually need **no new test code** — the harness discovers artifacts from `workspaces//metadata/*.yaml`. +- **Known gaps (don't fight them):** catalog-extending backend modules don't boot in a minimal `startTestBackend` (upstream `catalog-backend` registration issue — stay on Docker smoke until fixed); plugins whose `dynamicArtifact` is a local `./dynamic-plugins/dist/...` path ship inside the RHDH image and have no OCI artifact to validate here. +- **Never add UI-render tests here** — this repo has no app to render into. Delegate rendering to the `rhdh` cluster-free harness. +- `workspaces/*/e2e-tests` (cluster, Playwright + `e2e-test-utils`) exists for plugins whose value *is* cluster integration (topology, tekton, argocd…). Don't add one for UI-only behavior. + +### `rhdh` — the real app + +**L4a cluster-free harness** (`e2e-tests/playwright.legacy-local.config.ts`, docs at `docs/e2e-tests/local-e2e-harness.md`) — the only cheap place a frontend dynamic plugin can be *rendered*. To enable a spec/test: + +1. Plugin not yet installed by the harness? Add its OCI entry to `e2e-tests/local-harness/dynamic-plugins.yaml` (tags on `ghcr.io/redhat-developer/rhdh-plugin-export-overlays/`). If its mount-point config is not in the repo's static `app-config.dynamic-plugins.yaml`, attach the plugin's **canonical `pluginConfig`** (source of truth: `rhdh-plugins` → `workspaces//plugins//app-config.dynamic.yaml`) — the harness loads the generated `dynamic-plugins-root/app-config.dynamic-plugins.yaml` last, exactly like the production container. +2. Config that only exists in CI config maps (`.ci/pipelines/resources/config_map/*`)? Mirror just the needed keys into `app-config.local-e2e.yaml` (objects deep-merge; arrays replace). +3. Tag the test `{ tag: "@cluster-free" }` and add its spec file to the config's `testMatch` allowlist. +4. Repopulate + validate: `./e2e-tests/local-harness/populate.sh`, then `yarn --cwd e2e-tests e2e:legacy-local` (CI does this in ~4 min; skopeo is Linux/CI-only). + +**L4b cluster e2e** (`e2e-tests/playwright/e2e/**`) — only when the subject *is* cluster/platform behavior or a real external service. Requirements: `component` annotation in `beforeAll` (see `ci-e2e-testing` rule), correct config map choice (RBAC vs non-RBAC), project registration in `e2e-tests/playwright/projects.json` if a new project is needed. + +**L3 component tests** — pattern established on branch `RHIDP-13235-layer3-component-tests` (page-level RTL compositions). Prefer this over L4a when no dynamic-plugin loading is involved. + +**Catalog-index-wide sanity** (rhdh PR #4967, nightly) — nothing to write per plugin; it sweeps the whole index. + +## Not possible today (researched — don't burn time) + +- **Rendering a frontend dynamic plugin without an RHDH app** — the artifact is a legacy-frontend Scalprum bundle; no standalone host exists, and building one means maintaining a version-coupled "mini RHDH". +- **`@backstage/frontend-dynamic-feature-loader` for current plugins** — targets the new frontend system (alpha); our exported plugins are legacy-system bundles. Revisit when app-next matures (RHIDP-15082). +- **Catalog-extending modules in a minimal `startTestBackend`** — upstream `catalog-backend` issue (see overlays harness notes). + +## Output format + +Answer with a concrete recommendation: + +- **Repo:** which of the three (or the plugin's own source repo). +- **Layer / harness:** L1–L4b + the specific harness or suite. +- **Location:** the directory/file where the test goes. +- **Scaffolding:** the minimal steps or files to create, referencing an existing neighbor as template. +- **Why not elsewhere:** one line on the layers you rejected (especially if the dev proposed a more expensive one). +- **Cost:** rough feedback time (seconds / ~4 min cluster-free / cluster job). + +## References + +- Epic RHIDP-13501 (E2E Test Optimization) — per-repo responsibility split in the epic comments and the attached `rhdh-dynamic-plugin-testing-guideline.md`. +- `rhdh/docs/e2e-tests/layer-migration-matrix.md` — per-spec layer classification (RHIDP-15076). +- `rhdh/docs/e2e-tests/local-e2e-harness.md` — cluster-free harness usage. +- Reference PRs: overlays#2714 (native smoke), rhdh#4967 (catalog-index sanity), rhdh#5005 (cluster-free E2E harness). diff --git a/.opencode/skill/test-placement/SKILL.md b/.opencode/skill/test-placement/SKILL.md new file mode 100644 index 0000000000..1bb4ee54da --- /dev/null +++ b/.opencode/skill/test-placement/SKILL.md @@ -0,0 +1,112 @@ +--- +name: test-placement +description: >- + Propose where to test a change in the RHDH dynamic-plugin ecosystem: which + repo (rhdh-plugins, rhdh-plugin-export-overlays, rhdh), which test layer + (unit, integration, component, cluster-free E2E, cluster E2E), where the test + lives, and how to create it +--- +# Test Placement Advisor + +Given the context of a change, bug, or new feature in the RHDH dynamic-plugin ecosystem, propose **where** it should be tested: which repository, which test layer, where the test lives, and how to create it. The guiding rule: **pick the cheapest environment that can actually catch the bug — most plugin validation does not need a cluster, and an increasing part doesn't need Docker either.** + +## When to Use + +- A developer asks "where should I test this?", "should this be an e2e test?", "does this need a cluster?", or "which repo does this test belong in?" +- A new plugin, plugin version bump, or plugin config change needs test coverage. +- A bug escaped to a cluster e2e run and the team wants a cheaper regression test. +- Reviewing a PR that adds a test at the wrong layer (e.g. a cluster e2e for pure UI logic). + +## Step 1 — Gather context (ask if missing) + +Before recommending, establish: + +1. **What is being validated?** Plugin logic/UI component · packaging/published artifact · plugin loading/rendering inside RHDH · platform behavior (Helm/Operator/ingress/auth). +2. **Where did the change happen?** Plugin source (`rhdh-plugins` or another source repo) · packaging metadata (`rhdh-plugin-export-overlays`) · the RHDH app itself (`rhdh`). +3. **Does verifying it require a rendered UI?** (clicks, headings, navigation) +4. **Does it require real external services?** (GitHub, Keycloak, LDAP, managed DBs) +5. **Does it require cluster infrastructure?** (ConfigMap reload, routes, operator, pod logs, port-forward) + +## Step 2 — Decision table + +| The dev wants to verify… | Repo | Test type / harness | Cluster? | Docker? | +| --- | --- | --- | --- | --- | +| Plugin logic / components / API | `rhdh-plugins` (or the plugin's source repo) | unit + component tests, dev app (`yarn start`) | no | no | +| The plugin still builds as a dynamic plugin | `rhdh-plugins` | `export-dynamic` / `npx @red-hat-developer-hub/cli plugin export` | no | no | +| The **published OCI artifact** installs and the **backend plugin boots** | `rhdh-plugin-export-overlays` | native smoke harness (`smoke-tests-native/`, overlays PR #2714) | no | **no** | +| All plugins of a workspace boot together | `rhdh-plugin-export-overlays` | native smoke `--workspace` mode | no | **no** | +| A frontend artifact ships its bundle (`dist-scalprum/`, `plugin-manifest.json`) | `rhdh-plugin-export-overlays` | native smoke presence check | no | **no** | +| The plugin **loads inside the real RHDH app** and the **UI renders** | `rhdh` | cluster-free E2E harness (`e2e:legacy-local`, rhdh PR #5005) | no | no | +| Every plugin in the **catalog index** is sane | `rhdh` | plugin sanity check (rhdh PR #4967, nightly) | no | no | +| RHDH **container** behavior (image entrypoint, install script inside the image) | any | Docker smoke / [rhdh-local](https://github.com/redhat-developer/rhdh-local) | no | yes | +| Helm chart / Operator / ingress / real Keycloak / RBAC on OCP | `rhdh` e2e or `rhdh-plugin-export-overlays` `workspaces/*/e2e-tests` | cluster e2e | **yes** | — | + +Rule of thumb: **source bugs → rhdh-plugins · artifact bugs → overlays · integration/render bugs → rhdh · platform bugs → cluster e2e.** If a bug is catchable in more than one place, test it in the cheapest one and don't duplicate downstream. + +## Step 3 — Layer ladder (cheapest that catches the bug wins) + +| Layer | Scope | Tooling | Cluster | Typical time | +| --- | --- | --- | --- | --- | +| **L1** Unit | Pure functions / logic | Jest/Vitest | no | ms–s | +| **L2** Integration | Backend module + plugin API, mocked external deps | `startTestBackend` + supertest | no | s | +| **L3** Component | React component/page with a test harness | RTL + dev server | no | s–min | +| **L4a** E2E cluster-free | Full app, no managed infra | Playwright + local harness | no | min | +| **L4b** E2E full | Real OCP/K8s, managed DBs, real IdPs | Playwright + cluster | **yes** | min–h | + +The full per-spec classification of the RHDH e2e suite lives in `rhdh/docs/e2e-tests/layer-migration-matrix.md` (RHIDP-15076) — consult it before adding or migrating an e2e spec. + +## Step 4 — How to create the test (per placement) + +### `rhdh-plugins` (or other source repo) — plugin correctness + +- **Where:** `workspaces//plugins//src/**` next to the code, following that workspace's existing test setup (Jest/RTL; MSW for API mocks). +- **How:** copy the pattern of a neighboring `*.test.ts(x)`; run with the workspace's `yarn test`. Manual verification via the workspace dev app (`yarn start`). +- Also validate packaging when the plugin's build config changed: `npx @red-hat-developer-hub/cli plugin export` must produce `dist-dynamic/` (+ `dist-scalprum/` for frontend). +- **Cannot do here:** validate the *published* OCI artifact, or integration with RHDH's app shell/shared deps. + +### `rhdh-plugin-export-overlays` — the published artifact + +- **Where:** the native smoke harness (`smoke-tests-native/`, overlays PR #2714). Per-plugin: install from OCI with the real `install-dynamic-plugins` CLI and boot backend plugins via `startTestBackend`. Workspace mode: `yarn smoke --workspace ` boots every `oci://` artifact of the workspace together. +- New workspaces usually need **no new test code** — the harness discovers artifacts from `workspaces//metadata/*.yaml`. +- **Known gaps (don't fight them):** catalog-extending backend modules don't boot in a minimal `startTestBackend` (upstream `catalog-backend` registration issue — stay on Docker smoke until fixed); plugins whose `dynamicArtifact` is a local `./dynamic-plugins/dist/...` path ship inside the RHDH image and have no OCI artifact to validate here. +- **Never add UI-render tests here** — this repo has no app to render into. Delegate rendering to the `rhdh` cluster-free harness. +- `workspaces/*/e2e-tests` (cluster, Playwright + `e2e-test-utils`) exists for plugins whose value *is* cluster integration (topology, tekton, argocd…). Don't add one for UI-only behavior. + +### `rhdh` — the real app + +**L4a cluster-free harness** (`e2e-tests/playwright.legacy-local.config.ts`, docs at `docs/e2e-tests/local-e2e-harness.md`) — the only cheap place a frontend dynamic plugin can be *rendered*. To enable a spec/test: + +1. Plugin not yet installed by the harness? Add its OCI entry to `e2e-tests/local-harness/dynamic-plugins.yaml` (tags on `ghcr.io/redhat-developer/rhdh-plugin-export-overlays/`). If its mount-point config is not in the repo's static `app-config.dynamic-plugins.yaml`, attach the plugin's **canonical `pluginConfig`** (source of truth: `rhdh-plugins` → `workspaces//plugins//app-config.dynamic.yaml`) — the harness loads the generated `dynamic-plugins-root/app-config.dynamic-plugins.yaml` last, exactly like the production container. +2. Config that only exists in CI config maps (`.ci/pipelines/resources/config_map/*`)? Mirror just the needed keys into `app-config.local-e2e.yaml` (objects deep-merge; arrays replace). +3. Tag the test `{ tag: "@cluster-free" }` and add its spec file to the config's `testMatch` allowlist. +4. Repopulate + validate: `./e2e-tests/local-harness/populate.sh`, then `yarn --cwd e2e-tests e2e:legacy-local` (CI does this in ~4 min; skopeo is Linux/CI-only). + +**L4b cluster e2e** (`e2e-tests/playwright/e2e/**`) — only when the subject *is* cluster/platform behavior or a real external service. Requirements: `component` annotation in `beforeAll` (see `ci-e2e-testing` rule), correct config map choice (RBAC vs non-RBAC), project registration in `e2e-tests/playwright/projects.json` if a new project is needed. + +**L3 component tests** — pattern established on branch `RHIDP-13235-layer3-component-tests` (page-level RTL compositions). Prefer this over L4a when no dynamic-plugin loading is involved. + +**Catalog-index-wide sanity** (rhdh PR #4967, nightly) — nothing to write per plugin; it sweeps the whole index. + +## Not possible today (researched — don't burn time) + +- **Rendering a frontend dynamic plugin without an RHDH app** — the artifact is a legacy-frontend Scalprum bundle; no standalone host exists, and building one means maintaining a version-coupled "mini RHDH". +- **`@backstage/frontend-dynamic-feature-loader` for current plugins** — targets the new frontend system (alpha); our exported plugins are legacy-system bundles. Revisit when app-next matures (RHIDP-15082). +- **Catalog-extending modules in a minimal `startTestBackend`** — upstream `catalog-backend` issue (see overlays harness notes). + +## Output format + +Answer with a concrete recommendation: + +- **Repo:** which of the three (or the plugin's own source repo). +- **Layer / harness:** L1–L4b + the specific harness or suite. +- **Location:** the directory/file where the test goes. +- **Scaffolding:** the minimal steps or files to create, referencing an existing neighbor as template. +- **Why not elsewhere:** one line on the layers you rejected (especially if the dev proposed a more expensive one). +- **Cost:** rough feedback time (seconds / ~4 min cluster-free / cluster job). + +## References + +- Epic RHIDP-13501 (E2E Test Optimization) — per-repo responsibility split in the epic comments and the attached `rhdh-dynamic-plugin-testing-guideline.md`. +- `rhdh/docs/e2e-tests/layer-migration-matrix.md` — per-spec layer classification (RHIDP-15076). +- `rhdh/docs/e2e-tests/local-e2e-harness.md` — cluster-free harness usage. +- Reference PRs: overlays#2714 (native smoke), rhdh#4967 (catalog-index sanity), rhdh#5005 (cluster-free E2E harness). diff --git a/.rulesync/skills/test-placement/SKILL.md b/.rulesync/skills/test-placement/SKILL.md new file mode 100644 index 0000000000..fc57d45701 --- /dev/null +++ b/.rulesync/skills/test-placement/SKILL.md @@ -0,0 +1,114 @@ +--- +name: test-placement +description: >- + Propose where to test a change in the RHDH dynamic-plugin ecosystem: which + repo (rhdh-plugins, rhdh-plugin-export-overlays, rhdh), which test layer + (unit, integration, component, cluster-free E2E, cluster E2E), where the + test lives, and how to create it +targets: + - '*' +--- +# Test Placement Advisor + +Given the context of a change, bug, or new feature in the RHDH dynamic-plugin ecosystem, propose **where** it should be tested: which repository, which test layer, where the test lives, and how to create it. The guiding rule: **pick the cheapest environment that can actually catch the bug — most plugin validation does not need a cluster, and an increasing part doesn't need Docker either.** + +## When to Use + +- A developer asks "where should I test this?", "should this be an e2e test?", "does this need a cluster?", or "which repo does this test belong in?" +- A new plugin, plugin version bump, or plugin config change needs test coverage. +- A bug escaped to a cluster e2e run and the team wants a cheaper regression test. +- Reviewing a PR that adds a test at the wrong layer (e.g. a cluster e2e for pure UI logic). + +## Step 1 — Gather context (ask if missing) + +Before recommending, establish: + +1. **What is being validated?** Plugin logic/UI component · packaging/published artifact · plugin loading/rendering inside RHDH · platform behavior (Helm/Operator/ingress/auth). +2. **Where did the change happen?** Plugin source (`rhdh-plugins` or another source repo) · packaging metadata (`rhdh-plugin-export-overlays`) · the RHDH app itself (`rhdh`). +3. **Does verifying it require a rendered UI?** (clicks, headings, navigation) +4. **Does it require real external services?** (GitHub, Keycloak, LDAP, managed DBs) +5. **Does it require cluster infrastructure?** (ConfigMap reload, routes, operator, pod logs, port-forward) + +## Step 2 — Decision table + +| The dev wants to verify… | Repo | Test type / harness | Cluster? | Docker? | +| --- | --- | --- | --- | --- | +| Plugin logic / components / API | `rhdh-plugins` (or the plugin's source repo) | unit + component tests, dev app (`yarn start`) | no | no | +| The plugin still builds as a dynamic plugin | `rhdh-plugins` | `export-dynamic` / `npx @red-hat-developer-hub/cli plugin export` | no | no | +| The **published OCI artifact** installs and the **backend plugin boots** | `rhdh-plugin-export-overlays` | native smoke harness (`smoke-tests-native/`, overlays PR #2714) | no | **no** | +| All plugins of a workspace boot together | `rhdh-plugin-export-overlays` | native smoke `--workspace` mode | no | **no** | +| A frontend artifact ships its bundle (`dist-scalprum/`, `plugin-manifest.json`) | `rhdh-plugin-export-overlays` | native smoke presence check | no | **no** | +| The plugin **loads inside the real RHDH app** and the **UI renders** | `rhdh` | cluster-free E2E harness (`e2e:legacy-local`, rhdh PR #5005) | no | no | +| Every plugin in the **catalog index** is sane | `rhdh` | plugin sanity check (rhdh PR #4967, nightly) | no | no | +| RHDH **container** behavior (image entrypoint, install script inside the image) | any | Docker smoke / [rhdh-local](https://github.com/redhat-developer/rhdh-local) | no | yes | +| Helm chart / Operator / ingress / real Keycloak / RBAC on OCP | `rhdh` e2e or `rhdh-plugin-export-overlays` `workspaces/*/e2e-tests` | cluster e2e | **yes** | — | + +Rule of thumb: **source bugs → rhdh-plugins · artifact bugs → overlays · integration/render bugs → rhdh · platform bugs → cluster e2e.** If a bug is catchable in more than one place, test it in the cheapest one and don't duplicate downstream. + +## Step 3 — Layer ladder (cheapest that catches the bug wins) + +| Layer | Scope | Tooling | Cluster | Typical time | +| --- | --- | --- | --- | --- | +| **L1** Unit | Pure functions / logic | Jest/Vitest | no | ms–s | +| **L2** Integration | Backend module + plugin API, mocked external deps | `startTestBackend` + supertest | no | s | +| **L3** Component | React component/page with a test harness | RTL + dev server | no | s–min | +| **L4a** E2E cluster-free | Full app, no managed infra | Playwright + local harness | no | min | +| **L4b** E2E full | Real OCP/K8s, managed DBs, real IdPs | Playwright + cluster | **yes** | min–h | + +The full per-spec classification of the RHDH e2e suite lives in `rhdh/docs/e2e-tests/layer-migration-matrix.md` (RHIDP-15076) — consult it before adding or migrating an e2e spec. + +## Step 4 — How to create the test (per placement) + +### `rhdh-plugins` (or other source repo) — plugin correctness + +- **Where:** `workspaces//plugins//src/**` next to the code, following that workspace's existing test setup (Jest/RTL; MSW for API mocks). +- **How:** copy the pattern of a neighboring `*.test.ts(x)`; run with the workspace's `yarn test`. Manual verification via the workspace dev app (`yarn start`). +- Also validate packaging when the plugin's build config changed: `npx @red-hat-developer-hub/cli plugin export` must produce `dist-dynamic/` (+ `dist-scalprum/` for frontend). +- **Cannot do here:** validate the *published* OCI artifact, or integration with RHDH's app shell/shared deps. + +### `rhdh-plugin-export-overlays` — the published artifact + +- **Where:** the native smoke harness (`smoke-tests-native/`, overlays PR #2714). Per-plugin: install from OCI with the real `install-dynamic-plugins` CLI and boot backend plugins via `startTestBackend`. Workspace mode: `yarn smoke --workspace ` boots every `oci://` artifact of the workspace together. +- New workspaces usually need **no new test code** — the harness discovers artifacts from `workspaces//metadata/*.yaml`. +- **Known gaps (don't fight them):** catalog-extending backend modules don't boot in a minimal `startTestBackend` (upstream `catalog-backend` registration issue — stay on Docker smoke until fixed); plugins whose `dynamicArtifact` is a local `./dynamic-plugins/dist/...` path ship inside the RHDH image and have no OCI artifact to validate here. +- **Never add UI-render tests here** — this repo has no app to render into. Delegate rendering to the `rhdh` cluster-free harness. +- `workspaces/*/e2e-tests` (cluster, Playwright + `e2e-test-utils`) exists for plugins whose value *is* cluster integration (topology, tekton, argocd…). Don't add one for UI-only behavior. + +### `rhdh` — the real app + +**L4a cluster-free harness** (`e2e-tests/playwright.legacy-local.config.ts`, docs at `docs/e2e-tests/local-e2e-harness.md`) — the only cheap place a frontend dynamic plugin can be *rendered*. To enable a spec/test: + +1. Plugin not yet installed by the harness? Add its OCI entry to `e2e-tests/local-harness/dynamic-plugins.yaml` (tags on `ghcr.io/redhat-developer/rhdh-plugin-export-overlays/`). If its mount-point config is not in the repo's static `app-config.dynamic-plugins.yaml`, attach the plugin's **canonical `pluginConfig`** (source of truth: `rhdh-plugins` → `workspaces//plugins//app-config.dynamic.yaml`) — the harness loads the generated `dynamic-plugins-root/app-config.dynamic-plugins.yaml` last, exactly like the production container. +2. Config that only exists in CI config maps (`.ci/pipelines/resources/config_map/*`)? Mirror just the needed keys into `app-config.local-e2e.yaml` (objects deep-merge; arrays replace). +3. Tag the test `{ tag: "@cluster-free" }` and add its spec file to the config's `testMatch` allowlist. +4. Repopulate + validate: `./e2e-tests/local-harness/populate.sh`, then `yarn --cwd e2e-tests e2e:legacy-local` (CI does this in ~4 min; skopeo is Linux/CI-only). + +**L4b cluster e2e** (`e2e-tests/playwright/e2e/**`) — only when the subject *is* cluster/platform behavior or a real external service. Requirements: `component` annotation in `beforeAll` (see `ci-e2e-testing` rule), correct config map choice (RBAC vs non-RBAC), project registration in `e2e-tests/playwright/projects.json` if a new project is needed. + +**L3 component tests** — pattern established on branch `RHIDP-13235-layer3-component-tests` (page-level RTL compositions). Prefer this over L4a when no dynamic-plugin loading is involved. + +**Catalog-index-wide sanity** (rhdh PR #4967, nightly) — nothing to write per plugin; it sweeps the whole index. + +## Not possible today (researched — don't burn time) + +- **Rendering a frontend dynamic plugin without an RHDH app** — the artifact is a legacy-frontend Scalprum bundle; no standalone host exists, and building one means maintaining a version-coupled "mini RHDH". +- **`@backstage/frontend-dynamic-feature-loader` for current plugins** — targets the new frontend system (alpha); our exported plugins are legacy-system bundles. Revisit when app-next matures (RHIDP-15082). +- **Catalog-extending modules in a minimal `startTestBackend`** — upstream `catalog-backend` issue (see overlays harness notes). + +## Output format + +Answer with a concrete recommendation: + +- **Repo:** which of the three (or the plugin's own source repo). +- **Layer / harness:** L1–L4b + the specific harness or suite. +- **Location:** the directory/file where the test goes. +- **Scaffolding:** the minimal steps or files to create, referencing an existing neighbor as template. +- **Why not elsewhere:** one line on the layers you rejected (especially if the dev proposed a more expensive one). +- **Cost:** rough feedback time (seconds / ~4 min cluster-free / cluster job). + +## References + +- Epic RHIDP-13501 (E2E Test Optimization) — per-repo responsibility split in the epic comments and the attached `rhdh-dynamic-plugin-testing-guideline.md`. +- `rhdh/docs/e2e-tests/layer-migration-matrix.md` — per-spec layer classification (RHIDP-15076). +- `rhdh/docs/e2e-tests/local-e2e-harness.md` — cluster-free harness usage. +- Reference PRs: overlays#2714 (native smoke), rhdh#4967 (catalog-index sanity), rhdh#5005 (cluster-free E2E harness). From e45106038a797c648273fa04c816633c152e90ec Mon Sep 17 00:00:00 2001 From: Gustavo Lira Date: Thu, 2 Jul 2026 16:51:31 -0300 Subject: [PATCH 2/2] =?UTF-8?q?docs(ai):=20address=20review=20=E2=80=94=20?= =?UTF-8?q?verify-before-recommend,=20real=20smoke=20invocation,=20path=20?= =?UTF-8?q?prefixes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Mark harnesses that are still in review and instruct the assistant to verify paths exist on main before recommending them (the skill referenced #5005/#5044/#2714/#4967 deliverables in present tense). - Describe the overlays native smoke's real interface: yarn smoke --dynamic-plugins [--out] (no --workspace flag exists; the harness does not read workspaces/*/metadata — a workspace run is a dp.yaml listing its oci:// refs). - Replace the fork-only RHIDP-13235-layer3-component-tests branch pointer with the closed PR rhdh#4864. - Prefix every path with its repo (rhdh:/overlays:/plugins:) and use full URLs for cross-repo docs, so the skill resolves from any of its three homes. - Fix skopeo claim: it installs on macOS via brew; CI has it preinstalled. - Name the Docker smoke location (overlays smoke-tests/ + run-workspace-smoke-tests.yaml), replace the vague "any" repo cell, use "n/a (cluster)" for the cluster row's Docker column, move PR numbers to References with status, add trigger phrases to the frontmatter description, and deduplicate the guiding-rule sentence. Co-Authored-By: Claude Opus 4.8 (1M context) --- .claude/skills/test-placement/SKILL.md | 58 +++++++++++++----------- .cursor/skills/test-placement/SKILL.md | 57 ++++++++++++----------- .opencode/skill/test-placement/SKILL.md | 58 +++++++++++++----------- .rulesync/skills/test-placement/SKILL.md | 58 +++++++++++++----------- 4 files changed, 127 insertions(+), 104 deletions(-) diff --git a/.claude/skills/test-placement/SKILL.md b/.claude/skills/test-placement/SKILL.md index 1bb4ee54da..ce40d34f25 100644 --- a/.claude/skills/test-placement/SKILL.md +++ b/.claude/skills/test-placement/SKILL.md @@ -4,12 +4,15 @@ description: >- Propose where to test a change in the RHDH dynamic-plugin ecosystem: which repo (rhdh-plugins, rhdh-plugin-export-overlays, rhdh), which test layer (unit, integration, component, cluster-free E2E, cluster E2E), where the test - lives, and how to create it + lives, and how to create it. Use when a developer asks where to test + something, whether a test needs a cluster, or which repo a test belongs in --- # Test Placement Advisor Given the context of a change, bug, or new feature in the RHDH dynamic-plugin ecosystem, propose **where** it should be tested: which repository, which test layer, where the test lives, and how to create it. The guiding rule: **pick the cheapest environment that can actually catch the bug — most plugin validation does not need a cluster, and an increasing part doesn't need Docker either.** +Conventions in this skill: paths are prefixed with the repo they live in — `rhdh:`, `overlays:` (= rhdh-plugin-export-overlays), `plugins:` (= rhdh-plugins). Some harnesses referenced here are **still in review** (see References for PR status). Before recommending a harness or script, verify its path exists on the target repo's `main`; if it doesn't, tell the developer it is pending in the corresponding PR instead of asserting it exists. + ## When to Use - A developer asks "where should I test this?", "should this be an e2e test?", "does this need a cluster?", or "which repo does this test belong in?" @@ -32,14 +35,14 @@ Before recommending, establish: | The dev wants to verify… | Repo | Test type / harness | Cluster? | Docker? | | --- | --- | --- | --- | --- | | Plugin logic / components / API | `rhdh-plugins` (or the plugin's source repo) | unit + component tests, dev app (`yarn start`) | no | no | -| The plugin still builds as a dynamic plugin | `rhdh-plugins` | `export-dynamic` / `npx @red-hat-developer-hub/cli plugin export` | no | no | -| The **published OCI artifact** installs and the **backend plugin boots** | `rhdh-plugin-export-overlays` | native smoke harness (`smoke-tests-native/`, overlays PR #2714) | no | **no** | -| All plugins of a workspace boot together | `rhdh-plugin-export-overlays` | native smoke `--workspace` mode | no | **no** | +| The plugin still builds as a dynamic plugin | `rhdh-plugins` | `npx @red-hat-developer-hub/cli plugin export` | no | no | +| The **published OCI artifact** installs and the **backend plugin boots** | `rhdh-plugin-export-overlays` | native smoke harness (`overlays: smoke-tests-native/`) | no | **no** | +| All plugins of a **workspace** boot together | `rhdh-plugin-export-overlays` | native smoke with a `dynamic-plugins.yaml` listing the workspace's `oci://` refs | no | **no** | | A frontend artifact ships its bundle (`dist-scalprum/`, `plugin-manifest.json`) | `rhdh-plugin-export-overlays` | native smoke presence check | no | **no** | -| The plugin **loads inside the real RHDH app** and the **UI renders** | `rhdh` | cluster-free E2E harness (`e2e:legacy-local`, rhdh PR #5005) | no | no | -| Every plugin in the **catalog index** is sane | `rhdh` | plugin sanity check (rhdh PR #4967, nightly) | no | no | -| RHDH **container** behavior (image entrypoint, install script inside the image) | any | Docker smoke / [rhdh-local](https://github.com/redhat-developer/rhdh-local) | no | yes | -| Helm chart / Operator / ingress / real Keycloak / RBAC on OCP | `rhdh` e2e or `rhdh-plugin-export-overlays` `workspaces/*/e2e-tests` | cluster e2e | **yes** | — | +| The plugin **loads inside the real RHDH app** and the **UI renders** | `rhdh` | cluster-free E2E harness (`rhdh: e2e-tests/playwright.legacy-local.config.ts`, script `e2e:legacy-local`) | no | no | +| Every plugin in the **catalog index** is sane | `rhdh` | catalog-index plugin sanity check (nightly) | no | no | +| RHDH **container** behavior (image entrypoint, install script inside the image) | `rhdh-plugin-export-overlays` (artifacts) / any (manual) | Docker smoke (`overlays: smoke-tests/` + workflow `run-workspace-smoke-tests.yaml`) · manual: [rhdh-local](https://github.com/redhat-developer/rhdh-local) | no | yes | +| Helm chart / Operator / ingress / real Keycloak / RBAC on OCP | `rhdh` e2e or `overlays: workspaces/*/e2e-tests` | cluster e2e | **yes** | n/a (cluster) | Rule of thumb: **source bugs → rhdh-plugins · artifact bugs → overlays · integration/render bugs → rhdh · platform bugs → cluster e2e.** If a bug is catchable in more than one place, test it in the cheapest one and don't duplicate downstream. @@ -53,39 +56,40 @@ Rule of thumb: **source bugs → rhdh-plugins · artifact bugs → overlays · i | **L4a** E2E cluster-free | Full app, no managed infra | Playwright + local harness | no | min | | **L4b** E2E full | Real OCP/K8s, managed DBs, real IdPs | Playwright + cluster | **yes** | min–h | -The full per-spec classification of the RHDH e2e suite lives in `rhdh/docs/e2e-tests/layer-migration-matrix.md` (RHIDP-15076) — consult it before adding or migrating an e2e spec. +The full per-spec classification of the RHDH e2e suite lives in `rhdh: docs/e2e-tests/layer-migration-matrix.md` (see References) — consult it before adding or migrating an e2e spec. ## Step 4 — How to create the test (per placement) ### `rhdh-plugins` (or other source repo) — plugin correctness -- **Where:** `workspaces//plugins//src/**` next to the code, following that workspace's existing test setup (Jest/RTL; MSW for API mocks). +- **Where:** `plugins: workspaces//plugins//src/**` next to the code, following that workspace's existing test setup (Jest/RTL; MSW for API mocks). - **How:** copy the pattern of a neighboring `*.test.ts(x)`; run with the workspace's `yarn test`. Manual verification via the workspace dev app (`yarn start`). - Also validate packaging when the plugin's build config changed: `npx @red-hat-developer-hub/cli plugin export` must produce `dist-dynamic/` (+ `dist-scalprum/` for frontend). - **Cannot do here:** validate the *published* OCI artifact, or integration with RHDH's app shell/shared deps. ### `rhdh-plugin-export-overlays` — the published artifact -- **Where:** the native smoke harness (`smoke-tests-native/`, overlays PR #2714). Per-plugin: install from OCI with the real `install-dynamic-plugins` CLI and boot backend plugins via `startTestBackend`. Workspace mode: `yarn smoke --workspace ` boots every `oci://` artifact of the workspace together. -- New workspaces usually need **no new test code** — the harness discovers artifacts from `workspaces//metadata/*.yaml`. -- **Known gaps (don't fight them):** catalog-extending backend modules don't boot in a minimal `startTestBackend` (upstream `catalog-backend` registration issue — stay on Docker smoke until fixed); plugins whose `dynamicArtifact` is a local `./dynamic-plugins/dist/...` path ship inside the RHDH image and have no OCI artifact to validate here. +- **Where:** the native smoke harness, `overlays: smoke-tests-native/` (in review — see References). It installs plugins from OCI with the real `install-dynamic-plugins` CLI and boots backend plugins via `startTestBackend`. +- **How:** write a `dynamic-plugins.yaml` listing the `oci://` artifact refs to validate (a workspace's refs are the `spec.dynamicArtifact` values in `overlays: workspaces//metadata/*.yaml`), then run `yarn smoke --dynamic-plugins [--out results.json]`. Exit code 0 = pass; non-zero writes `results.json` detailing `fail-load` / `fail-start`. CI: `overlays: .github/workflows/native-smoke.yaml`. +- **Known gaps (don't fight them):** catalog-extending backend modules don't boot in a minimal `startTestBackend` (upstream `catalog-backend` registration issue — stay on the Docker smoke until fixed); plugins whose `dynamicArtifact` is a local `./dynamic-plugins/dist/...` path ship inside the RHDH image and have no OCI artifact to validate here. - **Never add UI-render tests here** — this repo has no app to render into. Delegate rendering to the `rhdh` cluster-free harness. -- `workspaces/*/e2e-tests` (cluster, Playwright + `e2e-test-utils`) exists for plugins whose value *is* cluster integration (topology, tekton, argocd…). Don't add one for UI-only behavior. +- The Docker smoke it replaces (backend scope) lives at `overlays: smoke-tests/` + workflow `run-workspace-smoke-tests.yaml`. +- `overlays: workspaces/*/e2e-tests` (cluster, Playwright + `e2e-test-utils`) exists for plugins whose value *is* cluster integration (topology, tekton, argocd…). Don't add one for UI-only behavior. ### `rhdh` — the real app -**L4a cluster-free harness** (`e2e-tests/playwright.legacy-local.config.ts`, docs at `docs/e2e-tests/local-e2e-harness.md`) — the only cheap place a frontend dynamic plugin can be *rendered*. To enable a spec/test: +**L4a cluster-free harness** (in review — see References; config `rhdh: e2e-tests/playwright.legacy-local.config.ts`, docs `rhdh: docs/e2e-tests/local-e2e-harness.md`) — the only cheap place a frontend dynamic plugin can be *rendered*. To enable a spec/test: -1. Plugin not yet installed by the harness? Add its OCI entry to `e2e-tests/local-harness/dynamic-plugins.yaml` (tags on `ghcr.io/redhat-developer/rhdh-plugin-export-overlays/`). If its mount-point config is not in the repo's static `app-config.dynamic-plugins.yaml`, attach the plugin's **canonical `pluginConfig`** (source of truth: `rhdh-plugins` → `workspaces//plugins//app-config.dynamic.yaml`) — the harness loads the generated `dynamic-plugins-root/app-config.dynamic-plugins.yaml` last, exactly like the production container. -2. Config that only exists in CI config maps (`.ci/pipelines/resources/config_map/*`)? Mirror just the needed keys into `app-config.local-e2e.yaml` (objects deep-merge; arrays replace). +1. Plugin not yet installed by the harness? Add its OCI entry to `rhdh: e2e-tests/local-harness/dynamic-plugins.yaml` (tags on `ghcr.io/redhat-developer/rhdh-plugin-export-overlays/`). If its mount-point config is not in the repo's static `app-config.dynamic-plugins.yaml`, attach the plugin's **canonical `pluginConfig`** (source of truth: `plugins: workspaces//plugins//app-config.dynamic.yaml`) — the harness loads the generated `dynamic-plugins-root/app-config.dynamic-plugins.yaml` last, exactly like the production container. +2. Config that only exists in CI config maps (`rhdh: .ci/pipelines/resources/config_map/*`)? Mirror just the needed keys into `rhdh: app-config.local-e2e.yaml` (objects deep-merge; arrays replace). 3. Tag the test `{ tag: "@cluster-free" }` and add its spec file to the config's `testMatch` allowlist. -4. Repopulate + validate: `./e2e-tests/local-harness/populate.sh`, then `yarn --cwd e2e-tests e2e:legacy-local` (CI does this in ~4 min; skopeo is Linux/CI-only). +4. Repopulate + validate: `./e2e-tests/local-harness/populate.sh`, then `yarn --cwd e2e-tests e2e:legacy-local` (CI runs this in ~4 min). Requires skopeo — preinstalled in CI; on macOS `brew install skopeo`. -**L4b cluster e2e** (`e2e-tests/playwright/e2e/**`) — only when the subject *is* cluster/platform behavior or a real external service. Requirements: `component` annotation in `beforeAll` (see `ci-e2e-testing` rule), correct config map choice (RBAC vs non-RBAC), project registration in `e2e-tests/playwright/projects.json` if a new project is needed. +**L4b cluster e2e** (`rhdh: e2e-tests/playwright/e2e/**`) — only when the subject *is* cluster/platform behavior or a real external service. Requirements: `component` annotation in `beforeAll` (see the repo's `ci-e2e-testing` rule), correct config map choice (RBAC vs non-RBAC), project registration in `e2e-tests/playwright/projects.json` if a new project is needed. -**L3 component tests** — pattern established on branch `RHIDP-13235-layer3-component-tests` (page-level RTL compositions). Prefer this over L4a when no dynamic-plugin loading is involved. +**L3 component tests** — page-level RTL compositions, pattern proposed under RHIDP-13235 in [rhdh#4864](https://github.com/redhat-developer/rhdh/pull/4864) (closed, not merged). Check for merged examples before recommending a concrete template; prefer L3 over L4a when no dynamic-plugin loading is involved. -**Catalog-index-wide sanity** (rhdh PR #4967, nightly) — nothing to write per plugin; it sweeps the whole index. +**Catalog-index-wide sanity** (nightly, in review — see References) — nothing to write per plugin; it sweeps the whole index. ## Not possible today (researched — don't burn time) @@ -104,9 +108,11 @@ Answer with a concrete recommendation: - **Why not elsewhere:** one line on the layers you rejected (especially if the dev proposed a more expensive one). - **Cost:** rough feedback time (seconds / ~4 min cluster-free / cluster job). -## References +## References (status as of 2026-07-02 — verify before citing as merged) -- Epic RHIDP-13501 (E2E Test Optimization) — per-repo responsibility split in the epic comments and the attached `rhdh-dynamic-plugin-testing-guideline.md`. -- `rhdh/docs/e2e-tests/layer-migration-matrix.md` — per-spec layer classification (RHIDP-15076). -- `rhdh/docs/e2e-tests/local-e2e-harness.md` — cluster-free harness usage. -- Reference PRs: overlays#2714 (native smoke), rhdh#4967 (catalog-index sanity), rhdh#5005 (cluster-free E2E harness). +- Epic **RHIDP-13501** (E2E Test Optimization) — the per-repo responsibility split lives in the epic's comments and its Jira attachment `rhdh-dynamic-plugin-testing-guideline.md` (Jira-only; if unreachable, the decision table above is the summary). +- Layer matrix: — in review in [rhdh#5044](https://github.com/redhat-developer/rhdh/pull/5044) (RHIDP-15076). +- Cluster-free harness + docs: in review in [rhdh#5005](https://github.com/redhat-developer/rhdh/pull/5005) (RHIDP-15075). +- Overlays native smoke: in review in [overlays#2714](https://github.com/redhat-developer/rhdh-plugin-export-overlays/pull/2714). +- Catalog-index sanity check: in review in [rhdh#4967](https://github.com/redhat-developer/rhdh/pull/4967). +- L3 pattern proposal: [rhdh#4864](https://github.com/redhat-developer/rhdh/pull/4864) (closed) under RHIDP-13235. diff --git a/.cursor/skills/test-placement/SKILL.md b/.cursor/skills/test-placement/SKILL.md index 64d14eb430..ae76c96f60 100644 --- a/.cursor/skills/test-placement/SKILL.md +++ b/.cursor/skills/test-placement/SKILL.md @@ -1,11 +1,13 @@ --- name: test-placement -description: 'Propose where to test a change in the RHDH dynamic-plugin ecosystem: which repo (rhdh-plugins, rhdh-plugin-export-overlays, rhdh), which test layer (unit, integration, component, cluster-free E2E, cluster E2E), where the test lives, and how to create it' +description: 'Propose where to test a change in the RHDH dynamic-plugin ecosystem: which repo (rhdh-plugins, rhdh-plugin-export-overlays, rhdh), which test layer (unit, integration, component, cluster-free E2E, cluster E2E), where the test lives, and how to create it. Use when a developer asks where to test something, whether a test needs a cluster, or which repo a test belongs in' --- # Test Placement Advisor Given the context of a change, bug, or new feature in the RHDH dynamic-plugin ecosystem, propose **where** it should be tested: which repository, which test layer, where the test lives, and how to create it. The guiding rule: **pick the cheapest environment that can actually catch the bug — most plugin validation does not need a cluster, and an increasing part doesn't need Docker either.** +Conventions in this skill: paths are prefixed with the repo they live in — `rhdh:`, `overlays:` (= rhdh-plugin-export-overlays), `plugins:` (= rhdh-plugins). Some harnesses referenced here are **still in review** (see References for PR status). Before recommending a harness or script, verify its path exists on the target repo's `main`; if it doesn't, tell the developer it is pending in the corresponding PR instead of asserting it exists. + ## When to Use - A developer asks "where should I test this?", "should this be an e2e test?", "does this need a cluster?", or "which repo does this test belong in?" @@ -28,14 +30,14 @@ Before recommending, establish: | The dev wants to verify… | Repo | Test type / harness | Cluster? | Docker? | | --- | --- | --- | --- | --- | | Plugin logic / components / API | `rhdh-plugins` (or the plugin's source repo) | unit + component tests, dev app (`yarn start`) | no | no | -| The plugin still builds as a dynamic plugin | `rhdh-plugins` | `export-dynamic` / `npx @red-hat-developer-hub/cli plugin export` | no | no | -| The **published OCI artifact** installs and the **backend plugin boots** | `rhdh-plugin-export-overlays` | native smoke harness (`smoke-tests-native/`, overlays PR #2714) | no | **no** | -| All plugins of a workspace boot together | `rhdh-plugin-export-overlays` | native smoke `--workspace` mode | no | **no** | +| The plugin still builds as a dynamic plugin | `rhdh-plugins` | `npx @red-hat-developer-hub/cli plugin export` | no | no | +| The **published OCI artifact** installs and the **backend plugin boots** | `rhdh-plugin-export-overlays` | native smoke harness (`overlays: smoke-tests-native/`) | no | **no** | +| All plugins of a **workspace** boot together | `rhdh-plugin-export-overlays` | native smoke with a `dynamic-plugins.yaml` listing the workspace's `oci://` refs | no | **no** | | A frontend artifact ships its bundle (`dist-scalprum/`, `plugin-manifest.json`) | `rhdh-plugin-export-overlays` | native smoke presence check | no | **no** | -| The plugin **loads inside the real RHDH app** and the **UI renders** | `rhdh` | cluster-free E2E harness (`e2e:legacy-local`, rhdh PR #5005) | no | no | -| Every plugin in the **catalog index** is sane | `rhdh` | plugin sanity check (rhdh PR #4967, nightly) | no | no | -| RHDH **container** behavior (image entrypoint, install script inside the image) | any | Docker smoke / [rhdh-local](https://github.com/redhat-developer/rhdh-local) | no | yes | -| Helm chart / Operator / ingress / real Keycloak / RBAC on OCP | `rhdh` e2e or `rhdh-plugin-export-overlays` `workspaces/*/e2e-tests` | cluster e2e | **yes** | — | +| The plugin **loads inside the real RHDH app** and the **UI renders** | `rhdh` | cluster-free E2E harness (`rhdh: e2e-tests/playwright.legacy-local.config.ts`, script `e2e:legacy-local`) | no | no | +| Every plugin in the **catalog index** is sane | `rhdh` | catalog-index plugin sanity check (nightly) | no | no | +| RHDH **container** behavior (image entrypoint, install script inside the image) | `rhdh-plugin-export-overlays` (artifacts) / any (manual) | Docker smoke (`overlays: smoke-tests/` + workflow `run-workspace-smoke-tests.yaml`) · manual: [rhdh-local](https://github.com/redhat-developer/rhdh-local) | no | yes | +| Helm chart / Operator / ingress / real Keycloak / RBAC on OCP | `rhdh` e2e or `overlays: workspaces/*/e2e-tests` | cluster e2e | **yes** | n/a (cluster) | Rule of thumb: **source bugs → rhdh-plugins · artifact bugs → overlays · integration/render bugs → rhdh · platform bugs → cluster e2e.** If a bug is catchable in more than one place, test it in the cheapest one and don't duplicate downstream. @@ -49,39 +51,40 @@ Rule of thumb: **source bugs → rhdh-plugins · artifact bugs → overlays · i | **L4a** E2E cluster-free | Full app, no managed infra | Playwright + local harness | no | min | | **L4b** E2E full | Real OCP/K8s, managed DBs, real IdPs | Playwright + cluster | **yes** | min–h | -The full per-spec classification of the RHDH e2e suite lives in `rhdh/docs/e2e-tests/layer-migration-matrix.md` (RHIDP-15076) — consult it before adding or migrating an e2e spec. +The full per-spec classification of the RHDH e2e suite lives in `rhdh: docs/e2e-tests/layer-migration-matrix.md` (see References) — consult it before adding or migrating an e2e spec. ## Step 4 — How to create the test (per placement) ### `rhdh-plugins` (or other source repo) — plugin correctness -- **Where:** `workspaces//plugins//src/**` next to the code, following that workspace's existing test setup (Jest/RTL; MSW for API mocks). +- **Where:** `plugins: workspaces//plugins//src/**` next to the code, following that workspace's existing test setup (Jest/RTL; MSW for API mocks). - **How:** copy the pattern of a neighboring `*.test.ts(x)`; run with the workspace's `yarn test`. Manual verification via the workspace dev app (`yarn start`). - Also validate packaging when the plugin's build config changed: `npx @red-hat-developer-hub/cli plugin export` must produce `dist-dynamic/` (+ `dist-scalprum/` for frontend). - **Cannot do here:** validate the *published* OCI artifact, or integration with RHDH's app shell/shared deps. ### `rhdh-plugin-export-overlays` — the published artifact -- **Where:** the native smoke harness (`smoke-tests-native/`, overlays PR #2714). Per-plugin: install from OCI with the real `install-dynamic-plugins` CLI and boot backend plugins via `startTestBackend`. Workspace mode: `yarn smoke --workspace ` boots every `oci://` artifact of the workspace together. -- New workspaces usually need **no new test code** — the harness discovers artifacts from `workspaces//metadata/*.yaml`. -- **Known gaps (don't fight them):** catalog-extending backend modules don't boot in a minimal `startTestBackend` (upstream `catalog-backend` registration issue — stay on Docker smoke until fixed); plugins whose `dynamicArtifact` is a local `./dynamic-plugins/dist/...` path ship inside the RHDH image and have no OCI artifact to validate here. +- **Where:** the native smoke harness, `overlays: smoke-tests-native/` (in review — see References). It installs plugins from OCI with the real `install-dynamic-plugins` CLI and boots backend plugins via `startTestBackend`. +- **How:** write a `dynamic-plugins.yaml` listing the `oci://` artifact refs to validate (a workspace's refs are the `spec.dynamicArtifact` values in `overlays: workspaces//metadata/*.yaml`), then run `yarn smoke --dynamic-plugins [--out results.json]`. Exit code 0 = pass; non-zero writes `results.json` detailing `fail-load` / `fail-start`. CI: `overlays: .github/workflows/native-smoke.yaml`. +- **Known gaps (don't fight them):** catalog-extending backend modules don't boot in a minimal `startTestBackend` (upstream `catalog-backend` registration issue — stay on the Docker smoke until fixed); plugins whose `dynamicArtifact` is a local `./dynamic-plugins/dist/...` path ship inside the RHDH image and have no OCI artifact to validate here. - **Never add UI-render tests here** — this repo has no app to render into. Delegate rendering to the `rhdh` cluster-free harness. -- `workspaces/*/e2e-tests` (cluster, Playwright + `e2e-test-utils`) exists for plugins whose value *is* cluster integration (topology, tekton, argocd…). Don't add one for UI-only behavior. +- The Docker smoke it replaces (backend scope) lives at `overlays: smoke-tests/` + workflow `run-workspace-smoke-tests.yaml`. +- `overlays: workspaces/*/e2e-tests` (cluster, Playwright + `e2e-test-utils`) exists for plugins whose value *is* cluster integration (topology, tekton, argocd…). Don't add one for UI-only behavior. ### `rhdh` — the real app -**L4a cluster-free harness** (`e2e-tests/playwright.legacy-local.config.ts`, docs at `docs/e2e-tests/local-e2e-harness.md`) — the only cheap place a frontend dynamic plugin can be *rendered*. To enable a spec/test: +**L4a cluster-free harness** (in review — see References; config `rhdh: e2e-tests/playwright.legacy-local.config.ts`, docs `rhdh: docs/e2e-tests/local-e2e-harness.md`) — the only cheap place a frontend dynamic plugin can be *rendered*. To enable a spec/test: -1. Plugin not yet installed by the harness? Add its OCI entry to `e2e-tests/local-harness/dynamic-plugins.yaml` (tags on `ghcr.io/redhat-developer/rhdh-plugin-export-overlays/`). If its mount-point config is not in the repo's static `app-config.dynamic-plugins.yaml`, attach the plugin's **canonical `pluginConfig`** (source of truth: `rhdh-plugins` → `workspaces//plugins//app-config.dynamic.yaml`) — the harness loads the generated `dynamic-plugins-root/app-config.dynamic-plugins.yaml` last, exactly like the production container. -2. Config that only exists in CI config maps (`.ci/pipelines/resources/config_map/*`)? Mirror just the needed keys into `app-config.local-e2e.yaml` (objects deep-merge; arrays replace). +1. Plugin not yet installed by the harness? Add its OCI entry to `rhdh: e2e-tests/local-harness/dynamic-plugins.yaml` (tags on `ghcr.io/redhat-developer/rhdh-plugin-export-overlays/`). If its mount-point config is not in the repo's static `app-config.dynamic-plugins.yaml`, attach the plugin's **canonical `pluginConfig`** (source of truth: `plugins: workspaces//plugins//app-config.dynamic.yaml`) — the harness loads the generated `dynamic-plugins-root/app-config.dynamic-plugins.yaml` last, exactly like the production container. +2. Config that only exists in CI config maps (`rhdh: .ci/pipelines/resources/config_map/*`)? Mirror just the needed keys into `rhdh: app-config.local-e2e.yaml` (objects deep-merge; arrays replace). 3. Tag the test `{ tag: "@cluster-free" }` and add its spec file to the config's `testMatch` allowlist. -4. Repopulate + validate: `./e2e-tests/local-harness/populate.sh`, then `yarn --cwd e2e-tests e2e:legacy-local` (CI does this in ~4 min; skopeo is Linux/CI-only). +4. Repopulate + validate: `./e2e-tests/local-harness/populate.sh`, then `yarn --cwd e2e-tests e2e:legacy-local` (CI runs this in ~4 min). Requires skopeo — preinstalled in CI; on macOS `brew install skopeo`. -**L4b cluster e2e** (`e2e-tests/playwright/e2e/**`) — only when the subject *is* cluster/platform behavior or a real external service. Requirements: `component` annotation in `beforeAll` (see `ci-e2e-testing` rule), correct config map choice (RBAC vs non-RBAC), project registration in `e2e-tests/playwright/projects.json` if a new project is needed. +**L4b cluster e2e** (`rhdh: e2e-tests/playwright/e2e/**`) — only when the subject *is* cluster/platform behavior or a real external service. Requirements: `component` annotation in `beforeAll` (see the repo's `ci-e2e-testing` rule), correct config map choice (RBAC vs non-RBAC), project registration in `e2e-tests/playwright/projects.json` if a new project is needed. -**L3 component tests** — pattern established on branch `RHIDP-13235-layer3-component-tests` (page-level RTL compositions). Prefer this over L4a when no dynamic-plugin loading is involved. +**L3 component tests** — page-level RTL compositions, pattern proposed under RHIDP-13235 in [rhdh#4864](https://github.com/redhat-developer/rhdh/pull/4864) (closed, not merged). Check for merged examples before recommending a concrete template; prefer L3 over L4a when no dynamic-plugin loading is involved. -**Catalog-index-wide sanity** (rhdh PR #4967, nightly) — nothing to write per plugin; it sweeps the whole index. +**Catalog-index-wide sanity** (nightly, in review — see References) — nothing to write per plugin; it sweeps the whole index. ## Not possible today (researched — don't burn time) @@ -100,9 +103,11 @@ Answer with a concrete recommendation: - **Why not elsewhere:** one line on the layers you rejected (especially if the dev proposed a more expensive one). - **Cost:** rough feedback time (seconds / ~4 min cluster-free / cluster job). -## References +## References (status as of 2026-07-02 — verify before citing as merged) -- Epic RHIDP-13501 (E2E Test Optimization) — per-repo responsibility split in the epic comments and the attached `rhdh-dynamic-plugin-testing-guideline.md`. -- `rhdh/docs/e2e-tests/layer-migration-matrix.md` — per-spec layer classification (RHIDP-15076). -- `rhdh/docs/e2e-tests/local-e2e-harness.md` — cluster-free harness usage. -- Reference PRs: overlays#2714 (native smoke), rhdh#4967 (catalog-index sanity), rhdh#5005 (cluster-free E2E harness). +- Epic **RHIDP-13501** (E2E Test Optimization) — the per-repo responsibility split lives in the epic's comments and its Jira attachment `rhdh-dynamic-plugin-testing-guideline.md` (Jira-only; if unreachable, the decision table above is the summary). +- Layer matrix: — in review in [rhdh#5044](https://github.com/redhat-developer/rhdh/pull/5044) (RHIDP-15076). +- Cluster-free harness + docs: in review in [rhdh#5005](https://github.com/redhat-developer/rhdh/pull/5005) (RHIDP-15075). +- Overlays native smoke: in review in [overlays#2714](https://github.com/redhat-developer/rhdh-plugin-export-overlays/pull/2714). +- Catalog-index sanity check: in review in [rhdh#4967](https://github.com/redhat-developer/rhdh/pull/4967). +- L3 pattern proposal: [rhdh#4864](https://github.com/redhat-developer/rhdh/pull/4864) (closed) under RHIDP-13235. diff --git a/.opencode/skill/test-placement/SKILL.md b/.opencode/skill/test-placement/SKILL.md index 1bb4ee54da..ce40d34f25 100644 --- a/.opencode/skill/test-placement/SKILL.md +++ b/.opencode/skill/test-placement/SKILL.md @@ -4,12 +4,15 @@ description: >- Propose where to test a change in the RHDH dynamic-plugin ecosystem: which repo (rhdh-plugins, rhdh-plugin-export-overlays, rhdh), which test layer (unit, integration, component, cluster-free E2E, cluster E2E), where the test - lives, and how to create it + lives, and how to create it. Use when a developer asks where to test + something, whether a test needs a cluster, or which repo a test belongs in --- # Test Placement Advisor Given the context of a change, bug, or new feature in the RHDH dynamic-plugin ecosystem, propose **where** it should be tested: which repository, which test layer, where the test lives, and how to create it. The guiding rule: **pick the cheapest environment that can actually catch the bug — most plugin validation does not need a cluster, and an increasing part doesn't need Docker either.** +Conventions in this skill: paths are prefixed with the repo they live in — `rhdh:`, `overlays:` (= rhdh-plugin-export-overlays), `plugins:` (= rhdh-plugins). Some harnesses referenced here are **still in review** (see References for PR status). Before recommending a harness or script, verify its path exists on the target repo's `main`; if it doesn't, tell the developer it is pending in the corresponding PR instead of asserting it exists. + ## When to Use - A developer asks "where should I test this?", "should this be an e2e test?", "does this need a cluster?", or "which repo does this test belong in?" @@ -32,14 +35,14 @@ Before recommending, establish: | The dev wants to verify… | Repo | Test type / harness | Cluster? | Docker? | | --- | --- | --- | --- | --- | | Plugin logic / components / API | `rhdh-plugins` (or the plugin's source repo) | unit + component tests, dev app (`yarn start`) | no | no | -| The plugin still builds as a dynamic plugin | `rhdh-plugins` | `export-dynamic` / `npx @red-hat-developer-hub/cli plugin export` | no | no | -| The **published OCI artifact** installs and the **backend plugin boots** | `rhdh-plugin-export-overlays` | native smoke harness (`smoke-tests-native/`, overlays PR #2714) | no | **no** | -| All plugins of a workspace boot together | `rhdh-plugin-export-overlays` | native smoke `--workspace` mode | no | **no** | +| The plugin still builds as a dynamic plugin | `rhdh-plugins` | `npx @red-hat-developer-hub/cli plugin export` | no | no | +| The **published OCI artifact** installs and the **backend plugin boots** | `rhdh-plugin-export-overlays` | native smoke harness (`overlays: smoke-tests-native/`) | no | **no** | +| All plugins of a **workspace** boot together | `rhdh-plugin-export-overlays` | native smoke with a `dynamic-plugins.yaml` listing the workspace's `oci://` refs | no | **no** | | A frontend artifact ships its bundle (`dist-scalprum/`, `plugin-manifest.json`) | `rhdh-plugin-export-overlays` | native smoke presence check | no | **no** | -| The plugin **loads inside the real RHDH app** and the **UI renders** | `rhdh` | cluster-free E2E harness (`e2e:legacy-local`, rhdh PR #5005) | no | no | -| Every plugin in the **catalog index** is sane | `rhdh` | plugin sanity check (rhdh PR #4967, nightly) | no | no | -| RHDH **container** behavior (image entrypoint, install script inside the image) | any | Docker smoke / [rhdh-local](https://github.com/redhat-developer/rhdh-local) | no | yes | -| Helm chart / Operator / ingress / real Keycloak / RBAC on OCP | `rhdh` e2e or `rhdh-plugin-export-overlays` `workspaces/*/e2e-tests` | cluster e2e | **yes** | — | +| The plugin **loads inside the real RHDH app** and the **UI renders** | `rhdh` | cluster-free E2E harness (`rhdh: e2e-tests/playwright.legacy-local.config.ts`, script `e2e:legacy-local`) | no | no | +| Every plugin in the **catalog index** is sane | `rhdh` | catalog-index plugin sanity check (nightly) | no | no | +| RHDH **container** behavior (image entrypoint, install script inside the image) | `rhdh-plugin-export-overlays` (artifacts) / any (manual) | Docker smoke (`overlays: smoke-tests/` + workflow `run-workspace-smoke-tests.yaml`) · manual: [rhdh-local](https://github.com/redhat-developer/rhdh-local) | no | yes | +| Helm chart / Operator / ingress / real Keycloak / RBAC on OCP | `rhdh` e2e or `overlays: workspaces/*/e2e-tests` | cluster e2e | **yes** | n/a (cluster) | Rule of thumb: **source bugs → rhdh-plugins · artifact bugs → overlays · integration/render bugs → rhdh · platform bugs → cluster e2e.** If a bug is catchable in more than one place, test it in the cheapest one and don't duplicate downstream. @@ -53,39 +56,40 @@ Rule of thumb: **source bugs → rhdh-plugins · artifact bugs → overlays · i | **L4a** E2E cluster-free | Full app, no managed infra | Playwright + local harness | no | min | | **L4b** E2E full | Real OCP/K8s, managed DBs, real IdPs | Playwright + cluster | **yes** | min–h | -The full per-spec classification of the RHDH e2e suite lives in `rhdh/docs/e2e-tests/layer-migration-matrix.md` (RHIDP-15076) — consult it before adding or migrating an e2e spec. +The full per-spec classification of the RHDH e2e suite lives in `rhdh: docs/e2e-tests/layer-migration-matrix.md` (see References) — consult it before adding or migrating an e2e spec. ## Step 4 — How to create the test (per placement) ### `rhdh-plugins` (or other source repo) — plugin correctness -- **Where:** `workspaces//plugins//src/**` next to the code, following that workspace's existing test setup (Jest/RTL; MSW for API mocks). +- **Where:** `plugins: workspaces//plugins//src/**` next to the code, following that workspace's existing test setup (Jest/RTL; MSW for API mocks). - **How:** copy the pattern of a neighboring `*.test.ts(x)`; run with the workspace's `yarn test`. Manual verification via the workspace dev app (`yarn start`). - Also validate packaging when the plugin's build config changed: `npx @red-hat-developer-hub/cli plugin export` must produce `dist-dynamic/` (+ `dist-scalprum/` for frontend). - **Cannot do here:** validate the *published* OCI artifact, or integration with RHDH's app shell/shared deps. ### `rhdh-plugin-export-overlays` — the published artifact -- **Where:** the native smoke harness (`smoke-tests-native/`, overlays PR #2714). Per-plugin: install from OCI with the real `install-dynamic-plugins` CLI and boot backend plugins via `startTestBackend`. Workspace mode: `yarn smoke --workspace ` boots every `oci://` artifact of the workspace together. -- New workspaces usually need **no new test code** — the harness discovers artifacts from `workspaces//metadata/*.yaml`. -- **Known gaps (don't fight them):** catalog-extending backend modules don't boot in a minimal `startTestBackend` (upstream `catalog-backend` registration issue — stay on Docker smoke until fixed); plugins whose `dynamicArtifact` is a local `./dynamic-plugins/dist/...` path ship inside the RHDH image and have no OCI artifact to validate here. +- **Where:** the native smoke harness, `overlays: smoke-tests-native/` (in review — see References). It installs plugins from OCI with the real `install-dynamic-plugins` CLI and boots backend plugins via `startTestBackend`. +- **How:** write a `dynamic-plugins.yaml` listing the `oci://` artifact refs to validate (a workspace's refs are the `spec.dynamicArtifact` values in `overlays: workspaces//metadata/*.yaml`), then run `yarn smoke --dynamic-plugins [--out results.json]`. Exit code 0 = pass; non-zero writes `results.json` detailing `fail-load` / `fail-start`. CI: `overlays: .github/workflows/native-smoke.yaml`. +- **Known gaps (don't fight them):** catalog-extending backend modules don't boot in a minimal `startTestBackend` (upstream `catalog-backend` registration issue — stay on the Docker smoke until fixed); plugins whose `dynamicArtifact` is a local `./dynamic-plugins/dist/...` path ship inside the RHDH image and have no OCI artifact to validate here. - **Never add UI-render tests here** — this repo has no app to render into. Delegate rendering to the `rhdh` cluster-free harness. -- `workspaces/*/e2e-tests` (cluster, Playwright + `e2e-test-utils`) exists for plugins whose value *is* cluster integration (topology, tekton, argocd…). Don't add one for UI-only behavior. +- The Docker smoke it replaces (backend scope) lives at `overlays: smoke-tests/` + workflow `run-workspace-smoke-tests.yaml`. +- `overlays: workspaces/*/e2e-tests` (cluster, Playwright + `e2e-test-utils`) exists for plugins whose value *is* cluster integration (topology, tekton, argocd…). Don't add one for UI-only behavior. ### `rhdh` — the real app -**L4a cluster-free harness** (`e2e-tests/playwright.legacy-local.config.ts`, docs at `docs/e2e-tests/local-e2e-harness.md`) — the only cheap place a frontend dynamic plugin can be *rendered*. To enable a spec/test: +**L4a cluster-free harness** (in review — see References; config `rhdh: e2e-tests/playwright.legacy-local.config.ts`, docs `rhdh: docs/e2e-tests/local-e2e-harness.md`) — the only cheap place a frontend dynamic plugin can be *rendered*. To enable a spec/test: -1. Plugin not yet installed by the harness? Add its OCI entry to `e2e-tests/local-harness/dynamic-plugins.yaml` (tags on `ghcr.io/redhat-developer/rhdh-plugin-export-overlays/`). If its mount-point config is not in the repo's static `app-config.dynamic-plugins.yaml`, attach the plugin's **canonical `pluginConfig`** (source of truth: `rhdh-plugins` → `workspaces//plugins//app-config.dynamic.yaml`) — the harness loads the generated `dynamic-plugins-root/app-config.dynamic-plugins.yaml` last, exactly like the production container. -2. Config that only exists in CI config maps (`.ci/pipelines/resources/config_map/*`)? Mirror just the needed keys into `app-config.local-e2e.yaml` (objects deep-merge; arrays replace). +1. Plugin not yet installed by the harness? Add its OCI entry to `rhdh: e2e-tests/local-harness/dynamic-plugins.yaml` (tags on `ghcr.io/redhat-developer/rhdh-plugin-export-overlays/`). If its mount-point config is not in the repo's static `app-config.dynamic-plugins.yaml`, attach the plugin's **canonical `pluginConfig`** (source of truth: `plugins: workspaces//plugins//app-config.dynamic.yaml`) — the harness loads the generated `dynamic-plugins-root/app-config.dynamic-plugins.yaml` last, exactly like the production container. +2. Config that only exists in CI config maps (`rhdh: .ci/pipelines/resources/config_map/*`)? Mirror just the needed keys into `rhdh: app-config.local-e2e.yaml` (objects deep-merge; arrays replace). 3. Tag the test `{ tag: "@cluster-free" }` and add its spec file to the config's `testMatch` allowlist. -4. Repopulate + validate: `./e2e-tests/local-harness/populate.sh`, then `yarn --cwd e2e-tests e2e:legacy-local` (CI does this in ~4 min; skopeo is Linux/CI-only). +4. Repopulate + validate: `./e2e-tests/local-harness/populate.sh`, then `yarn --cwd e2e-tests e2e:legacy-local` (CI runs this in ~4 min). Requires skopeo — preinstalled in CI; on macOS `brew install skopeo`. -**L4b cluster e2e** (`e2e-tests/playwright/e2e/**`) — only when the subject *is* cluster/platform behavior or a real external service. Requirements: `component` annotation in `beforeAll` (see `ci-e2e-testing` rule), correct config map choice (RBAC vs non-RBAC), project registration in `e2e-tests/playwright/projects.json` if a new project is needed. +**L4b cluster e2e** (`rhdh: e2e-tests/playwright/e2e/**`) — only when the subject *is* cluster/platform behavior or a real external service. Requirements: `component` annotation in `beforeAll` (see the repo's `ci-e2e-testing` rule), correct config map choice (RBAC vs non-RBAC), project registration in `e2e-tests/playwright/projects.json` if a new project is needed. -**L3 component tests** — pattern established on branch `RHIDP-13235-layer3-component-tests` (page-level RTL compositions). Prefer this over L4a when no dynamic-plugin loading is involved. +**L3 component tests** — page-level RTL compositions, pattern proposed under RHIDP-13235 in [rhdh#4864](https://github.com/redhat-developer/rhdh/pull/4864) (closed, not merged). Check for merged examples before recommending a concrete template; prefer L3 over L4a when no dynamic-plugin loading is involved. -**Catalog-index-wide sanity** (rhdh PR #4967, nightly) — nothing to write per plugin; it sweeps the whole index. +**Catalog-index-wide sanity** (nightly, in review — see References) — nothing to write per plugin; it sweeps the whole index. ## Not possible today (researched — don't burn time) @@ -104,9 +108,11 @@ Answer with a concrete recommendation: - **Why not elsewhere:** one line on the layers you rejected (especially if the dev proposed a more expensive one). - **Cost:** rough feedback time (seconds / ~4 min cluster-free / cluster job). -## References +## References (status as of 2026-07-02 — verify before citing as merged) -- Epic RHIDP-13501 (E2E Test Optimization) — per-repo responsibility split in the epic comments and the attached `rhdh-dynamic-plugin-testing-guideline.md`. -- `rhdh/docs/e2e-tests/layer-migration-matrix.md` — per-spec layer classification (RHIDP-15076). -- `rhdh/docs/e2e-tests/local-e2e-harness.md` — cluster-free harness usage. -- Reference PRs: overlays#2714 (native smoke), rhdh#4967 (catalog-index sanity), rhdh#5005 (cluster-free E2E harness). +- Epic **RHIDP-13501** (E2E Test Optimization) — the per-repo responsibility split lives in the epic's comments and its Jira attachment `rhdh-dynamic-plugin-testing-guideline.md` (Jira-only; if unreachable, the decision table above is the summary). +- Layer matrix: — in review in [rhdh#5044](https://github.com/redhat-developer/rhdh/pull/5044) (RHIDP-15076). +- Cluster-free harness + docs: in review in [rhdh#5005](https://github.com/redhat-developer/rhdh/pull/5005) (RHIDP-15075). +- Overlays native smoke: in review in [overlays#2714](https://github.com/redhat-developer/rhdh-plugin-export-overlays/pull/2714). +- Catalog-index sanity check: in review in [rhdh#4967](https://github.com/redhat-developer/rhdh/pull/4967). +- L3 pattern proposal: [rhdh#4864](https://github.com/redhat-developer/rhdh/pull/4864) (closed) under RHIDP-13235. diff --git a/.rulesync/skills/test-placement/SKILL.md b/.rulesync/skills/test-placement/SKILL.md index fc57d45701..332bc33d23 100644 --- a/.rulesync/skills/test-placement/SKILL.md +++ b/.rulesync/skills/test-placement/SKILL.md @@ -4,7 +4,8 @@ description: >- Propose where to test a change in the RHDH dynamic-plugin ecosystem: which repo (rhdh-plugins, rhdh-plugin-export-overlays, rhdh), which test layer (unit, integration, component, cluster-free E2E, cluster E2E), where the - test lives, and how to create it + test lives, and how to create it. Use when a developer asks where to test + something, whether a test needs a cluster, or which repo a test belongs in targets: - '*' --- @@ -12,6 +13,8 @@ targets: Given the context of a change, bug, or new feature in the RHDH dynamic-plugin ecosystem, propose **where** it should be tested: which repository, which test layer, where the test lives, and how to create it. The guiding rule: **pick the cheapest environment that can actually catch the bug — most plugin validation does not need a cluster, and an increasing part doesn't need Docker either.** +Conventions in this skill: paths are prefixed with the repo they live in — `rhdh:`, `overlays:` (= rhdh-plugin-export-overlays), `plugins:` (= rhdh-plugins). Some harnesses referenced here are **still in review** (see References for PR status). Before recommending a harness or script, verify its path exists on the target repo's `main`; if it doesn't, tell the developer it is pending in the corresponding PR instead of asserting it exists. + ## When to Use - A developer asks "where should I test this?", "should this be an e2e test?", "does this need a cluster?", or "which repo does this test belong in?" @@ -34,14 +37,14 @@ Before recommending, establish: | The dev wants to verify… | Repo | Test type / harness | Cluster? | Docker? | | --- | --- | --- | --- | --- | | Plugin logic / components / API | `rhdh-plugins` (or the plugin's source repo) | unit + component tests, dev app (`yarn start`) | no | no | -| The plugin still builds as a dynamic plugin | `rhdh-plugins` | `export-dynamic` / `npx @red-hat-developer-hub/cli plugin export` | no | no | -| The **published OCI artifact** installs and the **backend plugin boots** | `rhdh-plugin-export-overlays` | native smoke harness (`smoke-tests-native/`, overlays PR #2714) | no | **no** | -| All plugins of a workspace boot together | `rhdh-plugin-export-overlays` | native smoke `--workspace` mode | no | **no** | +| The plugin still builds as a dynamic plugin | `rhdh-plugins` | `npx @red-hat-developer-hub/cli plugin export` | no | no | +| The **published OCI artifact** installs and the **backend plugin boots** | `rhdh-plugin-export-overlays` | native smoke harness (`overlays: smoke-tests-native/`) | no | **no** | +| All plugins of a **workspace** boot together | `rhdh-plugin-export-overlays` | native smoke with a `dynamic-plugins.yaml` listing the workspace's `oci://` refs | no | **no** | | A frontend artifact ships its bundle (`dist-scalprum/`, `plugin-manifest.json`) | `rhdh-plugin-export-overlays` | native smoke presence check | no | **no** | -| The plugin **loads inside the real RHDH app** and the **UI renders** | `rhdh` | cluster-free E2E harness (`e2e:legacy-local`, rhdh PR #5005) | no | no | -| Every plugin in the **catalog index** is sane | `rhdh` | plugin sanity check (rhdh PR #4967, nightly) | no | no | -| RHDH **container** behavior (image entrypoint, install script inside the image) | any | Docker smoke / [rhdh-local](https://github.com/redhat-developer/rhdh-local) | no | yes | -| Helm chart / Operator / ingress / real Keycloak / RBAC on OCP | `rhdh` e2e or `rhdh-plugin-export-overlays` `workspaces/*/e2e-tests` | cluster e2e | **yes** | — | +| The plugin **loads inside the real RHDH app** and the **UI renders** | `rhdh` | cluster-free E2E harness (`rhdh: e2e-tests/playwright.legacy-local.config.ts`, script `e2e:legacy-local`) | no | no | +| Every plugin in the **catalog index** is sane | `rhdh` | catalog-index plugin sanity check (nightly) | no | no | +| RHDH **container** behavior (image entrypoint, install script inside the image) | `rhdh-plugin-export-overlays` (artifacts) / any (manual) | Docker smoke (`overlays: smoke-tests/` + workflow `run-workspace-smoke-tests.yaml`) · manual: [rhdh-local](https://github.com/redhat-developer/rhdh-local) | no | yes | +| Helm chart / Operator / ingress / real Keycloak / RBAC on OCP | `rhdh` e2e or `overlays: workspaces/*/e2e-tests` | cluster e2e | **yes** | n/a (cluster) | Rule of thumb: **source bugs → rhdh-plugins · artifact bugs → overlays · integration/render bugs → rhdh · platform bugs → cluster e2e.** If a bug is catchable in more than one place, test it in the cheapest one and don't duplicate downstream. @@ -55,39 +58,40 @@ Rule of thumb: **source bugs → rhdh-plugins · artifact bugs → overlays · i | **L4a** E2E cluster-free | Full app, no managed infra | Playwright + local harness | no | min | | **L4b** E2E full | Real OCP/K8s, managed DBs, real IdPs | Playwright + cluster | **yes** | min–h | -The full per-spec classification of the RHDH e2e suite lives in `rhdh/docs/e2e-tests/layer-migration-matrix.md` (RHIDP-15076) — consult it before adding or migrating an e2e spec. +The full per-spec classification of the RHDH e2e suite lives in `rhdh: docs/e2e-tests/layer-migration-matrix.md` (see References) — consult it before adding or migrating an e2e spec. ## Step 4 — How to create the test (per placement) ### `rhdh-plugins` (or other source repo) — plugin correctness -- **Where:** `workspaces//plugins//src/**` next to the code, following that workspace's existing test setup (Jest/RTL; MSW for API mocks). +- **Where:** `plugins: workspaces//plugins//src/**` next to the code, following that workspace's existing test setup (Jest/RTL; MSW for API mocks). - **How:** copy the pattern of a neighboring `*.test.ts(x)`; run with the workspace's `yarn test`. Manual verification via the workspace dev app (`yarn start`). - Also validate packaging when the plugin's build config changed: `npx @red-hat-developer-hub/cli plugin export` must produce `dist-dynamic/` (+ `dist-scalprum/` for frontend). - **Cannot do here:** validate the *published* OCI artifact, or integration with RHDH's app shell/shared deps. ### `rhdh-plugin-export-overlays` — the published artifact -- **Where:** the native smoke harness (`smoke-tests-native/`, overlays PR #2714). Per-plugin: install from OCI with the real `install-dynamic-plugins` CLI and boot backend plugins via `startTestBackend`. Workspace mode: `yarn smoke --workspace ` boots every `oci://` artifact of the workspace together. -- New workspaces usually need **no new test code** — the harness discovers artifacts from `workspaces//metadata/*.yaml`. -- **Known gaps (don't fight them):** catalog-extending backend modules don't boot in a minimal `startTestBackend` (upstream `catalog-backend` registration issue — stay on Docker smoke until fixed); plugins whose `dynamicArtifact` is a local `./dynamic-plugins/dist/...` path ship inside the RHDH image and have no OCI artifact to validate here. +- **Where:** the native smoke harness, `overlays: smoke-tests-native/` (in review — see References). It installs plugins from OCI with the real `install-dynamic-plugins` CLI and boots backend plugins via `startTestBackend`. +- **How:** write a `dynamic-plugins.yaml` listing the `oci://` artifact refs to validate (a workspace's refs are the `spec.dynamicArtifact` values in `overlays: workspaces//metadata/*.yaml`), then run `yarn smoke --dynamic-plugins [--out results.json]`. Exit code 0 = pass; non-zero writes `results.json` detailing `fail-load` / `fail-start`. CI: `overlays: .github/workflows/native-smoke.yaml`. +- **Known gaps (don't fight them):** catalog-extending backend modules don't boot in a minimal `startTestBackend` (upstream `catalog-backend` registration issue — stay on the Docker smoke until fixed); plugins whose `dynamicArtifact` is a local `./dynamic-plugins/dist/...` path ship inside the RHDH image and have no OCI artifact to validate here. - **Never add UI-render tests here** — this repo has no app to render into. Delegate rendering to the `rhdh` cluster-free harness. -- `workspaces/*/e2e-tests` (cluster, Playwright + `e2e-test-utils`) exists for plugins whose value *is* cluster integration (topology, tekton, argocd…). Don't add one for UI-only behavior. +- The Docker smoke it replaces (backend scope) lives at `overlays: smoke-tests/` + workflow `run-workspace-smoke-tests.yaml`. +- `overlays: workspaces/*/e2e-tests` (cluster, Playwright + `e2e-test-utils`) exists for plugins whose value *is* cluster integration (topology, tekton, argocd…). Don't add one for UI-only behavior. ### `rhdh` — the real app -**L4a cluster-free harness** (`e2e-tests/playwright.legacy-local.config.ts`, docs at `docs/e2e-tests/local-e2e-harness.md`) — the only cheap place a frontend dynamic plugin can be *rendered*. To enable a spec/test: +**L4a cluster-free harness** (in review — see References; config `rhdh: e2e-tests/playwright.legacy-local.config.ts`, docs `rhdh: docs/e2e-tests/local-e2e-harness.md`) — the only cheap place a frontend dynamic plugin can be *rendered*. To enable a spec/test: -1. Plugin not yet installed by the harness? Add its OCI entry to `e2e-tests/local-harness/dynamic-plugins.yaml` (tags on `ghcr.io/redhat-developer/rhdh-plugin-export-overlays/`). If its mount-point config is not in the repo's static `app-config.dynamic-plugins.yaml`, attach the plugin's **canonical `pluginConfig`** (source of truth: `rhdh-plugins` → `workspaces//plugins//app-config.dynamic.yaml`) — the harness loads the generated `dynamic-plugins-root/app-config.dynamic-plugins.yaml` last, exactly like the production container. -2. Config that only exists in CI config maps (`.ci/pipelines/resources/config_map/*`)? Mirror just the needed keys into `app-config.local-e2e.yaml` (objects deep-merge; arrays replace). +1. Plugin not yet installed by the harness? Add its OCI entry to `rhdh: e2e-tests/local-harness/dynamic-plugins.yaml` (tags on `ghcr.io/redhat-developer/rhdh-plugin-export-overlays/`). If its mount-point config is not in the repo's static `app-config.dynamic-plugins.yaml`, attach the plugin's **canonical `pluginConfig`** (source of truth: `plugins: workspaces//plugins//app-config.dynamic.yaml`) — the harness loads the generated `dynamic-plugins-root/app-config.dynamic-plugins.yaml` last, exactly like the production container. +2. Config that only exists in CI config maps (`rhdh: .ci/pipelines/resources/config_map/*`)? Mirror just the needed keys into `rhdh: app-config.local-e2e.yaml` (objects deep-merge; arrays replace). 3. Tag the test `{ tag: "@cluster-free" }` and add its spec file to the config's `testMatch` allowlist. -4. Repopulate + validate: `./e2e-tests/local-harness/populate.sh`, then `yarn --cwd e2e-tests e2e:legacy-local` (CI does this in ~4 min; skopeo is Linux/CI-only). +4. Repopulate + validate: `./e2e-tests/local-harness/populate.sh`, then `yarn --cwd e2e-tests e2e:legacy-local` (CI runs this in ~4 min). Requires skopeo — preinstalled in CI; on macOS `brew install skopeo`. -**L4b cluster e2e** (`e2e-tests/playwright/e2e/**`) — only when the subject *is* cluster/platform behavior or a real external service. Requirements: `component` annotation in `beforeAll` (see `ci-e2e-testing` rule), correct config map choice (RBAC vs non-RBAC), project registration in `e2e-tests/playwright/projects.json` if a new project is needed. +**L4b cluster e2e** (`rhdh: e2e-tests/playwright/e2e/**`) — only when the subject *is* cluster/platform behavior or a real external service. Requirements: `component` annotation in `beforeAll` (see the repo's `ci-e2e-testing` rule), correct config map choice (RBAC vs non-RBAC), project registration in `e2e-tests/playwright/projects.json` if a new project is needed. -**L3 component tests** — pattern established on branch `RHIDP-13235-layer3-component-tests` (page-level RTL compositions). Prefer this over L4a when no dynamic-plugin loading is involved. +**L3 component tests** — page-level RTL compositions, pattern proposed under RHIDP-13235 in [rhdh#4864](https://github.com/redhat-developer/rhdh/pull/4864) (closed, not merged). Check for merged examples before recommending a concrete template; prefer L3 over L4a when no dynamic-plugin loading is involved. -**Catalog-index-wide sanity** (rhdh PR #4967, nightly) — nothing to write per plugin; it sweeps the whole index. +**Catalog-index-wide sanity** (nightly, in review — see References) — nothing to write per plugin; it sweeps the whole index. ## Not possible today (researched — don't burn time) @@ -106,9 +110,11 @@ Answer with a concrete recommendation: - **Why not elsewhere:** one line on the layers you rejected (especially if the dev proposed a more expensive one). - **Cost:** rough feedback time (seconds / ~4 min cluster-free / cluster job). -## References +## References (status as of 2026-07-02 — verify before citing as merged) -- Epic RHIDP-13501 (E2E Test Optimization) — per-repo responsibility split in the epic comments and the attached `rhdh-dynamic-plugin-testing-guideline.md`. -- `rhdh/docs/e2e-tests/layer-migration-matrix.md` — per-spec layer classification (RHIDP-15076). -- `rhdh/docs/e2e-tests/local-e2e-harness.md` — cluster-free harness usage. -- Reference PRs: overlays#2714 (native smoke), rhdh#4967 (catalog-index sanity), rhdh#5005 (cluster-free E2E harness). +- Epic **RHIDP-13501** (E2E Test Optimization) — the per-repo responsibility split lives in the epic's comments and its Jira attachment `rhdh-dynamic-plugin-testing-guideline.md` (Jira-only; if unreachable, the decision table above is the summary). +- Layer matrix: — in review in [rhdh#5044](https://github.com/redhat-developer/rhdh/pull/5044) (RHIDP-15076). +- Cluster-free harness + docs: in review in [rhdh#5005](https://github.com/redhat-developer/rhdh/pull/5005) (RHIDP-15075). +- Overlays native smoke: in review in [overlays#2714](https://github.com/redhat-developer/rhdh-plugin-export-overlays/pull/2714). +- Catalog-index sanity check: in review in [rhdh#4967](https://github.com/redhat-developer/rhdh/pull/4967). +- L3 pattern proposal: [rhdh#4864](https://github.com/redhat-developer/rhdh/pull/4864) (closed) under RHIDP-13235.