docs(e2e): add E2E layer migration matrix (RHIDP-15076)#5044
docs(e2e): add E2E layer migration matrix (RHIDP-15076)#5044gustavolira wants to merge 2 commits into
Conversation
Phase-1 deliverable of RHIDP-15076 (E2E Test Optimization epic RHIDP-13501): classifies all 30 e2e specs by target layer (L1-L4b), maps which are supplementable by Layer 3 component tests or the cluster-free L4a harness, and includes the companion analysis of the rhdh-plugin-export-overlays test system. Referenced from the Jira story's closing comment; updated 2026-07-02 with the L4a harness validation results from PR redhat-developer#5005 (4 tests green cluster-free). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PR Summary by QodoAdd E2E→lower-layer migration matrix for test optimization (RHIDP-15076)
AI Description
Diagram
High-Level Assessment
Files changed (1)
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #5044 +/- ##
==========================================
- Coverage 55.39% 54.77% -0.62%
==========================================
Files 122 110 -12
Lines 2365 2147 -218
Branches 544 513 -31
==========================================
- Hits 1310 1176 -134
+ Misses 1049 970 -79
+ Partials 6 1 -5
Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
Code Review by Qodo
Context used✅ Compliance rules (platform):
46 rules✅ Skills:
e2e-verify-fix, e2e-diagnose-and-fix 1.
|
…tion, path prefixes - Mark harnesses that are still in review and instruct the assistant to verify paths exist on main before recommending them (the skill referenced redhat-developer#5005/redhat-developer#5044/redhat-developer#2714/redhat-developer#4967 deliverables in present tense). - Describe the overlays native smoke's real interface: yarn smoke --dynamic-plugins <file> [--out] (no --workspace flag exists; the harness does not read workspaces/*/metadata — a workspace run is a dp.yaml listing its oci:// refs). - Replace the fork-only RHIDP-13235-layer3-component-tests branch pointer with the closed PR rhdh#4864. - Prefix every path with its repo (rhdh:/overlays:/plugins:) and use full URLs for cross-repo docs, so the skill resolves from any of its three homes. - Fix skopeo claim: it installs on macOS via brew; CI has it preinstalled. - Name the Docker smoke location (overlays smoke-tests/ + run-workspace-smoke-tests.yaml), replace the vague "any" repo cell, use "n/a (cluster)" for the cluster row's Docker column, move PR numbers to References with status, add trigger phrases to the frontmatter description, and deduplicate the guiding-rule sentence. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…DME link - Fix the Tally L3 count (10, not 9) so the column totals sum to the 30-spec heading; reconcile the heading itself (29 on main + redhat-developer#19 pending in PR redhat-developer#4967). - "runs 4 tests" -> "runs 2 specs (4 test cases)" — the doc's accounting unit is the spec; note that spec numbers refer to the matrix below. - Soften "fully covers the 12 pure-backend workspaces" to load + API surface: scaffolder-backend-module-kubernetes also has a UI e2e that needs the render harness, so "fully" overstated the native-smoke scope. - Replace short commit hashes and the fork-only RHIDP-13235-layer3-component-tests branch name with the durable PR reference (rhdh#4864, closed) — hashes on a mutable branch dangle after a rebase or branch deletion. - Give DRAFT a promotion condition (groomed into RHIDP-13528/13529). - Link the matrix from docs/e2e-tests/README.md ("Adding a Test") so the doc is discoverable outside the Jira comment. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
The container image build workflow finished with status: |
|
| ### Tally | ||
|
|
||
| | Target | Count | Specs | | ||
| | -------------- | ----- | ------------------------------------------- | |
There was a problem hiding this comment.
The count says 9 but the Specs column lists 10 entries (#3 through #12), and the column sums give 4+9+5+11=29 while the matrix heading says 30. If #3 only half-counts because of the smoke footnote, shouldn't the count still be 10 with the asterisk carrying the caveat? This table feeds the RHIDP-13528/13529 batches, so the off-by-one propagates into ticket scoping.
|
|
||
| ## Update 2026-07-02 — L4a harness validation (RHIDP-15075, PR #5005) | ||
|
|
||
| (Spec numbers refer to the summary matrix below.) |
| ### Recommendation for the overlay repo (feeds RHIDP-13530 / RHDHPLAN-525) | ||
|
|
||
| Build **two harnesses**, not one Docker fixture: | ||
|
|
There was a problem hiding this comment.
scaffolder-backend-module-kubernetes is in this "fully covers" list, but it also sits in the "Frontend UI e2e (24)" bucket below where startTestBackend is explicitly insufficient, and in tier B of the cluster-coupling table. Should this read "covers the load + API surface of" instead of "fully covers", or does that workspace not belong in the pure-backend list? Whoever picks up RHIDP-13530 will use these buckets as a work breakdown.
| land. | ||
|
|
||
| Note: #19 `plugin-dynamic-loading` ships with PR #4967, which is still **open** — the | ||
| row below describes its state once merged. |
There was a problem hiding this comment.
| RHIDP-13235 (Layer 3 component tests), carried by PR #4864. These prove the pattern | ||
| works and should be the template for the rest (the PR is the durable reference — its | ||
| branch may be rebased or deleted): | ||
|
|
There was a problem hiding this comment.
These short hashes point at an unmerged branch — one rebase or force-push and they're dangling, and once the branch is deleted after merge GC can drop the commits entirely. Since this doc is meant to outlive the branch, referencing the test file paths (or the eventual PR number) would stay resolvable.
| @@ -0,0 +1,274 @@ | |||
| # E2E → Lower-Layer Migration Matrix (Phase 1) | |||
There was a problem hiding this comment.
Right now the only path to this doc is the Jira comment that references it — docs/e2e-tests/README.md doesn't link it and no index picks it up. A one-line link from the e2e-tests README (or CI.md, which covers test strategy) would finish the discoverability goal from the PR description.
| **Story**: RHIDP-15076 — Identify E2E specs supplementable by Layer 3 / cluster-free harness (Phase 1) | ||
| **Author**: Gustavo Lira e Silva | ||
| **Date**: 2026-06-26 (updated 2026-07-02 with L4a harness validation results) | ||
| **Status**: DRAFT — promote once the batches below are groomed into RHIDP-13528/13529 |
There was a problem hiding this comment.
nit: Docs merged as DRAFT tend to stay DRAFT forever — what promotes this one? Either drop the status line (merging is the publication event) or state the condition, e.g. "DRAFT until RHIDP-13528/13529 are groomed".
| - _Needs full app but only GitHub/IdP (no cluster infra)_ → **L4a** now; **L2/L3** only if the external call is mocked | ||
| - _Needs real cluster / managed DB / real IdP / ConfigMap reload_ → **stays L4b** | ||
|
|
||
| ## Update 2026-07-02 — L4a harness validation (RHIDP-15075, PR #5005) |
There was a problem hiding this comment.
1 participant
Summary
Publishes the Phase-1 deliverable of RHIDP-15076 ([Test Strategy] E2E Test Optimization epic RHIDP-13501): the E2E → lower-layer migration matrix.
The Jira story's closing comment references this document by path (
rhdh/docs/e2e-tests/layer-migration-matrix.md), but it had never been committed — this PR closes that gap.Contents
rhdh-plugin-export-overlaystest system (smoke vs e2e tiers, native harness recommendation — see overlays#2714).This is an additive analysis, not a removal plan — no E2E spec has to be deleted (per the epic).
Related
🤖 Generated with Claude Code