Tech debt: Create compatibility tests for extensions (v2)

[aalves08]

Is your feature request related to a problem? Please describe.

Based on internal documentation, we should be creating compatibility tests for a specific version of Elemental extension that should run and pass all test cases using Playwright MCP server.

Describe the solution you'd like

Describe alternatives you've considered

Additional context

[github-actions]

🎯 Agentic Issue Triage

This is a tech-debt/enhancement issue requesting the creation of compatibility tests for extensions (v2) using a Playwright MCP server, targeting a specific version of the Elemental extension. It is part of a broader compatibility testing initiative — a direct follow-up to #17189 (which prepares the Elemental extension clone).

---

📋 Issue Summary

Type: Tech Debt / Enhancement
Affected Area: Extensions, QA Automation, AI/MCP tooling
Priority: 1 (High) — already labeled

The goal is to create automated compatibility tests that verify a specific version of the Elemental extension passes all test cases when run against the dashboard. The test runner of choice is Playwright via the MCP server, suggesting an AI-assisted/agentic testing approach.

The issue body is sparse — no concrete test cases, Elemental version, or acceptance criteria are provided yet. This will likely need scoping before implementation begins.

---

🔗 Related Issues / Context

This issue is part of a series of related tech-debt items:

Issue	Title	Relationship
#17189	Tech debt: Prepare Elemental extension "clone" for compatibility tests	Prerequisite — sets up the Elemental clone with all extension points and Shell API usages
#17193	Integrate logic for unit tests, typescript and linting in extensions	Related — broader testing improvements for extensions
#16908	Tech debt: Improve storing of version information in Rancher Dashboard	Related — version tracking for extensions

Recommended order of execution: #17189 should be completed first (preparing the Elemental clone), then this issue (#17237) creates the actual Playwright-based compatibility tests against that clone.

---

🧩 Suggested Sub-tasks / Checklist

• Define the target Elemental extension version to be tested (align with Tech debt: Prepare Elemental extension "clone" for extension compatibility tests #17189 — version 99.99.xxx range)
• Identify the full set of compatibility test cases to cover (extension points from the [support matrix]((extensions.rancher.io/redacted) + Shell API)
• Set up Playwright MCP server in the test environment (document configuration)
• Write Playwright-based compatibility tests for each identified extension point
• Ensure tests run in CI (integrate with existing Cypress/Jest pipeline or as a separate workflow)
• Document how to run the compatibility tests locally and in CI
• Verify all tests pass against the target Elemental extension version

---

🛠️ Debugging / Implementation Strategies

Playwright MCP Integration:

• The Playwright MCP server enables AI-assisted browser automation. Ensure the MCP server is properly configured and accessible in the test runner environment.
• Review existing Cypress E2E patterns in ./cypress/ for conventions (spec structure, selectors, page objects) and apply similar patterns in Playwright.

Extension Compatibility Coverage:

• Use the [Extension API Support Matrix]((extensions.rancher.io/redacted) as the source of truth for what needs to be tested.
• Use the [Shell API docs]((extensions.rancher.io/redacted) to enumerate all Shell API usages that need coverage.

Version Pinning:

• The Elemental clone (from Tech debt: Prepare Elemental extension "clone" for extension compatibility tests #17189) will use version 99.99.xxx — tests should target this version specifically to avoid interfering with production releases.

---

📚 Useful Resources

• [Extension API Support Matrix]((extensions.rancher.io/redacted)
• [Shell API Documentation]((extensions.rancher.io/redacted)
• [Rancher UI Internal Testing Docs]((extensions.rancher.io/redacted)
• [Playwright MCP Documentation]((playwright.dev/redacted)
• Elemental UI Repository
• Existing Cypress E2E tests in ./cypress/ — for conventions reference

---

🏷️ Label note: Added kind/tech-debt to align with the "Tech debt" title prefix (issue already carries kind/enhancement, priority/1, area/extensions, QA/dev-automation, area/ai).

Generated by Agentic Triage for issue #17237 · ◷

[m13v]

i went down this exact path with extension compat testing last year, ~14 extensions across 3 product versions. the trap we hit wasn't writing the tests, it was the matrix explosion. wrote a clean Playwright suite for one extension, took two weeks. then realized we needed it parameterized across versions and the selectors drifted between them. ended up with a tagged spec structure (@v1, @v2, @elemental) and a small fixture loader that swapped baseURL + auth per version. the real win was dropping shared data-testid requirements onto the extension contract itself, otherwise every minor UI change in a child extension nuked the parent suite. compat tests without selector contracts age badly.