test: setup E2E testing via Playwright and implement some tests

[pkerschbaum]

Prerequisites checklist

• I have read the contributing guidelines.

What is the purpose of this pull request?

Introduces End-to-End (E2E) testing using Playwright to prevent regressions and increase confidence during development and refactoring.

What changes did you make? (Give an overview)

• Set up Playwright for E2E testing.
  • Configured GitHub Actions CI to run E2E tests.
  • Added setup based on Docker for running tests to ensure consistent screenshot generation via Playwright's toHaveScreenshot.
• Implemented initial E2E test scenarios.
• Also made some components here and there more accessible - which is useful for locating things in Playwright.

Is there anything you'd like reviewers to focus on?

Notes on using Docker

The new requirement, having Docker installed, is quite heavy...
Thing is that otherwise, screenshots created by screenshot assertions (toHaveScreenshot) are not consistent, leading to flaky behavior.
As noted by Playwright:

Browser rendering can vary based on the host OS, version, settings, hardware, power source (battery vs. power adapter), headless mode, and other factors. For consistent screenshots, run tests in the same environment where the baseline screenshots were generated.

Alternatives to Docker considered:

• not using screenshot assertions at all.
  • Problematic because they are the only way to assert aesthetics, which is important for ESLint Code Explorer
  • also, they are quite useful to get "much bang for the buck" as they cover much of a web application fast, compared to "fine-grained assertions"
• implement some workflow using GitHub Actions and Pull Request comments to create baseline screenshots only in CI (like https://medium.com/@haleywardo/streamlining-playwright-visual-regression-testing-with-github-actions-e077fd33c27c)
  • Problem is: how to develop E2E tests locally then?
  • Also, while we could drop the Docker requirement then, we would need to add instructions to install the system dependencies required for Playwright (https://playwright.dev/docs/browsers#install-system-dependencies) - seems like adding Playwright to a project adds a requirement to the development machines of contributors, one way or the other

Notes on using Playwright compared to other options

• vitest browser mode, downsides:
  • it is experimental currently
  • lacks documentation
  • looks like it has no tools like test recorder, time travel debugging, etc.
• Storybook Testing, downsides:
  • visual regression testing is only possible with their SaaS product "Chromatic" (https://storybook.js.org/docs/writing-tests/visual-testing)
  • I tried it out and and it just does not feel that mature and polished like Playwright (e.g. the locator API, expect APIs of Playwright are just great; Playwright produces a great report which is expecially useful in CI, ...)

Successful CI run: https://github.com/eslint/code-explorer/actions/runs/14296315788

[netlify]

✅ Deploy Preview for eslint-code-explorer ready!

Name	Link
🔨 Latest commit	cd10bf7
🔍 Latest deploy log	https://app.netlify.com/sites/eslint-code-explorer/deploys/67fbc5d80ab4060008ce90d5
😎 Deploy Preview	https://deploy-preview-88--eslint-code-explorer.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[nzakas]

Thanks for getting this started. I don't have time to dig in right now. I'm also not super familiar with this kind of setup, so could use some advice from the rest of the team. @eslint/eslint-team

[Copilot]

Copilot reviewed 51 out of 52 changed files in this pull request and generated no comments.

Files not reviewed (1)

• package.json: Language not supported

Comments suppressed due to low confidence (2)

src/components/ui/popover.tsx:27

• [nitpick] Verify that the use of role 'dialog' on PopoverContent is semantically correct for a popover component; if the component is used to display menus or tips rather than modal dialogs, a different role may be more appropriate.

role="dialog"

src/components/tree-entry.tsx:109

• Ensure that the li element is nested within a ul or ol element to maintain proper HTML semantics.

<li className="flex items-center gap-3">

[snitin315]

I'm familiar with such setup, will review this pr

[nzakas]

@snitin315 friendly ping to not forget about this

[snitin315]

@pkerschbaum Are we not running Docker in CI? If that's the case, it kind of defeats the purpose of using Docker, since the CI environment would always be inconsistent.

[pkerschbaum]

@pkerschbaum Are we not running Docker in CI? If that's the case, it kind of defeats the purpose of using Docker, since the CI environment would always be inconsistent.

It is always - locally and in CI - running via Docker.
The only difference is some settings related to networking.

[nzakas]

@pkerschbaum can you take a look at the merge conflicts?

[nzakas]

I tried running this locally and got this error:

> [email protected] test:e2e    
> cross-env PW_TEST_CONNECT_WS_ENDPOINT=ws://127.0.0.1:3000/ playwright test      
[WebServer] Unable to find image 'mcr.microsoft.com/playwright:v1.51.1-noble' locally
[WebServer] v1.51.1-noble: 
[WebServer] Pulling from playwright      
[WebServer] 107a4fb0af38: Pulling fs layer
[WebServer] 8c401718c984: Pulling fs layer
[WebServer] 39224aba52ce: Pulling fs layer
[WebServer] 08d1cdd3adad: Pulling fs layer
[WebServer] 08d1cdd3adad: Waiting        
[WebServer] 39224aba52ce: 
[WebServer] Verifying Checksum           
[WebServer] 39224aba52ce: Download complete
[WebServer] 107a4fb0af38: 
[WebServer] Verifying Checksum           
[WebServer] 107a4fb0af38: Download complete
[WebServer] 8c401718c984: Download complete
[WebServer] 107a4fb0af38: Pull complete
[WebServer] 8c401718c984: 
[WebServer] Pull complete                
[WebServer] 39224aba52ce: 
[WebServer] Pull complete                

Error: Timed out waiting 60000ms from config.webServer.

To open last HTML report run:

  npx playwright show-report

I'm running Windows 11.

[snitin315]

We need a more specific check because env vars are always string values.

Suggested change

	baseURL: `http://${process.env.CI ? "localhost" : "host.docker.internal"}:5173`,
	baseURL: `http://${process.env.CI === "true" ? "localhost" : "host.docker.internal"}:5173`,

Also as per this check we are running locally on CI and docker in local dev