Skip to content

👷 Add pinned Firefox 119 + WebKit 17.4 e2e test runner without BrowserStack#4560

Draft
thomas-lebeau wants to merge 8 commits intov7from
thomas.lebeau/no-bs
Draft

👷 Add pinned Firefox 119 + WebKit 17.4 e2e test runner without BrowserStack#4560
thomas-lebeau wants to merge 8 commits intov7from
thomas.lebeau/no-bs

Conversation

@thomas-lebeau
Copy link
Copy Markdown
Collaborator

Motivation

Replicate the BrowserStack Firefox 119 + WebKit 17.4 coverage locally so we can keep running pinned-browser e2e tests without BrowserStack, and lay the groundwork for a CI eval loop that iterates on GitLab jobs autonomously.

Changes

  • Add e2e-pinned GitLab job and test:e2e:pinned script driven by a new playwright.pinned.config.ts.
  • Run the pinned matrix via Playwright 1.40.1 run-server plus a small translation proxy (test/e2e/scripts/pinned-proxy.mjs) so the current 1.58 client can talk to the older server.
  • Pre-install Playwright browsers (current Chromium + pinned FF 119 / WK 17.4) in the CI image; bump CURRENT_CI_IMAGE to 106.
  • Use the test page's own origin in flushEvents so the flush works for pages served from custom domains (WebKit fix).
  • Skip iframe-without-src session cookie tests on Firefox.
  • Document the autonomous CI eval loop approach in APPROACH.md.

Test instructions

  • yarn test:e2e:pinned:init then yarn test:e2e:pinned locally to run Firefox 119 + WebKit 17.4.
  • Verify the e2e-pinned job runs green in GitLab on this branch.

Checklist

  • Tested locally
  • Tested on staging
  • Added unit tests for this change.
  • Added e2e/integration tests for this change.
  • Updated documentation and/or relevant AGENTS.md file

thomas-lebeau and others added 7 commits May 4, 2026 16:51
…rStack

- Add `playwright.pinned.config.ts` that boots a Playwright 1.40.1
  `run-server` (which bundles FF 119 / WK 17.4) plus a translation proxy,
  and connects the current 1.58 client over wsEndpoint.
- Add `scripts/pinned-proxy.mjs` to bridge the JSON-RPC schema gap
  between Playwright 1.58 and 1.40 (UA spoofing for the version check,
  patches to __create__ initializers and a few command params).
- Add `test:e2e:pinned:init` / `test:e2e:pinned` scripts to install the
  pinned browsers locally and run the pinned matrix.
- Ignore the local `test/e2e/.pinned-browsers/` install directory.
Make sure we navigate to a page on the same domain as the current
domain, else webkit won't trigger exit events.
Firefox does not allow setting cookies from iframes without src, so the SDK
won't start and these tests are not meaningful there.
@datadog-official
Copy link
Copy Markdown

datadog-official Bot commented May 6, 2026

Tests

Fix all issues with BitsAI or with Cursor

⚠️ Warnings

🧪 5 Tests failed

developer extension › should switch between tabs › async from developer-extension/developerExtension.scenario.ts   View in Datadog   (Fix with Cursor)
developerExtension.scenario.ts:8:13 async

[chromium] › developer-extension/developerExtension.scenario.ts:8:13 › developer extension › should switch between tabs › async 

    Error: page.goto: net::ERR_FILE_NOT_FOUND at chrome-extension://ddlbenlpndknhmiknkjdcnogckbkhojd/flush
    Call log:
      - navigating to "chrome-extension://ddlbenlpndknhmiknkjdcnogckbkhojd/flush", waiting until "load"


       at ../lib/framework/flushEvents.ts:7
...
developer extension › should switch between tabs › bundle from developer-extension/developerExtension.scenario.ts   View in Datadog   (Fix with Cursor)
developerExtension.scenario.ts:8:13 bundle

[chromium] › developer-extension/developerExtension.scenario.ts:8:13 › developer extension › should switch between tabs › bundle 

    Error: page.goto: net::ERR_FILE_NOT_FOUND at chrome-extension://ddlbenlpndknhmiknkjdcnogckbkhojd/flush
    Call log:
      - navigating to "chrome-extension://ddlbenlpndknhmiknkjdcnogckbkhojd/flush", waiting until "load"


       at ../lib/framework/flushEvents.ts:7
...
developer extension › should switch between tabs › npm from developer-extension/developerExtension.scenario.ts   View in Datadog   (Fix with Cursor)
developerExtension.scenario.ts:8:13 npm

[chromium] › developer-extension/developerExtension.scenario.ts:8:13 › developer extension › should switch between tabs › npm 

    Error: page.goto: net::ERR_FILE_NOT_FOUND at chrome-extension://ddlbenlpndknhmiknkjdcnogckbkhojd/flush
    Call log:
      - navigating to "chrome-extension://ddlbenlpndknhmiknkjdcnogckbkhojd/flush", waiting until "load"


       at ../lib/framework/flushEvents.ts:7
...
View all

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 77.47%

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: dac31a5 | Docs | Datadog PR Page | Give us feedback!

@cit-pr-commenter-54b7da
Copy link
Copy Markdown

cit-pr-commenter-54b7da Bot commented May 6, 2026

Bundles Sizes Evolution

📦 Bundle Name Base Size Local Size 𝚫 𝚫% Status
Rum N/A 169.21 KiB N/A N/A N/A
Rum Profiler N/A 6.10 KiB N/A N/A N/A
Rum Recorder N/A 21.22 KiB N/A N/A N/A
Logs N/A 54.56 KiB N/A N/A N/A
Rum Slim N/A 127.56 KiB N/A N/A N/A
Worker N/A 22.99 KiB N/A N/A N/A
🚀 CPU Performance
Action Name Base CPU Time (ms) Local CPU Time (ms) 𝚫%
addglobalcontext N/A 0.0018 N/A
addaction N/A 0.0126 N/A
adderror N/A 0.0108 N/A
addtiming N/A 0.0005 N/A
startview N/A 0.0107 N/A
startstopsessionreplayrecording N/A 0.0007 N/A
logmessage N/A 0.0154 N/A
🧠 Memory Performance
Action Name Base Memory Consumption Local Memory Consumption 𝚫
addglobalcontext N/A 38.21 KiB N/A
addaction N/A 62.24 KiB N/A
addtiming N/A 38.21 KiB N/A
adderror N/A 68.97 KiB N/A
startstopsessionreplayrecording N/A 41.46 KiB N/A
startview N/A 478.62 KiB N/A
logmessage N/A 54.26 KiB N/A

🔗 RealWorld

@thomas-lebeau thomas-lebeau changed the title test(e2e): pinned Firefox 119 + WebKit 17.4 runner without BrowserStack 👷 Add pinned Firefox 119 + WebKit 17.4 e2e test runner without BrowserStack May 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants