feat: headless app and api for testing js-waku in browser #2371

adklempner · 2025-04-22T14:59:28Z

Problem / Description

The current browser test framework has limitations in its structure and integration capabilities. We need a more streamlined approach that better aligns with the nwaku REST API, allowing for seamless testing across different environments including Docker containers used by the Distributed Systems Testing team.

Solution

Replaced the old browser test structure with a single web app that runs a js-waku node
Created a server component that exposes an API and runs Playwright to load the web app in a headless browser
Designed the API to match the nwaku REST API
Implemented a containerized solution that allows DST (or other teams) to run simulated browser instances in a Docker image

Notes

This PR establishes the foundation for browser testing infrastructure; additional API functions will be built out based on feedback and testing requirements
Docker image functionality needs validation from the DST team to ensure it meets their testing requirements
Related to [Task] Browser simulations #2217
Resolves feat: dockerized server for simulating js-waku in headless browser #2380

Checklist

Code changes are covered by unit tests.
Code changes are covered by e2e tests, if applicable.
Dogfooding has been performed, if feasible.
A test version has been published, if required.
All CI checks pass successfully.

github-actions · 2025-05-13T23:17:51Z

size-limit report 📦

Path	Size	Loading time (3g)	Running time (snapdragon)	Total time
Waku node	61.99 KB (0%)	1.3 s (0%)	6.6 s (-17.45% 🔽)	7.8 s
Waku Simple Light Node	114.83 KB (0%)	2.3 s (0%)	8.1 s (+15.28% 🔺)	10.4 s
ECIES encryption	23 KB (0%)	461 ms (0%)	4.4 s (+49.49% 🔺)	4.9 s
Symmetric encryption	22.44 KB (0%)	449 ms (0%)	2.2 s (-49.6% 🔽)	2.6 s
DNS discovery	50.83 KB (0%)	1.1 s (0%)	7.3 s (+88.82% 🔺)	8.3 s
Peer Exchange discovery	51.38 KB (0%)	1.1 s (0%)	4.7 s (+31.07% 🔺)	5.7 s
Local Peer Cache Discovery	44.7 KB (0%)	894 ms (0%)	3 s (-37.83% 🔽)	3.9 s
Privacy preserving protocols	52.94 KB (0%)	1.1 s (0%)	7.5 s (+19.11% 🔺)	8.6 s
Waku Filter	54.92 KB (0%)	1.1 s (0%)	7.6 s (+40.71% 🔺)	8.7 s
Waku LightPush	52.29 KB (0%)	1.1 s (0%)	6.3 s (-9.37% 🔽)	7.4 s
History retrieval protocols	52.61 KB (0%)	1.1 s (0%)	5.3 s (+27.86% 🔺)	6.3 s
Deterministic Message Hashing	7.33 KB (0%)	147 ms (0%)	1.2 s (-58.12% 🔽)	1.3 s

danisharora099 · 2025-05-14T12:04:15Z

Ready for review? Issue description would be nice:D

weboko · 2025-05-14T21:13:06Z

ping @adklempner

adklempner · 2025-05-14T21:35:46Z

@danisharora099 @weboko updated!

Copilot

Pull Request Overview

This PR introduces a new browser testing infrastructure that replaces the old testing structure with a headless web app and server API endpoints that align with the nwaku REST API. Key changes include:

Implementation of a new Express server to run a headless browser with Playwright.
Introduction of multiple API endpoints (info, push, admin, etc.) to interact with the Waku node.
Updates to configuration files, Dockerfile, and CI workflows to support the new structure.

Reviewed Changes

Copilot reviewed 42 out of 42 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
packages/browser-tests/src/server.ts.new	New server implementation to serve the headless app with API endpoints and browser initialization.
packages/browser-tests/src/routes/*	New API routes for legacy and REST API compatible pushes and node management.
packages/browser-tests/src/queue/index.ts	Message queue implementation for storing and retrieving messages.
packages/browser-tests/src/browser/index.ts	Browser initialization and page management for the headless tests.
packages/browser-tests/src/api/*	New shared API functions wrapping Waku SDK functionality for operations like push, node creation, and subscriptions.
packages/browser-tests/playwright.config.ts, package.json, Dockerfile, CI workflows	Config and build updates to integrate the headless app and testing infrastructure.
packages/browser-tests/README.md	Updated documentation describing the new architecture and usage examples.

Comments suppressed due to low confidence (1)

Dockerfile:33

Ensure consistency in naming; the Dockerfile refers to a 'headless' directory while other scripts and CI workflows refer to 'headless-tests'. Align these names to prevent confusion across environments.

COPY headless/package.json ./headless/

Copilot · 2025-05-19T21:52:11Z

packages/browser-tests/src/api/push.ts

+export async function pushMessage(
+  waku: LightNode,
+  contentTopic: string,
+  payload?: Uint8Array<ArrayBuffer>


Consider revising the type annotation for payload; using 'Uint8Array' without a generic parameter is standard and may avoid potential type mismatches.

weboko · 2025-05-19T21:53:06Z

.eslintrc.json

+    "build",
+    "coverage",
+    "proto",
+    "**/webpack.config.js",


can't we suppress it in tsconfig of a particular projects?

weboko · 2025-05-19T21:57:28Z

General comment to CI, it seems there is some permission issue:

weboko · 2025-05-19T21:59:18Z

.github/workflows/ci.yml

          repository: waku-org/js-waku
      - uses: actions/setup-node@v3
        with:
          node-version: ${{ env.NODE_JS }}
      - uses: ./.github/actions/npm
+      - name: Install headless-tests dependencies


Since headless-tests is part of packages in root package.json - this step is not needed as - uses: ./.github/actions/npm should trigger installation

weboko · 2025-05-19T22:00:00Z

.github/workflows/ci.yml

          repository: waku-org/js-waku
      - uses: actions/setup-node@v3
        with:
          node-version: ${{ env.NODE_JS }}
      - uses: ./.github/actions/npm
+      - name: Install headless-tests dependencies


isn't Install headless-tests dependencies needed only for browser-test step?

weboko · 2025-05-19T22:00:27Z

.github/workflows/playwright.yml

@@ -29,6 +29,12 @@ jobs:

    - uses: ./.github/actions/npm

+    - name: Install headless-tests dependencies


doesn't seem to be needed due to ./.github/actions/npm called prior

weboko · 2025-05-19T22:04:42Z

package.json

@@ -17,6 +17,7 @@
    "packages/rln",
    "packages/tests",
    "packages/browser-tests",


if browser-tests depends on headless-tests - then they should be defined in different order in the array

also, about the naming, maybe it is better to rename:

browser-tests -> browser-container;

headless-tests -> browser-tests;

That way we define that one is container and another is test cases.

weboko · 2025-05-19T22:05:48Z

File .DS_Store in https://github.com/waku-org/js-waku/pull/2371/files#diff-87e08db8c823827e5089d6683ef36ab06ae1e884f85adca5ec774cc50f9e2a24 should be deleted

weboko · 2025-05-19T22:07:07Z

packages/browser-tests/README.md

+2. Start a headless browser to load the app
+3. Expose API endpoints to interact with Waku
+
+## API Endpoints


weboko · 2025-05-19T22:09:41Z

packages/browser-tests/package.json

-    "start:build": "node ./src/build-example.js",
-    "start:serve": "npx serve -p 8080 --no-port-switching ./example",
-    "test": "npx playwright test"
+    "start": "npm run start:server",


nit: can we group similar commands, please?

Suggested change

"start": "npm run start:server",

"start": "npm run start:server",

"start:serve": "cd ../headless-tests && npx serve -p 8080 --no-port-switching . -s",

"start:server": "node ./dist/server.js",

weboko · 2025-05-19T22:12:16Z

packages/browser-tests/package.json

+    "start": "npm run start:server",
+    "start:serve": "cd ../headless-tests && npx serve -p 8080 --no-port-switching . -s",
+    "test": "npx playwright test",
+    "build:headless": "cd ../headless-tests && npm run build",


I think we should be able instead of cd ../headless-tests && npm run build to do something like npm run build --workspace=headless-tests in this and similar places, removing cd ../headless-tests altogether

let me know if it works, I am not 100% sure

weboko · 2025-05-19T22:13:08Z

packages/browser-tests/playwright.config.ts

-      name: "firefox",
-      use: { ...devices["Desktop Firefox"] }
-    },
+    // {


not needed anymore? if so, let's remove

weboko · 2025-05-19T22:16:23Z

packages/headless-tests/package.json

+    "scripts": {
+        "start": "serve .",
+        "build": "webpack",
+        "deploy": "gh-pages -d .",


is it really deployed to GitHub pages?

weboko · 2025-05-19T22:17:24Z

packages/headless-tests/.eslintrc.cjs

+  },
+  rules: {
+    // Disable rules that might cause issues with this package
+    "no-undef": "off"


let's disable them here instead of inline comments?

weboko · 2025-05-19T22:17:54Z

packages/browser-tests/types/dotenv-flow.d.ts

@@ -0,0 +1,2 @@
+declare module "dotenv-flow/config";


I am wondering why we need these declarations?

weboko · 2025-05-19T22:19:31Z

packages/browser-tests/src/server.ts.new

@@ -0,0 +1,106 @@
+/* eslint-disable no-console */


why this file has extension .ts.new? :)

removed, unnecessary artifact

weboko · 2025-05-19T22:21:18Z

packages/browser-tests/src/browser/index.ts

@@ -0,0 +1,52 @@
+/* eslint-disable no-console */


so many of these, can we suppress it on the level of package's config?

weboko · 2025-05-19T22:22:03Z

packages/browser-tests/src/api/shared.ts

+  } catch (err) {
+    // Just log, don't fail
+    // eslint-disable-next-line no-console
+    console.error(`Error checking node protocols: ${String(err)}`);


do we actually need these log statements? why can't we use debug package?

This same code needs to work both in the server and in the web app loaded via headless browser, so it makes minimal use of imported packages besides @waku/sdk

adklempner force-pushed the feat/browser-headless-test branch from 3b260f8 to f295a35 Compare April 22, 2025 15:28

adklempner force-pushed the feat/browser-headless-test branch 7 times, most recently from 0ab2d81 to 8c53464 Compare May 13, 2025 02:32

feat: headless browser test

cffbef3

adklempner force-pushed the feat/browser-headless-test branch from 8c53464 to cffbef3 Compare May 13, 2025 03:24

adklempner force-pushed the feat/browser-headless-test branch 2 times, most recently from 1d822b1 to 365833c Compare May 14, 2025 00:46

adklempner marked this pull request as ready for review May 14, 2025 01:19

adklempner requested a review from a team as a code owner May 14, 2025 01:19

weboko requested a review from Copilot May 19, 2025 21:51

Copilot AI reviewed May 19, 2025

View reviewed changes

weboko reviewed May 19, 2025

View reviewed changes

fix: restructure headless browser app to its own folder

49bc580

adklempner force-pushed the feat/browser-headless-test branch from 365833c to 49bc580 Compare May 20, 2025 03:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: headless app and api for testing js-waku in browser #2371

feat: headless app and api for testing js-waku in browser #2371

adklempner commented Apr 22, 2025 •

edited

Loading

github-actions bot commented May 13, 2025 •

edited

Loading

danisharora099 commented May 14, 2025 •

edited

Loading

weboko commented May 14, 2025

adklempner commented May 14, 2025

Copilot AI left a comment

Copilot AI May 19, 2025

weboko May 19, 2025

weboko commented May 19, 2025

weboko May 19, 2025

weboko May 19, 2025

weboko May 19, 2025

weboko May 19, 2025

weboko commented May 19, 2025

weboko May 19, 2025

weboko May 19, 2025

weboko May 19, 2025

weboko May 19, 2025

weboko May 19, 2025

weboko May 19, 2025

weboko May 19, 2025

weboko May 19, 2025 •

edited

Loading

adklempner May 20, 2025

weboko May 19, 2025

weboko May 19, 2025

adklempner May 19, 2025 •

edited

Loading

		@@ -29,6 +29,12 @@ jobs:

		- uses: ./.github/actions/npm

		- name: Install headless-tests dependencies

feat: headless app and api for testing js-waku in browser #2371

Are you sure you want to change the base?

feat: headless app and api for testing js-waku in browser #2371

Conversation

adklempner commented Apr 22, 2025 • edited Loading

Problem / Description

Solution

Notes

Checklist

github-actions bot commented May 13, 2025 • edited Loading

size-limit report 📦

danisharora099 commented May 14, 2025 • edited Loading

weboko commented May 14, 2025

adklempner commented May 14, 2025

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Copilot AI May 19, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

weboko commented May 19, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

weboko commented May 19, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

weboko May 19, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adklempner May 19, 2025 • edited Loading

Choose a reason for hiding this comment

adklempner commented Apr 22, 2025 •

edited

Loading

github-actions bot commented May 13, 2025 •

edited

Loading

danisharora099 commented May 14, 2025 •

edited

Loading

weboko May 19, 2025 •

edited

Loading

adklempner May 19, 2025 •

edited

Loading