[Feature] Headless Electron

[bpasero]

I could not find a way to run Electron tests headless like that is possible with browser tests.

//cc @pavelfeldman

[pavelfeldman]

Electron itself does not have the headless mode. It can probably be simulated via hiding the window, but there is no built-in functionality of swapping the graphics stack like there is in Chromium. What do your existing tests do today?

[bpasero]

What do your existing tests do today?

:-), got me there. Yeah, not headless. I think its fine, I was just thinking that browser and electron should support the same set of options, but its not a big deal for us I would say.

In our CI though we do run headless browser tests and maybe that means a bit faster execution time, but I see the issue that Electron itself does not really support this mode.

[pavelfeldman]

It wasn't a gotcha reply. I was hoping you have an ad-hoc solution, something like hiding the window in test mode, that was working for you.

[tanishqkancharla]

I was able to get headless electron tests working with this snippet: override the default behavior of showing the browser window once it's ready to show:

this.browserWindow.once("ready-to-show", () => {
	if (config.test?.headless) return
	this.browserWindow.show()
})

So you can run the app electron main.js --test-headless=true, parse the process.argv, and when actually ready to display a window, you read the config file, and defer showing the window. This also means you can also test "head-fully" if, for example, playwright is invoked in DEBUG mode:

const isDebug = Boolean(process.env.PWDEBUG)

const config: Partial<Config> = {
  test: {
	  headless: !isDebug,
  },
  partition,
}

const args = [rootPath("build/main.js"), ...makeAppCliArgs(config)]

The renderer process is still running, just completely headlessly, without a display, so the tests are all valid. For this to work on CI (on linux), you need xvfb.

Also these would be great comments to go on the docs.

[greenwood-vaarst]

@tanishqkancharla can I ask are you then still able to do

const app = await electron.launch(someConfig);
const page = await app.firstWindow();

or are you creating a different kind of page somehow?

thanks

[tanishqkancharla]

@greenwood-vaarst that's exactly what I'm doing. I use fixtures, and the api comes out pretty clean. Here's a snippet of what that looks like:

export const e2eTest = baseTest.extend<E2ETestFixture, E2EWorkerFixture>({
	testData: [
		async ({ harness }, use, testInfo) => {
                        // ...
			// Run head-full if debugging
			// npx playwright --debug
			const isDebug = Boolean(process.env.PWDEBUG)

			const config: Partial<Config> = {
				test: {
					MAIN_PORT,
					RENDERER_PORT,
					headless: !isDebug,
					id: testId,
				},
				partition,
			}

			// ...

			const app = await electron.launch({ args, colorScheme: "dark" })
			await app.firstWindow()

			await harness.waitUntilTestIsReady(testId)

			try {
				await use({ app, harness, partition, testId })
			} finally {
				harness.dispatch.unmountTest(testId)
				await app.close()
			}
		},
		{ auto: true, timeout: 5000 },
	],
})

And here's how it gets used:

e2eTest("Finds match", async ({ testData }) => {
		const h = e2eDialect(testData)
		const w = h.window(0)

		await h.createHtmlFile("test.html", `<p>some words</p>`)
		await w.openFile("test.html")
                // ...
})

When I was working on this project, I didn't really use the playwright page object directly very much. Instead I hooked up a third test harness server that connected to both the main and renderer process and communicated with them directly. In the second code snippet, h.createHtmlFile calls a remote handler on the main process and w is an alias to the first window.

[greenwood-vaarst]

Thanks so much for your reply @tanishqkancharla