Summary
In web-platform-tests/wpt, an explicit ./wpt run --headless chromium ... run launches Chromium with --headless=new, but the generated wptreport records run_info.headless as false.
This was reproduced from WPT revision 54f8f933629e7c010ae98a246729af01f8abcda5. The focused test completed successfully, so the report was produced by a functioning headless browser session rather than an aborted setup path.
Steps to reproduce
-
Check out WPT revision 54f8f933629e7c010ae98a246729af01f8abcda5 and install the Chromium product requirements.
-
Add a minimal testharness test, for example crabbox-skill-proof/basic.html:
<!doctype html>
<meta charset="utf-8">
<title>WPT headless run-info reproduction</title>
<script src="/resources/testharness.js"></script>
<script src="/resources/testharnessreport.js"></script>
<script>
test(() => assert_true(true), "testharness executes a focused assertion");
</script>
-
Configure the WPT host entries, then run the test with Chromium and ChromeDriver paths appropriate for the machine:
./wpt run \
--binary /usr/bin/chromium \
--webdriver-binary /usr/bin/chromedriver \
--headless \
--processes 1 \
--no-manifest-download \
--log-wptreport=/tmp/wptreport.json \
chromium /crabbox-skill-proof/basic.html
In a container where Chromium requires it, add --binary-arg=--no-sandbox; that flag does not affect the reported headless value.
-
Inspect the report:
jq '.run_info.headless' /tmp/wptreport.json
Expected behavior
run_info.headless is true when the run was requested with --headless. A control run with --no-headless should report false.
Actual behavior
The headless run reports:
{
"product": "chromium",
"browser_version": "151.0.7922.71 built on Debian GNU/Linux 12 (bookworm)",
"revision": "54f8f933629e7c010ae98a246729af01f8abcda5",
"processor": "aarch64",
"headless": false
}The same report records the test as OK and its subtest as PASS.
Affected area
tools/wptrunner/wptrunner/browsers/chrome.pytools/wptrunner/wptrunner/products.pytools/wptrunner/wptrunner/wpttest.py- Chromium/Chrome
wptreport run metadata
Runtime or environment
- WPT revision:
54f8f933629e7c010ae98a246729af01f8abcda5
- OS: Debian GNU/Linux 12 (Bookworm), AArch64
- Python: 3.11.2
- Chromium: 151.0.7922.71
- ChromeDriver: 151.0.7922.71
Evidence
The Chromium adapter receives the headless option and appends --headless=new in tools/wptrunner/wptrunner/browsers/chrome.py, so the browser invocation and the requested mode agree.
However, Chrome's __wptrunner__ registration does not provide run_info_extras. The generic product fallback returns an empty mapping, after which RunInfo defaults an absent headless field to false. Firefox already demonstrates the expected product integration by returning "headless": kwargs.get("headless", False) or "MOZ_HEADLESS" in os.environ from its run_info_extras implementation.
The red/green closure signal is direct: before a fix, the reproduction above yields false; after a fix, the same --headless Chromium run must yield true, while a --no-headless control must remain false.
Impact
Consumers of wptreport cannot reliably distinguish headless Chromium results from headed results. That can misclassify test runs, expectation dimensions, dashboards, or downstream comparisons even though the browser was actually launched headlessly.
Additional context
Issue #49771 discusses broader ownership and discovery of run_info, but it does not report this concrete Chromium mismatch or provide the headless/non-headless closure condition above. Searches of open and closed WPT issues and pull requests for the exact Chromium/headless report symptom found no matching report as of 2026-08-08.
Summary
In
web-platform-tests/wpt, an explicit./wpt run --headless chromium ...run launches Chromium with--headless=new, but the generatedwptreportrecordsrun_info.headlessasfalse.This was reproduced from WPT revision
54f8f933629e7c010ae98a246729af01f8abcda5. The focused test completed successfully, so the report was produced by a functioning headless browser session rather than an aborted setup path.Steps to reproduce
Check out WPT revision
54f8f933629e7c010ae98a246729af01f8abcda5and install the Chromium product requirements.Add a minimal testharness test, for example
crabbox-skill-proof/basic.html:Configure the WPT host entries, then run the test with Chromium and ChromeDriver paths appropriate for the machine:
In a container where Chromium requires it, add
--binary-arg=--no-sandbox; that flag does not affect the reported headless value.Inspect the report:
jq '.run_info.headless' /tmp/wptreport.jsonExpected behavior
run_info.headlessistruewhen the run was requested with--headless. A control run with--no-headlessshould reportfalse.Actual behavior
The headless run reports:
{ "product": "chromium", "browser_version": "151.0.7922.71 built on Debian GNU/Linux 12 (bookworm)", "revision": "54f8f933629e7c010ae98a246729af01f8abcda5", "processor": "aarch64", "headless": false }The same report records the test as
OKand its subtest asPASS.Affected area
tools/wptrunner/wptrunner/browsers/chrome.pytools/wptrunner/wptrunner/products.pytools/wptrunner/wptrunner/wpttest.pywptreportrun metadataRuntime or environment
54f8f933629e7c010ae98a246729af01f8abcda5Evidence
The Chromium adapter receives the headless option and appends
--headless=newintools/wptrunner/wptrunner/browsers/chrome.py, so the browser invocation and the requested mode agree.However, Chrome's
__wptrunner__registration does not providerun_info_extras. The generic product fallback returns an empty mapping, after whichRunInfodefaults an absentheadlessfield tofalse. Firefox already demonstrates the expected product integration by returning"headless": kwargs.get("headless", False) or "MOZ_HEADLESS" in os.environfrom itsrun_info_extrasimplementation.The red/green closure signal is direct: before a fix, the reproduction above yields
false; after a fix, the same--headlessChromium run must yieldtrue, while a--no-headlesscontrol must remainfalse.Impact
Consumers of
wptreportcannot reliably distinguish headless Chromium results from headed results. That can misclassify test runs, expectation dimensions, dashboards, or downstream comparisons even though the browser was actually launched headlessly.Additional context
Issue #49771 discusses broader ownership and discovery of
run_info, but it does not report this concrete Chromium mismatch or provide the headless/non-headless closure condition above. Searches of open and closed WPT issues and pull requests for the exact Chromium/headless report symptom found no matching report as of 2026-08-08.