feat: download browsers as TAR #34033

Skn0tt · 2024-12-16T15:53:00Z

Some of our browsers are already available as .tar.br. Compared to the current .zip archives, the brotli tarballs are ~10-30% smaller. This PR makes us download brotli files for chromium and webkit.

packages/playwright-core/src/server/registry/oopDownloadBrowserMain.ts

pavelfeldman · 2024-12-18T22:40:27Z

packages/playwright-core/bundles/zip/package.json

@@ -10,6 +10,7 @@
  },
  "dependencies": {
    "extract-zip": "2.0.1",
+    "tar-fs": "^3.0.6",


I understand the library is popular, but its deps list seem to be excessive for what it does a little. Did we consider alternatives?

Oh wow, "tar" is even more...

Yes, we considered tar-fs, tar and writing our own. Writing our own turned out more complex than imagined, because webkit has very long path names and the format becomes tricky when that's involved. Of the three, tar-fs seemed the most focused.

mxschmitt · 2024-12-19T16:13:06Z

packages/playwright-core/browsers.json

@@ -48,8 +48,8 @@
      "revision": "1011",
      "installByDefault": true,
      "revisionOverrides": {
-        "mac12": "1010",
-        "mac12-arm64": "1010"
+        "mac12": "1011",


whats the motivation for changing this?

1010 doesn't have .tar.br, and 1010 is identical to 1011 in functionality

pavelfeldman · 2025-01-04T00:17:58Z

packages/playwright-core/ThirdPartyNotices.txt

@@ -1229,6 +2813,6 @@ END OF [email protected] AND INFORMATION

 SUMMARY BEGIN HERE
 =========================================
-Total Packages: 48
+Total Packages: 60


We are paying a 25% bump in # of deps for a feature that does not link to a user report linked. Usually not a very good sign.

True. It also increases the zip bundle size from 112kb to 202kb. We had this attempt of writing our own tar parser, maybe we should give it another try.

pavelfeldman · 2025-01-04T00:29:13Z

Just as an idea, can we utilize extract-zip's non-compression mode for tar? That way we use zip for tar and don't need all this extra code? i.e. the files will be .zip.br, not .tar.br.

Skn0tt · 2025-01-06T12:01:01Z

Just as an idea, can we utilize extract-zip's non-compression mode for tar?

That'd save some dependencies, but would result in slightly larger bundles¹ and it'd prevent streaming extraction. I'd prefer to stick with TAR, gonna take a stab at reducing the bundle size for that.

tested on firefox-mac: .zip is 92mb, .zip.br is 67.2mb and .tar.br is 65.36mb ↩

Skn0tt · 2025-01-06T14:04:07Z

Alright, i've vendored tar-fs and replaced all of its dependencies with the built-in equivalents. (fun fact: pump, one of the deps, ended up landing in Node.js core as stream.pipeline)

That way we're using a tried & tested tar parser, but don't pay the download price. Let me know if you like that approach, and where's the best place for a vendored module to live / any license specifics we need to follow.

pavelfeldman · 2025-01-13T03:48:29Z

packages/playwright-core/src/server/registry/index.ts

    if (!downloadPathTemplate)
      return [];
+    // old webkit versions don't have brotli


curious why only old webkit revisions don't have brotli

webkit is the only browser we have revisionOverrides overrides for that point to old versions, so the CI script that created them didn't yet create brotli

pavelfeldman · 2025-01-13T03:54:41Z

packages/playwright-core/src/server/registry/oopDownloadBrowserMain.ts

+    }
+    log(`SUCCESS downloading and extracting ${options.title}`);
+  } else {
+    await downloadFile(options);


I'm seeing different error handling code in this branch, including explicit checks for ECONNRESET. Is walking away from them intended? Should we do both changes at a time? I'd be more comfortable with leaving the download code as is and swapping piping into file with piping into broti.

The new branch is intended to be as similar as possible, while also making the code a little more linear. The ECONNRESET check only changed the error message, so I didn't include that.

Let me see if I can refactor it to make the change less spooky.

I've refactored it so we can reuse the existing download function. Good pointer, thanks!

pavelfeldman · 2025-01-13T03:58:23Z

packages/playwright-core/src/utils/tar/README.md

@@ -0,0 +1 @@
+This directory contains a modified copy of the `tar-stream` library that's used exclusively to extract TAR files.


Make sure all the third party files are under the third_party folder and corresponding license files are provided beside the files. Make sure they end up in third party list or in a distributed bundle

Done! See diff.patch for all my changes.

pavelfeldman · 2025-01-13T03:59:36Z

packages/playwright-core/src/utils/tar/extract.js

+  }
+
+  shiftFirst (size) {
+    return this._buffered === 0 ? null : this._next(size)


Is this a bug on 21st line of this library? (I don't see this._buffered defined)

looks like it! It's also in the original: https://github.com/mafintosh/tar-stream/blob/126968fd3c4a39eba5f8318c255e04cedbbad176/extract.js#L23C17-L23C26

pavelfeldman · 2025-01-13T04:00:48Z

packages/playwright-core/src/utils/tar/extract.js

@@ -0,0 +1,311 @@
+const { Writable, Readable, getStreamError } = require('stream')


getStreamError is not a thing. How is it supposed to work?

Good find! Removed the usage of it by moving _predestroy into _destroy. Once I add the diff this will make more sense.

pavelfeldman · 2025-01-13T04:29:52Z

packages/playwright-core/src/utils/tar/headers.js

+    const len = parseInt(buf.toString('ascii', 0, i), 10)
+    if (!len) return result
+
+    const b = buf.subarray('ascii', i + 1, len - 1)


ChatGPT thinks it is a bug since the value called len is used in the subarray(start, end) signature. Given that the start is i + 1, which points to right after the parsed len, len - 1 can't be a valid end, did they want to say i + len here?

this is what the original library does 🤷
https://github.com/mafintosh/tar-stream/blob/126968fd3c4a39eba5f8318c255e04cedbbad176/headers.js#L40

This seems to be some sort of special case checking. Maybe in some implementations of TAR/PAX, len doesn't contain a length, but an index?

github-actions · 2025-01-13T11:57:35Z

Test results for "tests 1"

6 failed
❌ [installation tests] › tests/playwright-cdn.spec.ts:75:5 › npx playwright install should not hang when CDN closes the connection @package-installations-macos-latest
❌ [installation tests] › tests/playwright-cdn.spec.ts:102:5 › npx playwright install should not hang when CDN TCP connection stalls @package-installations-macos-latest
❌ [installation tests] › tests/playwright-cdn.spec.ts:75:5 › npx playwright install should not hang when CDN closes the connection @package-installations-ubuntu-latest
❌ [installation tests] › tests/playwright-cdn.spec.ts:102:5 › npx playwright install should not hang when CDN TCP connection stalls @package-installations-ubuntu-latest
❌ [installation tests] › tests/playwright-cdn.spec.ts:75:5 › npx playwright install should not hang when CDN closes the connection @package-installations-windows-latest
❌ [installation tests] › tests/playwright-cdn.spec.ts:102:5 › npx playwright install should not hang when CDN TCP connection stalls @package-installations-windows-latest

7 flaky

⚠️ [firefox-page] › tests/page/page-evaluate.spec.ts:403:3 › should throw for too deep reference chain @firefox-ubuntu-22.04-node18
⚠️ [playwright-test] › tests/ui-mode-trace.spec.ts:341:5 › should work behind reverse proxy @macos-latest-node18-1
⚠️ [installation tests] › tests/playwright-electron-should-work.spec.ts:31:5 › electron should work with special characters in path @package-installations-macos-latest
⚠️ [webkit-library] › tests/library/browsertype-connect.spec.ts:198:5 › launchServer › should be able to connect two browsers at the same time @webkit-ubuntu-22.04-node18
⚠️ [webkit-library] › tests/library/proxy.spec.ts:93:11 › should proxy local network requests › by default › loopback address @webkit-ubuntu-22.04-node18
⚠️ [webkit-library] › tests/library/trace-viewer.spec.ts:1575:1 › canvas clipping in iframe @webkit-ubuntu-22.04-node18
⚠️ [webkit-page] › tests/page/page-set-input-files.spec.ts:147:3 › should upload large file @webkit-ubuntu-22.04-node18

37574 passed, 648 skipped
✔️✔️✔️

Merge workflow run.

Skn0tt added 4 commits December 16, 2024 16:43

add second branch

a9b40bd

move chromium, chromium-tot, and webkit to brotli

0b139f1

fix import

98d6138

feat: download browsers as .tar.br

657e435

Skn0tt self-assigned this Dec 16, 2024

This comment has been minimized.

Sign in to view

Skn0tt added 6 commits December 18, 2024 12:47

respect revisionOverrides

c5ab0c5

update thirdparty

3b72c3b

update log line matcher

411cf29

update expected error codes

d880a22

ffmpeg also has brotli builds now

f39ae50

remove conflicting tar-fs types

f071337

This comment has been minimized.

Sign in to view

add workaround for manually-created ffmpeg archive

45b84b7

This comment has been minimized.

Sign in to view

remove console statement

a497b6b

This comment has been minimized.

Sign in to view

Skn0tt marked this pull request as ready for review December 18, 2024 14:28

Merge branch 'main' into tar-download-3rd-party-lib

b048d2a

This comment has been minimized.

Sign in to view

mxschmitt reviewed Dec 18, 2024

View reviewed changes

packages/playwright-core/src/server/registry/oopDownloadBrowserMain.ts Outdated Show resolved Hide resolved

pavelfeldman reviewed Dec 18, 2024

View reviewed changes

Skn0tt added 2 commits December 19, 2024 16:47

Merge branch 'main' into tar-download-3rd-party-lib

affff4d

adapt to new ffmpeg roll

879afb1

mxschmitt reviewed Dec 19, 2024

View reviewed changes

This comment has been minimized.

Sign in to view

pavelfeldman reviewed Jan 4, 2025

View reviewed changes

Skn0tt added 4 commits January 6, 2025 14:59

vendor modified tar-fs

16ff2df

update thirdpartynotices

eaf6a52

reset package-lock

220e28b

reset package.json

0ccbef1

This comment has been minimized.

Sign in to view

pavelfeldman reviewed Jan 13, 2025

View reviewed changes

Skn0tt added 8 commits January 13, 2025 11:00

Merge branch 'main' into tar-download-3rd-party-lib

30c5ec6

firefox and chrome-headless-shell have tar now

e2b1500

reuse existing code better

25494cb

continue using close

2224fd2

move to third_party

41a552f

add license/readme

dce95a5

remove getStreamError

2749367

add diff

7744cc5

This comment has been minimized.

Sign in to view

Skn0tt added 2 commits January 13, 2025 11:58

fix _predestroy

c717419

add notices

a2d214f

This comment has been minimized.

Sign in to view

Skn0tt added 2 commits January 13, 2025 12:13

fix linter

5086a56

add deps

e24a0d7

This comment has been minimized.

Sign in to view

		@@ -0,0 +1 @@
		This directory contains a modified copy of the `tar-stream` library that's used exclusively to extract TAR files.

		@@ -0,0 +1,311 @@
		const { Writable, Readable, getStreamError } = require('stream')

feat: download browsers as TAR #34033

Are you sure you want to change the base?

feat: download browsers as TAR #34033

Conversation

Skn0tt commented Dec 16, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Skn0tt Dec 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment has been minimized.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pavelfeldman commented Jan 4, 2025

Skn0tt commented Jan 6, 2025 • edited Loading

Footnotes

Skn0tt commented Jan 6, 2025 • edited Loading

This comment has been minimized.

Choose a reason for hiding this comment

Skn0tt Jan 13, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Skn0tt Jan 13, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Jan 13, 2025

Test results for "tests 1"

Skn0tt Dec 19, 2024 •

edited

Loading

Skn0tt commented Jan 6, 2025 •

edited

Loading

Skn0tt commented Jan 6, 2025 •

edited

Loading

Skn0tt Jan 13, 2025 •

edited

Loading

Skn0tt Jan 13, 2025 •

edited

Loading