brozzle-page Not Working With Recent Version of Google Chrome

[treid003]

When I recently used a brozzle-page command with a recent version of Google Chrome, I noticed that brozzler does not load the web page that should be archived.

This results in the web page not being archived successfully.

WARC file: WARCPROX-20230519163909687-00000-0so5t1md.warc

This issue also occurred when trying to archive other web pages:

• https://www.tomshardware.com/reviews/nvidia-geforce-rtx-4090-review/3
• https://armorgames.com/
• https://www.w3schools.com/tags/default.asp
• https://www.geeksforgeeks.org/python-classes-and-objects/?ref=lbp
• https://mortalkombat.fandom.com/wiki/Tag_Team_Ladder
• https://www.youtube.com/watch?v=PZShwWiepeY
• https://arquivo.pt/services/complete-page?url=https://pt.ign.com/&timestamp=20220202150304
• https://www.bloomberg.com/news/newsletters/2023-01-08/when-will-apple-launch-the-reality-pro-mixed-reality-headset-apple-2023-devices-lcnfzkc7
• https://www.theverge.com/23496511/ces-2023-news-products-announcements-tvs-laptops-fitness-trackers

The commands I used are listed below (video example):

warcprox -p 8081 -d ./warcs/IGN/brozzle_page/2023_05_19 --dedup-db-file /dev/null

export BROZZLER_EXTRA_CHROME_ARGS="--ignore-certificate-errors"

brozzle-page --chrome-exe '/usr/bin/google-chrome' --proxy localhost:8081 'https://www.ign.com/articles/the-last-of-us-season-1-review'

A “WebSocketBadStatusException: Handshake status 403 Forbidden” occurred when recently running these commands on Ubuntu (22.04.2 LTS and 20.04.6 LTS) and macOS (Ventura 13.3.1).

When I used these commands earlier this year it was working successfully (video):

After noticing this issue, I went through the recent stable versions of Google Chrome and found the last stable version that worked with the brozzle-page command was version 109.0.5414.119 which was released on January 24, 2023.

chrome.deb URI (109.0.5414.119): https://dl.google.com/linux/chrome/deb/pool/main/g/google-chrome-stable/google-chrome-stable_109.0.5414.119-1_amd64.deb

Crawling session: https://youtu.be/A-zr6zVTZSo?t=5569

Replay session: https://youtu.be/A-zr6zVTZSo?t=6345

The first stable version of Chrome that did not work with the brozzle-page command is version 111.0.5563.110 which was released on March 21, 2023.

chrome.deb URI (111.0.5563.110): https://dl.google.com/linux/chrome/deb/pool/main/g/google-chrome-stable/google-chrome-stable_111.0.5563.110-1_amd64.deb

Crawling session: https://youtu.be/A-zr6zVTZSo?t=4903

Replay session: https://youtu.be/A-zr6zVTZSo?t=4992

Chrome release blog post for 111.0.5563.110: https://chromereleases.googleblog.com/2023/03/stable-channel-update-for-desktop_21.html

• Most of the fixed security issues for version 111.0.5563.110 were associated with an attacker creating an HTML page and then exploiting heap corruption:
  • High CVE-2023-1528: https://nvd.nist.gov/vuln/detail/CVE-2023-1528
  • High CVE-2023-1530: https://nvd.nist.gov/vuln/detail/CVE-2023-1530
  • High CVE-2023-1531: https://nvd.nist.gov/vuln/detail/CVE-2023-1531
  • High CVE-2023-1532: https://nvd.nist.gov/vuln/detail/CVE-2023-1532
  • High CVE-2023-1533: https://nvd.nist.gov/vuln/detail/CVE-2023-1533
  • High CVE-2023-1534: https://nvd.nist.gov/vuln/detail/CVE-2023-1534
• There was also a security fix that involved internal security work so there was less details for this update to Chrome.

[galgeek]

Thank you, @treid003, for your detailed report!

Current brozzler code is compatible with Chrome up to v.110. This linux version runs without apparent issue:

https://dl.google.com/linux/chrome/deb/pool/main/g/google-chrome-stable/google-chrome-stable_110.0.5481.177-1_amd64.deb

I've also run current code on Apple M1/M2 with Thorium v.110, using the branch here:
#255

[treid003]

Thanks for testing v.110.

When going through the different Chrome versions I accidentally skipped v.110 since the blog post titles were different for v.110 Chrome releases.

[benTawiah]

How do you run it headmode

[benTawiah]

I mean how can i make it open the browser when crawling

[galgeek]

To make brozzler open the browser when crawling, you can remove / comment out the code in brozzler's chrome.py that adds the various --headless params for chrome.

see: https://github.com/internetarchive/brozzler/blob/master/brozzler/chrome.py#L211

[galgeek]

Note that brozzler now runs with Chrome v.130 (and possibly later)