Chromedriver `Segmentation fault` when used with jemalloc buildpack

[guiferrpereira]

We just migrated from the 2 packages to this one, but we are getting segmentation fault on our heroku dyno:

[edmorley]

Hi!

When I try those commands I can't reproduce the error:

$ mkdir chrome-test && cd $_
$ git init && touch foo && git add -A && git commit -m '.'
$ heroku create --buildpack heroku-community/chrome-for-testing
$ git push heroku main
...
remote: -----> Building on the Heroku-22 stack
remote: -----> Using buildpack: heroku-community/chrome-for-testing
remote: -----> Chrome for Testing app detected
remote: -----> Installing Chrome for Testing
remote:        Resolved STABLE version 125.0.6422.141
remote:        Downloading Chrome
remote:        Downloading Chromedriver
remote: -----> Installing Chrome dependencies
...
remote:        Writing profile script
remote:        Installed Chrome dependencies for heroku-22
remote:        Adding executables to PATH
remote:        /tmp/build_81b70c10/.chrome-for-testing/chrome-linux64/chrome
remote:        /tmp/build_81b70c10/.chrome-for-testing/chromedriver-linux64/chromedriver
remote:        Installed Chrome for Testing STABLE version 125.0.6422.141
remote: -----> Discovering process types
remote:        Procfile declares types -> (none)
remote:
remote: -----> Compressing...
remote:        Done: 160.4M

$ heroku run bash
Running bash on ⬢ serene-falls-81270... up, run.6115 (Basic)
~ $ chromedriver --headless
Starting ChromeDriver 125.0.6422.141 (4b1e83937122185343ba92e909b021f307c719ca-refs/branch-heads/6422@{#1186}) on port 9515
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
ChromeDriver was started successfully.

I also tried with Heroku-20 (you didn't say what stack your app was running on), and that worked too.

That suggests it might be something related to one of the following:

• other buildpacks being used by your ap
• config vars set on your app (eg things like LD_LIBRARY_PATH, PATH etc, which in general should only be set by buildpacks)
• bad state in the build cache (https://help.heroku.com/18PI5RSY/how-do-i-clear-the-build-cache)

Perhaps start by clearing the build cache, triggering a new build, and seeing if that helps?

[guiferrpereira]

Hi @edmorley, thanks for your answer.

The stack is Stack heroku-20 indeed.

$ heroku builds:cache:purge -a xxx
 ▸    WARNING: This will delete the build cache for ⬢ xxx
 ▸    Clearing your build cache can have unintended side effects, such as updating your language version if you don't explicitly specify one.
 ▸    To proceed, type xxx or re-run this command with --confirm xxx

> xxx
Purging build cache for ⬢ xxx... done
$ heroku run bash --app xxx
Running bash on ⬢ xxx... up, run.9320 (Basic)
~ $ echo $LD_LIBRARY_PATH
/app/.apt/usr/lib/x86_64-linux-gnu:/app/.apt/usr/lib/i386-linux-gnu:/app/.apt/usr/lib:
~ $ echo $PATH
/app/bin:/app/vendor/bundle/bin:/app/vendor/bundle/ruby/3.2.0/bin:/app/vendor/libreoffice/opt/libreoffice24.2/program:/app/.chrome-for-testing/chrome-linux64:/app/.chrome-for-testing/chromedriver-linux64:/app/.apt/usr/bin:/usr/local/bin:/usr/bin:/bin:/app/vendor/jemalloc/bin
~ $ which chrome
/app/.chrome-for-testing/chrome-linux64/chrome
~ $ which chromedriver
/app/.chrome-for-testing/chromedriver-linux64/chromedriver
~ $ chromedriver --headless
Starting ChromeDriver 125.0.6422.141 (4b1e83937122185343ba92e909b021f307c719ca-refs/branch-heads/6422@{#1186}) on port 9515
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
Segmentation fault
~ $ 

We don't have any confirguration for $LD_LIBRARY_PATHor $PATH

[edmorley]

@guiferrpereira Thank you for confirming the stack. I don't see another build being performed in your console output after clearing the cache. Was a new build performed?

[guiferrpereira]

Yes i did it trough heroku UI.

[guiferrpereira]

The buildpacks that we have:

This is working with the old ones:

With the old config I get:

$ heroku run bash --app prod-xxx
Running bash on ⬢ prod-xxx... up, run.8180 (Standard-1X)
~ $ chromedriver
Starting ChromeDriver 125.0.6422.141 (4b1e83937122185343ba92e909b021f307c719ca-refs/branch-heads/6422@{#1186}) on port 9515
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
ChromeDriver was started successfully.

[guiferrpereira]

The build logs:

-----> Using buildpacks:
       1. https://github.com/gaffneyc/heroku-buildpack-jemalloc.git
       2. https://github.com/heroku/heroku-buildpack-chrome-for-testing
       3. https://github.com/kudocs/buildpack-libreoffice-for-heroku.git
       4. heroku/ruby
-----> jemalloc app detected
-----> jemalloc: Installing 5.3.0 from source
       jemalloc: Downloading https://github.com/jemalloc/jemalloc/releases/download/5.3.0/jemalloc-5.3.0.tar.bz2
       jemalloc: Compiling 5.3.0
-----> jemalloc: Done
-----> Chrome for Testing app detected
-----> Installing Chrome for Testing
       Resolved STABLE version 125.0.6422.141
       Downloading Chrome
       Downloading Chromedriver
-----> Installing Chrome dependencies
       Purging cache
       Updating apt caches
       Get:1 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]
       Get:2 http://archive.ubuntu.com/ubuntu focal-security InRelease [128 kB]
       Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB]
       Get:4 http://apt.postgresql.org/pub/repos/apt focal-pgdg InRelease [123 kB]
       Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages [1,275 kB]
       Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [11.3 MB]
       Get:7 http://archive.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,211 kB]
       Get:8 http://archive.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,675 kB]
       Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [4,154 kB]
       Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,509 kB]
       Get:11 http://apt.postgresql.org/pub/repos/apt focal-pgdg/main amd64 Packages [533 kB]
       Fetched 24.3 MB in 2s (13.3 MB/s)
       Reading package lists...
       Fetching .debs for gconf-service
       Reading package lists...
       Building dependency tree...
       The following additional packages will be installed:
         gconf-service-backend gconf2-common libdbus-glib-1-2 libgconf-2-4
       The following NEW packages will be installed:
         gconf-service gconf-service-backend gconf2-common libdbus-glib-1-2
         libgconf-2-4
       0 upgraded, 5 newly installed, 0 to remove and 5 not upgraded.
       Need to get 918 kB of archives.
       After this operation, 8,309 kB of additional disk space will be used.
       Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbus-glib-1-2 amd64 0.110-5fakssync1 [59.1 kB]
       Get:2 http://archive.ubuntu.com/ubuntu focal/universe amd64 gconf2-common all 3.2.6-6ubuntu1 [698 kB]
       Get:3 http://archive.ubuntu.com/ubuntu focal/universe amd64 libgconf-2-4 amd64 3.2.6-6ubuntu1 [84.8 kB]
       Get:4 http://archive.ubuntu.com/ubuntu focal/universe amd64 gconf-service-backend amd64 3.2.6-6ubuntu1 [58.6 kB]
       Get:5 http://archive.ubuntu.com/ubuntu focal/universe amd64 gconf-service amd64 3.2.6-6ubuntu1 [17.4 kB]
       Fetched 918 kB in 0s (7,365 kB/s)
       Download complete and in download only mode
W: --force-yes is deprecated, use one of the options starting with --allow instead.
       Fetching .debs for libappindicator1
       Reading package lists...
       Building dependency tree...
       The following additional packages will be installed:
         adwaita-icon-theme gtk-update-icon-cache humanity-icon-theme libatk1.0-0
         libatk1.0-data libdbusmenu-glib4 libdbusmenu-gtk4 libgail-common libgail18
         libgtk2.0-0 libgtk2.0-bin libgtk2.0-common libxcomposite1 libxcursor1
         libxdamage1 libxfixes3 libxi6 libxinerama1 libxrandr2 ubuntu-mono
       Suggested packages:
         indicator-application gvfs
       The following NEW packages will be installed:
         adwaita-icon-theme gtk-update-icon-cache humanity-icon-theme
         libappindicator1 libatk1.0-0 libatk1.0-data libdbusmenu-glib4
         libdbusmenu-gtk4 libgail-common libgail18 libgtk2.0-0 libgtk2.0-bin
         libgtk2.0-common libxcomposite1 libxcursor1 libxdamage1 libxfixes3 libxi6
         libxinerama1 libxrandr2 ubuntu-mono
       0 upgraded, 21 newly installed, 0 to remove and 5 not upgraded.
       Need to get 7,162 kB of archives.
       After this operation, 40.4 MB of additional disk space will be used.
       Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 gtk-update-icon-cache amd64 3.24.20-0ubuntu1.1 [28.8 kB]
       Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 humanity-icon-theme all 0.6.15 [1,250 kB]
       Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 ubuntu-mono all 19.04-0ubuntu3 [147 kB]
       Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 adwaita-icon-theme all 3.36.1-2ubuntu0.20.04.2 [3,441 kB]
       Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 libdbusmenu-glib4 amd64 16.04.1+18.10.20180917-0ubuntu6 [41.2 kB]
       Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 libatk1.0-data all 2.35.1-1ubuntu2 [2,964 B]
       Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 libatk1.0-0 amd64 2.35.1-1ubuntu2 [45.5 kB]
       Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdbusmenu-gtk4 amd64 16.04.1+18.10.20180917-0ubuntu6 [27.5 kB]
       Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libgtk2.0-common all 2.24.32-4ubuntu4 [126 kB]
       Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 libxcomposite1 amd64 1:0.4.5-1 [6,976 B]
       Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 libxfixes3 amd64 1:5.0.3-2 [10.9 kB]
       Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 libxcursor1 amd64 1:1.2.0-2 [20.1 kB]
       Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 libxdamage1 amd64 1:1.1.5-2 [6,996 B]
       Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 libxi6 amd64 2:1.7.10-0ubuntu1 [29.9 kB]
       Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 libxinerama1 amd64 2:1.1.4-2 [6,904 B]
       Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 libxrandr2 amd64 2:1.5.2-0ubuntu1 [18.5 kB]
       Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 libgtk2.0-0 amd64 2.24.32-4ubuntu4 [1,791 kB]
       Get:18 http://archive.ubuntu.com/ubuntu focal/universe amd64 libappindicator1 amd64 12.10.1+20.04.20200408.1-0ubuntu1 [22.8 kB]
       Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 libgail18 amd64 2.24.32-4ubuntu4 [14.7 kB]
       Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 libgail-common amd64 2.24.32-4ubuntu4 [116 kB]
       Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 libgtk2.0-bin amd64 2.24.32-4ubuntu4 [7,728 B]
       Fetched 7,162 kB in 0s (33.4 MB/s)
       Download complete and in download only mode
W: --force-yes is deprecated, use one of the options starting with --allow instead.
       Fetching .debs for libasound2
       Reading package lists...
       Building dependency tree...
       The following additional packages will be installed:
         alsa-topology-conf alsa-ucm-conf libasound2-data
       Suggested packages:
         libasound2-plugins alsa-utils
       The following NEW packages will be installed:
         alsa-topology-conf alsa-ucm-conf libasound2 libasound2-data
       0 upgraded, 4 newly installed, 0 to remove and 5 not upgraded.
       Need to get 389 kB of archives.
       After this operation, 1,879 kB of additional disk space will be used.
       Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 alsa-topology-conf all 1.2.2-1 [7,364 B]
       Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 alsa-ucm-conf all 1.2.2-1ubuntu0.13 [27.0 kB]
       Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libasound2-data all 1.2.2-2.1ubuntu2.5 [20.1 kB]
       Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libasound2 amd64 1.2.2-2.1ubuntu2.5 [335 kB]
       Fetched 389 kB in 1s (691 kB/s)
       Download complete and in download only mode
W: --force-yes is deprecated, use one of the options starting with --allow instead.
       Fetching .debs for libatk1.0-0
       Reading package lists...
       Building dependency tree...
       The following additional packages will be installed:
         libatk1.0-data
       The following NEW packages will be installed:
         libatk1.0-0 libatk1.0-data
       0 upgraded, 2 newly installed, 0 to remove and 5 not upgraded.
       Need to get 0 B/48.5 kB of archives.
       After this operation, 246 kB of additional disk space will be used.
       Download complete and in download only mode
W: --force-yes is deprecated, use one of the options starting with --allow instead.
       Fetching .debs for libatk-bridge2.0-0
       Reading package lists...
       Building dependency tree...
       The following additional packages will be installed:
         at-spi2-core libatk1.0-0 libatk1.0-data libatspi2.0-0 libxtst6
       The following NEW packages will be installed:
         at-spi2-core libatk-bridge2.0-0 libatk1.0-0 libatk1.0-data libatspi2.0-0
         libxtst6
       0 upgraded, 6 newly installed, 0 to remove and 5 not upgraded.
       Need to get 184 kB/232 kB of archives.
       After this operation, 1,074 kB of additional disk space will be used.
       Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libatspi2.0-0 amd64 2.36.0-2 [64.2 kB]
       Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libxtst6 amd64 2:1.2.3-1 [12.8 kB]
       Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 at-spi2-core amd64 2.36.0-2 [48.7 kB]
       Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libatk-bridge2.0-0 amd64 2.34.2-0ubuntu2~20.04.1 [58.2 kB]
       Fetched 184 kB in 1s (354 kB/s)
       Download complete and in download only mode
W: --force-yes is deprecated, use one of the options starting with --allow instead.
       Fetching .debs for libcairo-gobject2
       Reading package lists...
       Building dependency tree...
       0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 5 not upgraded.
       Need to get 17.2 kB of archives.
       After this operation, 0 B of additional disk space will be used.
       Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libcairo-gobject2 amd64 1.16.0-4ubuntu1 [17.2 kB]
       Fetched 17.2 kB in 0s (338 kB/s)
       Download complete and in download only mode
W: --force-yes is deprecated, use one of the options starting with --allow instead.
       Fetching .debs for libdrm2
       Reading package lists...
       Building dependency tree...
       The following additional packages will be installed:
         libdrm-common
       The following NEW packages will be installed:
         libdrm-common libdrm2
       0 upgraded, 2 newly installed, 0 to remove and 5 not upgraded.
       Need to get 39.5 kB of archives.
       After this operation, 177 kB of additional disk space will be used.
       Get:1 http://archive.ubuntu.com/ubuntu focal-security/main amd64 libdrm-common all 2.4.107-8ubuntu1~20.04.2 [5,396 B]
       Get:2 http://archive.ubuntu.com/ubuntu focal-security/main amd64 libdrm2 amd64 2.4.107-8ubuntu1~20.04.2 [34.1 kB]
       Fetched 39.5 kB in 0s (659 kB/s)
       Download complete and in download only mode
W: --force-yes is deprecated, use one of the options starting with --allow instead.
       Fetching .debs for libgbm1
       Reading package lists...
       Building dependency tree...
       The following additional packages will be installed:
         libdrm-common libdrm2 libwayland-server0
       The following NEW packages will be installed:
         libdrm-common libdrm2 libgbm1 libwayland-server0
       0 upgraded, 4 newly installed, 0 to remove and 5 not upgraded.
       Need to get 60.5 kB/100.0 kB of archives.
       After this operation, 434 kB of additional disk space will be used.
       Get:1 http://archive.ubuntu.com/ubuntu focal-security/main amd64 libwayland-server0 amd64 1.18.0-1ubuntu0.1 [31.3 kB]
       Get:2 http://archive.ubuntu.com/ubuntu focal-security/main amd64 libgbm1 amd64 21.2.6-0ubuntu0.1~20.04.2 [29.2 kB]
       Fetched 60.5 kB in 0s (685 kB/s)
       Download complete and in download only mode
W: --force-yes is deprecated, use one of the options starting with --allow instead.
       Fetching .debs for libgconf-2-4
       Reading package lists...
       Building dependency tree...
       The following additional packages will be installed:
         gconf-service gconf-service-backend gconf2-common libdbus-glib-1-2
       The following NEW packages will be installed:
         gconf-service gconf-service-backend gconf2-common libdbus-glib-1-2
         libgconf-2-4
       0 upgraded, 5 newly installed, 0 to remove and 5 not upgraded.
       Need to get 0 B/918 kB of archives.
       After this operation, 8,309 kB of additional disk space will be used.
       Download complete and in download only mode
W: --force-yes is deprecated, use one of the options starting with --allow instead.
       Fetching .debs for libgtk-3-0
       Reading package lists...
       Building dependency tree...
       The following additional packages will be installed:
         adwaita-icon-theme at-spi2-core dbus dbus-user-session
         dconf-gsettings-backend dconf-service dmsetup glib-networking
         glib-networking-common glib-networking-services gsettings-desktop-schemas
         gtk-update-icon-cache humanity-icon-theme libapparmor1 libatk-bridge2.0-0
         libatk1.0-0 libatk1.0-data libatspi2.0-0 libcolord2 libcryptsetup12
         libdconf1 libdevmapper1.02.1 libepoxy0 libgtk-3-bin libgtk-3-common
         libip4tc2 libjson-glib-1.0-0 libjson-glib-1.0-common libnss-systemd
         libpam-systemd libproxy1v5 librest-0.7-0 libsoup-gnome2.4-1 libsoup2.4-1
         libwayland-client0 libwayland-cursor0 libwayland-egl1 libxcomposite1
         libxcursor1 libxdamage1 libxfixes3 libxi6 libxinerama1 libxkbcommon0
         libxrandr2 libxtst6 networkd-dispatcher python3-dbus python3-gi systemd
         systemd-sysv systemd-timesyncd ubuntu-mono xkb-data
       Suggested packages:
         colord gvfs iw | wireless-tools python-dbus-doc python3-dbus-dbg
         systemd-container policykit-1
       The following NEW packages will be installed:
         adwaita-icon-theme at-spi2-core dbus dbus-user-session
         dconf-gsettings-backend dconf-service dmsetup glib-networking
         glib-networking-common glib-networking-services gsettings-desktop-schemas
         gtk-update-icon-cache humanity-icon-theme libapparmor1 libatk-bridge2.0-0
         libatk1.0-0 libatk1.0-data libatspi2.0-0 libcolord2 libcryptsetup12
         libdconf1 libdevmapper1.02.1 libepoxy0 libgtk-3-0 libgtk-3-bin
         libgtk-3-common libip4tc2 libjson-glib-1.0-0 libjson-glib-1.0-common
         libnss-systemd libpam-systemd libproxy1v5 librest-0.7-0 libsoup-gnome2.4-1
         libsoup2.4-1 libwayland-client0 libwayland-cursor0 libwayland-egl1
         libxcomposite1 libxcursor1 libxdamage1 libxfixes3 libxi6 libxinerama1
         libxkbcommon0 libxrandr2 libxtst6 networkd-dispatcher python3-dbus
         python3-gi systemd systemd-sysv systemd-timesyncd ubuntu-mono xkb-data
       0 upgraded, 55 newly installed, 0 to remove and 5 not upgraded.
       Need to get 9,322 kB/14.5 MB of archives.
       After this operation, 74.2 MB of additional disk space will be used.
       Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libapparmor1 amd64 2.13.3-7ubuntu5.3 [35.4 kB]
       Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 libdevmapper1.02.1 amd64 2:1.02.167-1ubuntu1 [127 kB]
       Get:3 http://archive.ubuntu.com/ubuntu focal-security/main amd64 libcryptsetup12 amd64 2:2.2.2-3ubuntu2.4 [166 kB]
       Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libip4tc2 amd64 1.8.4-3ubuntu2.1 [19.1 kB]
       Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 systemd-timesyncd amd64 245.4-4ubuntu3.23 [28.1 kB]
       Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 systemd amd64 245.4-4ubuntu3.23 [3,811 kB]
       Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 systemd-sysv amd64 245.4-4ubuntu3.23 [10.3 kB]
       Get:8 http://archive.ubuntu.com/ubuntu focal-security/main amd64 dbus amd64 1.12.16-2ubuntu2.3 [151 kB]
       Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 dmsetup amd64 2:1.02.167-1ubuntu1 [75.6 kB]
       Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libnss-systemd amd64 245.4-4ubuntu3.23 [96.2 kB]
       Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpam-systemd amd64 245.4-4ubuntu3.23 [186 kB]
       Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-dbus amd64 1.2.16-1build1 [94.0 kB]
       Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-gi amd64 3.36.0-1 [165 kB]
       Get:14 http://archive.ubuntu.com/ubuntu focal-security/main amd64 networkd-dispatcher all 2.1-2~ubuntu20.04.3 [15.5 kB]
       Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 xkb-data all 2.29-2 [349 kB]
       Get:16 http://archive.ubuntu.com/ubuntu focal-security/main amd64 dbus-user-session amd64 1.12.16-2ubuntu2.3 [9,424 B]
       Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 libdconf1 amd64 0.36.0-1 [37.4 kB]
       Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 dconf-service amd64 0.36.0-1 [29.8 kB]
       Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 dconf-gsettings-backend amd64 0.36.0-1 [22.5 kB]
       Get:20 http://archive.ubuntu.com/ubuntu focal-security/main amd64 libproxy1v5 amd64 0.4.15-10ubuntu1.2 [49.1 kB]
       Get:21 http://archive.ubuntu.com/ubuntu focal-security/main amd64 glib-networking-common all 2.64.2-1ubuntu0.1 [5,040 B]
       Get:22 http://archive.ubuntu.com/ubuntu focal-security/main amd64 glib-networking-services amd64 2.64.2-1ubuntu0.1 [10.6 kB]
       Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 gsettings-desktop-schemas all 3.36.0-1ubuntu1 [29.0 kB]
       Get:24 http://archive.ubuntu.com/ubuntu focal-security/main amd64 glib-networking amd64 2.64.2-1ubuntu0.1 [58.2 kB]
       Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 libcolord2 amd64 1.4.4-2 [133 kB]
       Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 libepoxy0 amd64 1.5.4-1 [191 kB]
       Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 libjson-glib-1.0-common all 1.4.4-2ubuntu2 [3,468 B]
       Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 libjson-glib-1.0-0 amd64 1.4.4-2ubuntu2 [59.4 kB]
       Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 libsoup2.4-1 amd64 2.70.0-1 [262 kB]
       Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 libsoup-gnome2.4-1 amd64 2.70.0-1 [6,136 B]
       Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 librest-0.7-0 amd64 0.8.1-1 [32.2 kB]
       Get:32 http://archive.ubuntu.com/ubuntu focal-security/main amd64 libwayland-client0 amd64 1.18.0-1ubuntu0.1 [23.9 kB]
       Get:33 http://archive.ubuntu.com/ubuntu focal-security/main amd64 libwayland-cursor0 amd64 1.18.0-1ubuntu0.1 [10.3 kB]
       Get:34 http://archive.ubuntu.com/ubuntu focal-security/main amd64 libwayland-egl1 amd64 1.18.0-1ubuntu0.1 [5,596 B]
       Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 libxkbcommon0 amd64 0.10.0-1 [98.4 kB]
       Get:36 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libgtk-3-common all 3.24.20-0ubuntu1.1 [234 kB]
       Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libgtk-3-0 amd64 3.24.20-0ubuntu1.1 [2,620 kB]
       Get:38 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libgtk-3-bin amd64 3.24.20-0ubuntu1.1 [61.9 kB]
       Fetched 9,322 kB in 4s (2,199 kB/s)
       Download complete and in download only mode
W: --force-yes is deprecated, use one of the options starting with --allow instead.
       Fetching .debs for libnspr4
       Reading package lists...
       Building dependency tree...
       0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 5 not upgraded.
       Need to get 108 kB of archives.
       After this operation, 0 B of additional disk space will be used.
       Get:1 http://archive.ubuntu.com/ubuntu focal-security/main amd64 libnspr4 amd64 2:4.35-0ubuntu0.20.04.1 [108 kB]
       Fetched 108 kB in 0s (274 kB/s)
       Download complete and in download only mode
W: --force-yes is deprecated, use one of the options starting with --allow instead.
       Fetching .debs for libnss3
       Reading package lists...
       Building dependency tree...
       0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 5 not upgraded.
       Need to get 1,391 kB of archives.
       After this operation, 0 B of additional disk space will be used.
       Get:1 http://archive.ubuntu.com/ubuntu focal-security/main amd64 libnss3 amd64 2:3.98-0ubuntu0.20.04.2 [1,391 kB]
       Fetched 1,391 kB in 0s (10.0 MB/s)
       Download complete and in download only mode
W: --force-yes is deprecated, use one of the options starting with --allow instead.
       Fetching .debs for libx11-xcb1
       Reading package lists...
       Building dependency tree...
       The following NEW packages will be installed:
         libx11-xcb1
       0 upgraded, 1 newly installed, 0 to remove and 5 not upgraded.
       Need to get 9,448 B of archives.
       After this operation, 87.0 kB of additional disk space will be used.
       Get:1 http://archive.ubuntu.com/ubuntu focal-security/main amd64 libx11-xcb1 amd64 2:1.6.9-2ubuntu1.6 [9,448 B]
       Fetched 9,448 B in 0s (263 kB/s)
       Download complete and in download only mode
W: --force-yes is deprecated, use one of the options starting with --allow instead.
       Fetching .debs for libxcb-dri3-0
       Reading package lists...
       Building dependency tree...
       The following NEW packages will be installed:
         libxcb-dri3-0
       0 upgraded, 1 newly installed, 0 to remove and 5 not upgraded.
       Need to get 6,552 B of archives.
       After this operation, 43.0 kB of additional disk space will be used.
       Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libxcb-dri3-0 amd64 1.14-2 [6,552 B]
       Fetched 6,552 B in 0s (176 kB/s)
       Download complete and in download only mode
W: --force-yes is deprecated, use one of the options starting with --allow instead.
       Fetching .debs for libxcomposite1
       Reading package lists...
       Building dependency tree...
       The following NEW packages will be installed:
         libxcomposite1
       0 upgraded, 1 newly installed, 0 to remove and 5 not upgraded.
       Need to get 0 B/6,976 B of archives.
       After this operation, 32.8 kB of additional disk space will be used.
       Download complete and in download only mode
W: --force-yes is deprecated, use one of the options starting with --allow instead.
       Fetching .debs for libxcursor1
       Reading package lists...
       Building dependency tree...
       The following additional packages will be installed:
         libxfixes3
       The following NEW packages will be installed:
         libxcursor1 libxfixes3
       0 upgraded, 2 newly installed, 0 to remove and 5 not upgraded.
       Need to get 0 B/31.0 kB of archives.
       After this operation, 117 kB of additional disk space will be used.
       Download complete and in download only mode
W: --force-yes is deprecated, use one of the options starting with --allow instead.
       Fetching .debs for libxdamage1
       Reading package lists...
       Building dependency tree...
       The following NEW packages will be installed:
         libxdamage1
       0 upgraded, 1 newly installed, 0 to remove and 5 not upgraded.
       Need to get 0 B/6,996 B of archives.
       After this operation, 31.7 kB of additional disk space will be used.
       Download complete and in download only mode
W: --force-yes is deprecated, use one of the options starting with --allow instead.
       Fetching .debs for libxfixes3
       Reading package lists...
       Building dependency tree...
       The following NEW packages will be installed:
         libxfixes3
       0 upgraded, 1 newly installed, 0 to remove and 5 not upgraded.
       Need to get 0 B/10.9 kB of archives.
       After this operation, 48.1 kB of additional disk space will be used.
       Download complete and in download only mode
W: --force-yes is deprecated, use one of the options starting with --allow instead.
       Fetching .debs for libxi6
       Reading package lists...
       Building dependency tree...
       The following NEW packages will be installed:
         libxi6
       0 upgraded, 1 newly installed, 0 to remove and 5 not upgraded.
       Need to get 0 B/29.9 kB of archives.
       After this operation, 94.2 kB of additional disk space will be used.
       Download complete and in download only mode
W: --force-yes is deprecated, use one of the options starting with --allow instead.
       Fetching .debs for libxinerama1
       Reading package lists...
       Building dependency tree...
       The following NEW packages will be installed:
         libxinerama1
       0 upgraded, 1 newly installed, 0 to remove and 5 not upgraded.
       Need to get 0 B/6,904 B of archives.
       After this operation, 33.8 kB of additional disk space will be used.
       Download complete and in download only mode
W: --force-yes is deprecated, use one of the options starting with --allow instead.
       Fetching .debs for libxrandr2
       Reading package lists...
       Building dependency tree...
       The following NEW packages will be installed:
         libxrandr2
       0 upgraded, 1 newly installed, 0 to remove and 5 not upgraded.
       Need to get 0 B/18.5 kB of archives.
       After this operation, 70.7 kB of additional disk space will be used.
       Download complete and in download only mode
W: --force-yes is deprecated, use one of the options starting with --allow instead.
       Fetching .debs for libxshmfence1
       Reading package lists...
       Building dependency tree...
       The following NEW packages will be installed:
         libxshmfence1
       0 upgraded, 1 newly installed, 0 to remove and 5 not upgraded.
       Need to get 5,028 B of archives.
       After this operation, 22.5 kB of additional disk space will be used.
       Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libxshmfence1 amd64 1.3-1 [5,028 B]
       Fetched 5,028 B in 0s (31.0 kB/s)
       Download complete and in download only mode
W: --force-yes is deprecated, use one of the options starting with --allow instead.
       Fetching .debs for libxss1
       Reading package lists...
       Building dependency tree...
       The following NEW packages will be installed:
         libxss1
       0 upgraded, 1 newly installed, 0 to remove and 5 not upgraded.
       Need to get 8,140 B of archives.
       After this operation, 32.8 kB of additional disk space will be used.
       Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 libxss1 amd64 1:1.2.3-1 [8,140 B]
       Fetched 8,140 B in 0s (51.0 kB/s)
       Download complete and in download only mode
W: --force-yes is deprecated, use one of the options starting with --allow instead.
       Fetching .debs for libxtst6
       Reading package lists...
       Building dependency tree...
       The following NEW packages will be installed:
         libxtst6
       0 upgraded, 1 newly installed, 0 to remove and 5 not upgraded.
       Need to get 0 B/12.8 kB of archives.
       After this operation, 45.1 kB of additional disk space will be used.
       Download complete and in download only mode
W: --force-yes is deprecated, use one of the options starting with --allow instead.
       Fetching .debs for fonts-liberation
       Reading package lists...
       Building dependency tree...
       The following NEW packages will be installed:
         fonts-liberation
       0 upgraded, 1 newly installed, 0 to remove and 5 not upgraded.
       Need to get 822 kB of archives.
       After this operation, 2,139 kB of additional disk space will be used.
       Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 fonts-liberation all 1:1.07.4-11 [822 kB]
       Fetched 822 kB in 0s (5,804 kB/s)
       Download complete and in download only mode
W: --force-yes is deprecated, use one of the options starting with --allow instead.
       Installing adwaita-icon-theme_3.36.1-2ubuntu0.20.04.2_all.deb
       Installing alsa-topology-conf_1.2.2-1_all.deb
       Installing alsa-ucm-conf_1.2.2-1ubuntu0.13_all.deb
       Installing at-spi2-core_2.36.0-2_amd64.deb
       Installing dbus_1.12.16-2ubuntu2.3_amd64.deb
       Installing dbus-user-session_1.12.16-2ubuntu2.3_amd64.deb
       Installing dconf-gsettings-backend_0.36.0-1_amd64.deb
       Installing dconf-service_0.36.0-1_amd64.deb
       Installing dmsetup_2%3a1.02.167-1ubuntu1_amd64.deb
       Installing fonts-liberation_1%3a1.07.4-11_all.deb
       Installing gconf2-common_3.2.6-6ubuntu1_all.deb
       Installing gconf-service_3.2.6-6ubuntu1_amd64.deb
       Installing gconf-service-backend_3.2.6-6ubuntu1_amd64.deb
       Installing glib-networking_2.64.2-1ubuntu0.1_amd64.deb
       Installing glib-networking-common_2.64.2-1ubuntu0.1_all.deb
       Installing glib-networking-services_2.64.2-1ubuntu0.1_amd64.deb
       Installing gsettings-desktop-schemas_3.36.0-1ubuntu1_all.deb
       Installing gtk-update-icon-cache_3.24.20-0ubuntu1.1_amd64.deb
       Installing humanity-icon-theme_0.6.15_all.deb
       Installing libapparmor1_2.13.3-7ubuntu5.3_amd64.deb
       Installing libappindicator1_12.10.1+20.04.20200408.1-0ubuntu1_amd64.deb
       Installing libasound2_1.2.2-2.1ubuntu2.5_amd64.deb
       Installing libasound2-data_1.2.2-2.1ubuntu2.5_all.deb
       Installing libatk1.0-0_2.35.1-1ubuntu2_amd64.deb
       Installing libatk1.0-data_2.35.1-1ubuntu2_all.deb
       Installing libatk-bridge2.0-0_2.34.2-0ubuntu2~20.04.1_amd64.deb
       Installing libatspi2.0-0_2.36.0-2_amd64.deb
       Installing libcairo-gobject2_1.16.0-4ubuntu1_amd64.deb
       Installing libcolord2_1.4.4-2_amd64.deb
       Installing libcryptsetup12_2%3a2.2.2-3ubuntu2.4_amd64.deb
       Installing libdbus-glib-1-2_0.110-5fakssync1_amd64.deb
       Installing libdbusmenu-glib4_16.04.1+18.10.20180917-0ubuntu6_amd64.deb
       Installing libdbusmenu-gtk4_16.04.1+18.10.20180917-0ubuntu6_amd64.deb
       Installing libdconf1_0.36.0-1_amd64.deb
       Installing libdevmapper1.02.1_2%3a1.02.167-1ubuntu1_amd64.deb
       Installing libdrm2_2.4.107-8ubuntu1~20.04.2_amd64.deb
       Installing libdrm-common_2.4.107-8ubuntu1~20.04.2_all.deb
       Installing libepoxy0_1.5.4-1_amd64.deb
       Installing libgail18_2.24.32-4ubuntu4_amd64.deb
       Installing libgail-common_2.24.32-4ubuntu4_amd64.deb
       Installing libgbm1_21.2.6-0ubuntu0.1~20.04.2_amd64.deb
       Installing libgconf-2-4_3.2.6-6ubuntu1_amd64.deb
       Installing libgtk2.0-0_2.24.32-4ubuntu4_amd64.deb
       Installing libgtk2.0-bin_2.24.32-4ubuntu4_amd64.deb
       Installing libgtk2.0-common_2.24.32-4ubuntu4_all.deb
       Installing libgtk-3-0_3.24.20-0ubuntu1.1_amd64.deb
       Installing libgtk-3-bin_3.24.20-0ubuntu1.1_amd64.deb
       Installing libgtk-3-common_3.24.20-0ubuntu1.1_all.deb
       Installing libip4tc2_1.8.4-3ubuntu2.1_amd64.deb
       Installing libjson-glib-1.0-0_1.4.4-2ubuntu2_amd64.deb
       Installing libjson-glib-1.0-common_1.4.4-2ubuntu2_all.deb
       Installing libnspr4_2%3a4.35-0ubuntu0.20.04.1_amd64.deb
       Installing libnss3_2%3a3.98-0ubuntu0.20.04.2_amd64.deb
       Installing libnss-systemd_245.4-4ubuntu3.23_amd64.deb
       Installing libpam-systemd_245.4-4ubuntu3.23_amd64.deb
       Installing libproxy1v5_0.4.15-10ubuntu1.2_amd64.deb
       Installing librest-0.7-0_0.8.1-1_amd64.deb
       Installing libsoup2.4-1_2.70.0-1_amd64.deb
       Installing libsoup-gnome2.4-1_2.70.0-1_amd64.deb
       Installing libwayland-client0_1.18.0-1ubuntu0.1_amd64.deb
       Installing libwayland-cursor0_1.18.0-1ubuntu0.1_amd64.deb
       Installing libwayland-egl1_1.18.0-1ubuntu0.1_amd64.deb
       Installing libwayland-server0_1.18.0-1ubuntu0.1_amd64.deb
       Installing libx11-xcb1_2%3a1.6.9-2ubuntu1.6_amd64.deb
       Installing libxcb-dri3-0_1.14-2_amd64.deb
       Installing libxcomposite1_1%3a0.4.5-1_amd64.deb
       Installing libxcursor1_1%3a1.2.0-2_amd64.deb
       Installing libxdamage1_1%3a1.1.5-2_amd64.deb
       Installing libxfixes3_1%3a5.0.3-2_amd64.deb
       Installing libxi6_2%3a1.7.10-0ubuntu1_amd64.deb
       Installing libxinerama1_2%3a1.1.4-2_amd64.deb
       Installing libxkbcommon0_0.10.0-1_amd64.deb
       Installing libxrandr2_2%3a1.5.2-0ubuntu1_amd64.deb
       Installing libxshmfence1_1.3-1_amd64.deb
       Installing libxss1_1%3a1.2.3-1_amd64.deb
       Installing libxtst6_2%3a1.2.3-1_amd64.deb
       Installing networkd-dispatcher_2.1-2~ubuntu20.04.3_all.deb
       Installing python3-dbus_1.2.16-1build1_amd64.deb
       Installing python3-gi_3.36.0-1_amd64.deb
       Installing systemd_245.4-4ubuntu3.23_amd64.deb
       Installing systemd-sysv_245.4-4ubuntu3.23_amd64.deb
       Installing systemd-timesyncd_245.4-4ubuntu3.23_amd64.deb
       Installing ubuntu-mono_19.04-0ubuntu3_all.deb
       Installing xkb-data_2.29-2_all.deb
       Writing profile script
       Installed Chrome dependencies for heroku-20
       Adding executables to PATH
       /tmp/build_23d4355b/.chrome-for-testing/chrome-linux64/chrome
       /tmp/build_23d4355b/.chrome-for-testing/chromedriver-linux64/chromedriver
       Installed Chrome for Testing STABLE version 125.0.6422.141

[edmorley]

Thank you for the build logs. With them + a process of elimination I was able to work out that this issue is caused by the jemalloc buildpack you are using.

Once I added it + found out about the JEMALLOC_ENABLED env var (that causes it to be enabled for all dynos), I was able to reproduce.

ie:

$ mkdir chrome-test && cd $_
$ git init && touch foo && git add -A && git commit -m '.'
$ heroku create --stack heroku-20
$ heroku buildpacks:set --index 1 https://github.com/gaffneyc/heroku-buildpack-jemalloc.git
$ heroku buildpacks:set --index 2 heroku-community/chrome-for-testing
$ heroku config:set JEMALLOC_ENABLED=true
$ git push heroku main
$ heroku run bash
Running bash on ⬢ boiling-forest-07615... up, run.5957 (Basic)
~ $ chromedriver --headless
Starting ChromeDriver 125.0.6422.141 (4b1e83937122185343ba92e909b021f307c719ca-refs/branch-heads/6422@{#1186}) on port 9515
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
Segmentation fault

Note how if I unset LD_PRELOAD (which is what the jemalloc buildpack's .profile.d/ script sets iff JEMALLOC_ENABLED is "true") then everything works again (without even having to do a new build):

~ $ unset LD_PRELOAD 
~ $ chromedriver --headless
Starting ChromeDriver 125.0.6422.141 (4b1e83937122185343ba92e909b021f307c719ca-refs/branch-heads/6422@{#1186}) on port 9515
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
ChromeDriver was started successfully.

It seems that the Chromedriver that Google provides in the Chrome for Testing bundle doesn't interact well with jemalloc - or at the least, doesn't interact well with either this version of jemalloc or the way that the third party buildpack configures it. I'm not familiar enough with Chromedriver or jemalloc to say where the issue might lie without doing more research - but I'm pretty sure this isn't due to anything we're doing in the heroku-buildpack-chrome-for-testing buildpack itself.

One option might be to only enable jemalloc on a per dyno basis, per:
https://github.com/gaffneyc/heroku-buildpack-jemalloc?tab=readme-ov-file#usage

@mars Any other ideas? :-)

[guiferrpereira]

It will work with the old JEMALLOC version 3.6.0. thanks for your help @edmorley

[edmorley]

Ah great - good to know changing the version helped! Happy to help :-)

[lucafrollo]

Hi
We are currently unable to upgrade to heroku-24 because of the issues between this buildpack and jmalloc.
I commented on the jmalloc buildpack issue page but I am a little stuck at this stage, any idea on possible workarounds?
Thanks.

gaffneyc/heroku-buildpack-jemalloc#36 (comment)