Several upstream bottles cause segfault. related to old `patchelf.rb` bugs.

[redspot]

brew gist-logs <formula> link OR brew config AND brew doctor output

$ brew config
HOMEBREW_VERSION: 4.2.9-88-g94e987e
ORIGIN: https://github.com/Homebrew/brew
HEAD: 94e987ee65fa6ae564f86d37fe21b5ad81a258d4
Last commit: 26 minutes ago
Core tap HEAD: 3e47b3bac60e610ea8fcdfcf9693c25812ef1727
Core tap last commit: 3 months ago
Core tap JSON: 22 Feb 21:25 UTC
HOMEBREW_PREFIX: /home/linuxbrew/.linuxbrew
HOMEBREW_CASK_OPTS: []
HOMEBREW_DISPLAY: :0
HOMEBREW_GITHUB_API_TOKEN: set
HOMEBREW_MAKE_JOBS: 8
HOMEBREW_SORBET_RUNTIME: set
SUDO_ASKPASS: /home/wmartin45/bin/askpass-kdewallet
Homebrew Ruby: 3.1.4 => /home/linuxbrew/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/3.1.4/bin/ruby
CPU: octa-core 64-bit skylake
Clang: 17.0.6
Git: 2.39.3 => /bin/git
Curl: 7.61.1 => /bin/curl
Kernel: Linux 4.18.0-513.5.1.el8_9.x86_64 x86_64 GNU/Linux
OS: Red Hat Enterprise Linux release 8.9 (Ootpa)
Host glibc: 2.28
/usr/bin/gcc: 8.5.0
/usr/bin/ruby: N/A
glibc: 2.35_1
gcc@11: N/A
gcc: 13.2.0
xorg: N/A

$ brew doctor
Your system is ready to brew.
$ brew update --auto-update  # no output

Verification

• My "brew doctor output" says Your system is ready to brew. and am still able to reproduce my issue.
• I ran brew update and am still able to reproduce my issue.
• I have resolved all warnings from brew doctor and that did not fix my problem.
• I searched for recent similar issues at https://github.com/Homebrew/homebrew-core/issues?q=is%3Aissue and found no duplicates.

What were you trying to do (and why)?

I was attempting to use several formulae installed via brew.

• ssdeep
• go
• pkg-config

There are several related issues:
#137991 related to old patchelf.rb bug
#132852 bottles crashing on Centos
#116841 bottles crashing on RHEL

What happened (include all command output)?

$ brew reinstall --force-bottle ssdeep
==> Downloading https://ghcr.io/v2/homebrew/core/ssdeep/manifests/2.14.1
Already downloaded: /home/wmartin45/.cache/Homebrew/downloads/5d5c962b71849c67a5e3e6c0df39b82fd0b823ac868d942f93e6a1bec0a6a03b--ssdeep-2.14.1.bottle_manifest.json
==> Fetching ssdeep
==> Downloading https://ghcr.io/v2/homebrew/core/ssdeep/blobs/sha256:debece05c49ee73f650afa080d3f0953ac64449c79169525001e871d0888edef
Already downloaded: /home/wmartin45/.cache/Homebrew/downloads/89333f5f972034d0bbcfbdab464d33f04ced2b2f6e86f532cf6e6c95484a2b7e--ssdeep--2.14.1.x86_64_linux.bottle.tar.gz
==> Reinstalling ssdeep
==> Pouring ssdeep--2.14.1.x86_64_linux.bottle.tar.gz
🍺  /home/linuxbrew/.linuxbrew/Cellar/ssdeep/2.14.1: 16 files, 194.9KB
==> Running `brew cleanup ssdeep`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
$ /home/linuxbrew/.linuxbrew/bin/ssdeep -V
Segmentation fault
$ dmesg | grep ssdeep
[4717362.289573] 3149001 (ssdeep): Uhuuh, elf segment at 0000000000401000 requested but the memory is mapped already

Note the segment that causes the problem, 0x0000000000401000

Now, take a look at the on-disk segments:

$ readelf -l /home/linuxbrew/.linuxbrew/bin/ssdeep
Elf file type is EXEC (Executable file)
Entry point 0x4028e0
There are 11 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  PHDR           0x0000000000000040 0x00000000003ff040 0x00000000003ff040
                 0x0000000000000268 0x0000000000000268  R E    0x8
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RW     0x10
  LOAD           0x0000000000000000 0x00000000003ff000 0x00000000003ff000
                 0x00000000000022ae 0x00000000000022ae  RW     0x1000
  INTERP         0x0000000000001308 0x0000000000400308 0x0000000000400308
                 0x0000000000000025 0x0000000000000025  R      0x1
      [Requesting program interpreter: /home/linuxbrew/.linuxbrew/lib/ld.so]
  NOTE           0x0000000000001330 0x0000000000400330 0x0000000000400330
                 0x0000000000000020 0x0000000000000020  R      0x4
  NOTE           0x0000000000001350 0x0000000000400350 0x0000000000400350
                 0x0000000000000024 0x0000000000000024  R      0x4
  LOAD           0x00000000000022ae 0x00000000004012ae 0x00000000004012ae
                 0x0000000000009917 0x0000000000009917  R E    0x1000
  GNU_EH_FRAME   0x000000000000a580 0x0000000000409580 0x0000000000409580
                 0x00000000000002cc 0x00000000000002cc  R      0x4
  LOAD           0x000000000000bda0 0x000000000060ada0 0x000000000060ada0
                 0x00000000000004f4 0x00000000000005a8  RW     0x1000
  GNU_RELRO      0x000000000000bda0 0x000000000060ada0 0x000000000060ada0
                 0x0000000000000260 0x0000000000000260  R      0x1
  DYNAMIC        0x000000000000bdf8 0x000000000060adf8 0x000000000060adf8
                 0x0000000000000200 0x0000000000000200  RW     0x8

$ readelf -l /home/linuxbrew/.linuxbrew/bin/ssdeep | grep '0x0000000000401... '
  LOAD           0x00000000000022ae 0x00000000004012ae 0x00000000004012ae

Note the segment 0x00000000004012ae

Also, of interest, when run using brew's interpreter, programs seem to work:

$ /home/linuxbrew/.linuxbrew/lib/ld.so /home/linuxbrew/.linuxbrew/bin/ssdeep -V
2.14.1

Here is the results when built from source:

$ brew install --build-from-source ssdeep
==> Fetching ssdeep
==> Downloading https://raw.githubusercontent.com/Homebrew/homebrew-core/11dc5d3906c536d30245ec1bace1b73338857c9b/Formula/s/ssdeep.rb
######################################################################################################################################### 100.0%
==> Downloading https://github.com/ssdeep-project/ssdeep/releases/download/release-2.14.1/ssdeep-2.14.1.tar.gz
==> Downloading from https://objects.githubusercontent.com/github-production-release-asset-2e65be/93355216/ae5313b0-c3b0-11e7-88a8-251ebd913c9d?
######################################################################################################################################### 100.0%
==> ./configure
==> make install
🍺  /home/linuxbrew/.linuxbrew/Cellar/ssdeep/2.14.1: 16 files, 201.6KB, built in 9 seconds
==> Running `brew cleanup ssdeep`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
$ readelf -l /home/linuxbrew/.linuxbrew/bin/ssdeep | grep '0x0000000000401... '  # no matches
$ /home/linuxbrew/.linuxbrew/bin/ssdeep -V
2.14.1

What did you expect to happen?

I expect the upstream pre-built bottles to not have lingering issues with patchelf.rb.

Now, the issues with patchelf.rb have been fixed, and maybe those Centos/RHEL related issues as well.

It's that the bottles still have the problem.

Step-by-step reproduction instructions (by running brew commands)

As mentioned in this comment: #132852 (comment)

Some formulas work:

$ /home/linuxbrew/.linuxbrew/bin/xz --version
xz (XZ Utils) 5.4.5
liblzma 5.4.5

Some formulas fail:

$ /home/linuxbrew/.linuxbrew/opt/pkg-config/bin/pkg-config --version
Segmentation fault
$ dmesg | grep pkg-config
[4719592.428429] 3158801 (pkg-config): Uhuuh, elf segment at 0000000000401000 requested but the memory is mapped already
$ readelf -l /home/linuxbrew/.linuxbrew/opt/pkg-config/bin/pkg-config | grep '0x0000000000401... '
  LOAD           0x0000000000006dd2 0x0000000000401dd2 0x0000000000401dd2
$ /home/linuxbrew/.linuxbrew/lib/ld.so /home/linuxbrew/.linuxbrew/opt/pkg-config/bin/pkg-config --version
0.29.2
$ readlink -f /home/linuxbrew/.linuxbrew/lib/ld.so
/home/linuxbrew/.linuxbrew/Cellar/glibc/2.35_1/lib/ld-linux-x86-64.so.2

Note the kernel reporting the same segment, 0x0000000000401000, and readelf showing that segment. And, it strangely works when run with brew's interpreter.

when building from source:

$ brew reinstall --build-from-source pkg-config
Warning: building from source is not supported!
You're on your own. Failures are expected so don't create any issues, please!
==> Fetching pkg-config
==> Downloading https://raw.githubusercontent.com/Homebrew/homebrew-core/11dc5d3906c536d30245ec1bace1b73338857c9b/Formula/p/pkg-config.rb
######################################################################################################################################### 100.0%
==> Downloading https://pkgconfig.freedesktop.org/releases/pkg-config-0.29.2.tar.gz
######################################################################################################################################### 100.0%
==> Reinstalling pkg-config
==> ./configure --disable-host-tool --with-internal-glib --with-pc-path=/home/linuxbrew/.linuxbrew/lib/pkgconfig:/home/linuxbrew/.linuxbrew/shar
==> make
==> make install
🍺  /home/linuxbrew/.linuxbrew/Cellar/pkg-config/0.29.2_3: 11 files, 750.6KB, built in 40 seconds
==> Running `brew cleanup pkg-config`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
$ readelf -l /home/linuxbrew/.linuxbrew/opt/pkg-config/bin/pkg-config | grep '0x0000000000401... '  # no matches
$ /home/linuxbrew/.linuxbrew/opt/pkg-config/bin/pkg-config --version
0.29.2

It also seems reliable to detect the bad packages by looking for a segment that matches the regex 0x0000000000401....

[redspot]

I don't know why all of the Markdown blocks got flipped around.

Here are the related issues and comment:
#137991 related to old patchelf.rb bug
#132852 bottles crashing on Centos
#116841 bottles crashing on RHEL

#132852 (comment)

[redspot]

This issue seems related: #136432

[redspot]

I think the root cause related to many homebrew Linux segmentation faults is the homebrew build bot.

The build bot making the bottles is still somehow using an older patchelf.rb which causes elf binaries to be generated with improper segments causing memory map collisions.

[iMichka]

Latest patchelf updates in brew:
Homebrew/brew#16694
Homebrew/brew#14094

Most issues have been fixed with version 1.4.0 in Nov. 2022.

Why do you think it's still using an old patchelf.rb version?

In your config:

Core tap last commit: 3 months ago

This looks old. Can you update first?

[redspot]

Latest patchelf updates in brew: Homebrew/brew#16694 Homebrew/brew#14094

Most issues have been fixed with version 1.4.0 in Nov. 2022.

Why do you think it's still using an old patchelf.rb version?

It's not an issue with patchelf.rb. It's an issue with the pre built bottles.

See above the part with brew reinstall --force-bottle ssdeep which segfaults, but works fine when built from source.

In your config:

Core tap last commit: 3 months ago

This looks old. Can you update first?

How do I do that?

I ran brew update --auto-update with no output given.

[gromgit]

How do I do that?

HOMEBREW_NO_INSTALL_FROM_API=1 brew update should do the necessary.

[iMichka]

Don't worry about the 3 months old update message: you are now using the API to get the latest version, so I guess no need to update the git repo on your computer to get the latest version. I think that message is confusing, I'll have a look into it later.

I think I know what happened.

patchelf was updated in Nov 2022, but I did not realise patchelf was only updated 1 week ago in brew ...

We will have to rebuild these bottles.

I have triggered a ssdeep rebuild: https://github.com/Homebrew/homebrew-core/actions/runs/8060846410

Once this is done, you can brew update, and the brew reinstall ssdepp and try the new bottle.

[redspot]

Sounds great. There are several other bottles with crashes, such as golang.

[pdelre]

👋 Been quietly watching this and previous issues.

In WSL Ubuntu this is pretty wide spread from packages built with go and those that are built with gcc. I believe I saw less issues when I tested in a Ubuntu VM.

What could be an effective way forward for the project maintainers? I imagine as the underlying packages upgrade, they'll be rebuilt, so as users, should we just wait? Would the repo owners find a list of affected packages be helpful? I'd be happy to send along the packages I've had issues with and help UAT rebuilds.

[iMichka]

@redspot a new ssdeep bottle has been added. Can you brew update && brew reinstall ssdeppto confirm that this fixes it for you.

I imagine as the underlying packages upgrade, they'll be rebuilt, so as users, should we just wait? Would the repo owners find a list of affected packages be helpful?

Yes, if you have a list, we can at least rebuild all of these. Most packages get updates / rebuilds at least once per year, but some others will take time to get update. Let's fix what is known to be broken.

[pdelre]

Here's a list from my bash/zsh history, though I think I had an issue with every package that uses go as a build dependency.

• act
• docker
• docker-compose
• findutils
• fzf
• gh
• go
• go-task
• gron
• k6
• k9s
• noti
• pandoc
• unzip
• whalebrew
• yq

[redspot]

see this comment for more potentially broken bottles #132852 (comment)

I definitely had issues with these formulas:

• ssdeep
• xz
• pkg-config

but, there could have been others. It's not that I would like you to fix a specific formula. It's that you most likely have many broken bottles that need a rebuild.

I guess you could come up with a way to find bottles that are:

• Linux x86_64
• has an ELF64 file with:
• a memory segment that matches '0x00000000004..... ', maybe check with readelf -l

[iMichka]

Can just someone confirm that brew update && brew reinstall ssdepp fixes it before I start rebottling stuff?

Note: we probably won't rebottle everything, I'll go through the lists above, for anything else users will have to open an issue or post here.