Crystal 1.1.0 Segmentation fault on stripped binary

[samwalt]

OS Windows 10 21H1
WSL Ubuntu 18.04.5 LTS

I used sudo apt upgrade to upgrade crystal version from 1.0.0 to 1.1.0.
After it finished, I checked crystal version using crystal -v. It showed
error "Segmentation fault (core dumped)"

[straight-shoota]

This seems to be a problem in the packages for some older Ubuntu releases. As mentioned in #10949 (comment) you can install 1.1.0 from the repository for a newer Ubuntu release as a workaround.

More context on the sefgault:

$ strace /usr/lib/crystal/bin/crystal
execve("/usr/lib/crystal/bin/crystal", ["/usr/lib/crystal/bin/crystal"], 0x7ffeda162230 /* 8 vars */) = -1 EPERM (Operation not permitted)
--- SIGSEGV {si_signo=SIGSEGV, si_code=SI_KERNEL, si_addr=NULL} ---
+++ killed by SIGSEGV +++
Segmentation fault

The possible reasons for EPERM all don't seem super convincing for this 🤔

[straight-shoota]

We had already noticed similar segfaults in the packages for some RPM repositories (for example
https://build.opensuse.org/package/live_build_log/devel:languages:crystal/crystal/RHEL_7/x86_64). Only RHEL and derivates (CentOS, Scientific Linux) are affected. Those package builds failed, so on these platforms, the repository still contains Crystal 1.0.0 as latest release. Workaround is similar to the DEB packages (#10949 (comment)): Just install the packages from a different RPM distribution like Fedora. It should work fine. The package build scripts are pretty much identical DEB and RPM are different, of course, but all RPM repositories and all DEB repositories use the same config.

[straight-shoota]

I managed to execute the binary from the Ubuntu 18.04 package as superuser, but not as normal user. This worked in a in Ubuntu 20.04 VM, but in the ubuntu:20.04 docker image however, it does not even run as superuser.

File system is not mounted with nosuid in both cases.

[straight-shoota]

The most likely EPERM explanation is this:

A "capability-dumb" applications would not obtain the full set of permitted capabilities granted by the executable file.

I'm not very familiar with Linux capabilities. But as I understand it, a "capability-dump" application is when the binary file has capabilities set, but is itself not aware of capabilities.

When executing a capability-dumb binary, the kernel checks if the process obtained all permitted capabilities that were specified in the file permitted set, after the capability transformations described above have been performed. (The typical reason why this might not occur is that the capability bounding set masked out some of the capabilities in the file permitted set.) If the process did not obtain the full set of file permitted capabilities, then execve(2) fails with the error EPERM. This prevents possible security risks that could arise when a capability-dumb application is executed with less privilege that it needs. Note that, by definition, the application could not itself recognize this problem, since it does not employ the libcap(3) API.

But getcap /usr/lib/crystal/bin/crystal is empty, so no capabilities would be permitted (as expected, the compiler doesn't need any).

Btw. the same error happens to the shards executable as well.

[straight-shoota]

For reproducing, running this command on any APT-based system should give you a non-working crystal executable:

$ wget -O- https://crystal-lang.org/installer.sh | sudo DISTRO_REPO=xUbuntu_18.04 bash
$ crystal --version
Segmentation fault
Segmentation fault

(The reason for double segfault is that crystal is a wrapper script which calls the actual compiler binary /usr/lib/crystal/bin/crystal twice.)

[codybswaney]

I think what you have here is a problem with strace privileges / ptrace_scope in both docker and WSL.

See, running from docker without the --privileged flag doesn't set the right PTRACE cap.

# if you run this, you'll get a segfault at the end
docker run -it ubuntu:18.04 bash -c 'apt-get -y update && apt-get install -y wget strace && wget -O- https://crystal-lang.org/install.sh | DISTRO_REPO=xUbuntu_18.04 bash && crystal -v'

# but when I run this, it works
docker run -it --privileged ubuntu:18.04 bash -c 'apt-get -y update && apt-get install -y wget strace && wget -O- https://crystal-lang.org/install.sh | DISTRO_REPO=xUbuntu_18.04 bash && crystal -v'

For docker, the solution is simple. For WSL, I'm not entirely sure. Normally you'd modify the ptrace_scope manually.

EDIT:
To be clear, I'm assuming PTRACE might be the culprit with no real research but SELinux or AppArmor or something might also be suspects.

[straight-shoota]

Thanks @codybswaney. It might not have been explicitly stated, but the problem is not specific to containers or WSL. It reproduces on native systems.

I have been able to dig something up.

It seems file attributes are actually not relevant here. Copying the file contents to a different file retains the error, copying the contents of a working executable (from the release tarball) into the packaged file gets rid of the error. So apparently the content of the binary is at fault.

The executable from the tarball and from the debian package are different. So I investigated what happens in the DEB packaging process:

[   71s] 	strip --remove-section=.comment --remove-section=.note debian/crystal/usr/lib/crystal/bin/shards
[   72s] 	strip --remove-section=.comment --remove-section=.note debian/crystal/usr/lib/crystal/bin/crystal
[   72s] 	strip --strip-debug --remove-section=.comment --remove-section=.note --enable-deterministic-archives debian/crystal/usr/lib/crystal/lib/libgc.a

We can skip the stripping and yes, that makes the binary work again. It looks like strip is breaking the binary in Ubuntu 18.04.

Applying the same strip command on Ubuntu 20.04 works flawlessly. The stripped binary has exactly the same size, but different content.

On Ubuntu 18.04 strip is from binutils 2.30, on 20.04 it's 2.34.

[straight-shoota]

A simple workaround to get all packages going is to disable binary stripping in the package build files. That seems to fix the same problem in RPM packages as well (all green on https://build.opensuse.org/package/show/home:straight-shoota:branches:devel:languages:crystal/crystal).

Still, it is unclear what exactly causes this. The strip behaviour on Ubuntu 18.04 did not change between Crystal 1.0 and 1.1. So it looks like this was introduced by a change in Crystal between 1.0 and 1.1. Stripping both compiler versions on Ubuntu 18.04, stripped 1.0 works but stripped 1.1 segfaults. It might also be related to LLVM which changed from 10.0 to 10.1. This needs further investigation to figure out, why stripping would break a binary. This might not just affect the compiler, but any Crystal program.

[straight-shoota]

Stripping the binary on Ubuntu 18.04 and Ubuntu 20.04 results in a single byte difference in the program header:

Program Header:
-    LOAD off    0x0000000000000000 vaddr 0x0000000000000000 paddr 0x0000000000400000 align 2**12
+    LOAD off    0x0000000000000000 vaddr 0x0000000000400000 paddr 0x0000000000400000 align 2**12

diff <(hexdump crystal-stripped-18) <(hexdump crystal-stripped-20)

--- /dev/fd/63  2021-07-16 13:13:29.160000000 +0200
+++ /dev/fd/62  2021-07-16 13:13:29.160000000 +0200
@@ -3,7 +3,7 @@
 0000020 0040 0000 0000 0000 d990 0286 0000 0000
 0000030 0000 0000 0040 0038 0007 0040 0010 000f
 0000040 0001 0000 0004 0000 0000 0000 0000 0000
-0000050 0000 0000 0000 0000 0000 0040 0000 0000
+0000050 0000 0040 0000 0000 0000 0040 0000 0000
 0000060 01c8 0000 0000 0000 01c8 0000 0000 0000
 0000070 1000 0000 0000 0000 0001 0000 0005 0000
 0000080 1000 0000 0000 0000 1000 0040 0000 0000

This difference does not appear with the 1.0 binary which strips identically on both Ubuntu 18.04 and 20.04.
It seems the program header of the 1.0 binary is quite different from the 1.1 binary.