bazel/libgeos: segfault on s390x under cross-build

[golgeek]

Describe the problem

When running (current master branch) ./cockroach demo with the bazel built (cross-built) libgeos.so and libgeos_c.so libs on s390x, the binary crashes when trying to init the CGO code.

SIGSEGV: segmentation violation
PC=0x3ffae98afea m=5 sigcode=1 addr=0x3ffce3b4000
signal arrived during cgo execution

goroutine 1 gp=0xc0000021c0 m=5 mp=0xc000180008 [syscall]:
runtime.cgocall(0x77cb120, 0xc0031e9290)
	GOROOT/src/runtime/cgocall.go:167 +0x6a fp=0xc0031e9260 sp=0xc0031e9230 pc=0x110d00a
github.com/cockroachdb/cockroach/pkg/geo/geos._Cfunc_CR_GEOS_Init({0xc0025e1608, 0x11}, {0xc001c91620, 0xf}, 0xc001c915d8)
	_cgo_gotypes.go:408 +0x4c fp=0xc0031e9288 sp=0xc0031e9260 pc=0x3139e9c
github.com/cockroachdb/cockroach/pkg/geo/geos.initGEOS.func1({0xc001c9137d, 0x3}, 0xc001c915d8)
	com_github_cockroachdb_cockroach/pkg/geo/geos/geos.go:200 +0x27a fp=0xc0031e93a8 sp=0xc0031e9288 pc=0x313d95a
github.com/cockroachdb/cockroach/pkg/geo/geos.initGEOS({0xc001483140, 0x3, 0x4})
	com_github_cockroachdb_cockroach/pkg/geo/geos/geos.go:200 +0xd0 fp=0xc0031e9480 sp=0xc0031e93a8 pc=0x313d3a0
github.com/cockroachdb/cockroach/pkg/geo/geos.ensureInit.func1()
	com_github_cockroachdb_cockroach/pkg/geo/geos/geos.go:91 +0x60 fp=0xc0031e94c8 sp=0xc0031e9480 pc=0x313c5f0
sync.(*Once).doSlow(0xe44baa8, 0xc0031e9568)

Full stacktrace:
cockroach.stderr.log

Valgrind:
valgrind.cockroach.stderr.log

--

I tried building the same version of libgeos locally, and didn't face the issue after replacing the bazel built libs:

ubuntu@teamcity-19444336-1744830048-01-n9cpu4-geo-0001:~/lib$ readelf -h libgeos.so.bazel libgeos.so.local

File: libgeos.so.bazel
ELF Header:
  Magic:   7f 45 4c 46 02 02 01 03 00 00 00 00 00 00 00 00
  Class:                             ELF64
  Data:                              2's complement, big endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - GNU
  ABI Version:                       0
  Type:                              DYN (Shared object file)
  Machine:                           IBM S/390
  Version:                           0x1
  Entry point address:               0x19c7f0
  Start of program headers:          64 (bytes into file)
  Start of section headers:          6722936 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         8
  Size of section headers:           64 (bytes)
  Number of section headers:         37
  Section header string table index: 35

File: libgeos.so.local
ELF Header:
  Magic:   7f 45 4c 46 02 02 01 03 00 00 00 00 00 00 00 00
  Class:                             ELF64
  Data:                              2's complement, big endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - GNU
  ABI Version:                       0
  Type:                              DYN (Shared object file)
  Machine:                           IBM S/390
  Version:                           0x1
  Entry point address:               0x0
  Start of program headers:          64 (bytes into file)
  Start of section headers:          3545688 (bytes into file)
  Flags:                             0x0
  Size of this header:               64 (bytes)
  Size of program headers:           56 (bytes)
  Number of program headers:         7
  Size of section headers:           64 (bytes)
  Number of section headers:         29
  Section header string table index: 28

ubuntu@teamcity-19444336-1744830048-01-n9cpu4-geo-0001:~$ ll lib/
total 11716
drwxrwx---  2 ubuntu ubuntu    4096 Apr 16 20:49 ./
drwxr-x--- 12 ubuntu ubuntu    4096 Apr 16 20:51 ../
lrwxrwxrwx  1 ubuntu ubuntu      18 Apr 16 20:49 libgeos_c.so -> libgeos_c.so.local*
-rwxrwx---  1 ubuntu ubuntu  593360 Apr 16 19:03 libgeos_c.so.bazel*
-rwxrwx---  1 ubuntu ubuntu  360672 Apr 16 20:22 libgeos_c.so.local*
lrwxrwxrwx  1 ubuntu ubuntu      16 Apr 16 20:49 libgeos.so -> libgeos.so.local*
-rwxrwx---  1 ubuntu ubuntu 7479056 Apr 16 19:03 libgeos.so.bazel*
-rwxrwx---  1 ubuntu ubuntu 3547544 Apr 16 20:21 libgeos.so.local*

ubuntu@teamcity-19444336-1744830048-01-n9cpu4-geo-0001:~$ ./cockroach demo
<jemalloc>: No THP support: thp:never
#
# Welcome to the CockroachDB demo database!
[...]
#
demo@127.0.0.1:26257/demoapp/movr> SELECT ST_IsValid(ST_MakePoint(1,2));
  st_isvalid
--------------
      t
(1 row)

Time: 1ms total (execution 1ms / network 0ms)

demo@127.0.0.1:26257/demoapp/movr>

Jira issue: CRDB-49436

[blathers-crl]

Hi @golgeek, please add branch-* labels to identify which branch(es) this C-bug affects.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[golgeek]

Investigation progress update

TL;DR;

1. Bumping to geos 3.11.5 (from current 3.11.2) doesn't improve anything.
2. Segfault happens during dynamic loading of the .so when looking for _ITM_deregisterTMCloneTable

---

The issue appears during the dynamic loading of the library, which can be reproduced with this simple C script:

cat test_dlopen.c
#include <stdio.h>
#include <dlfcn.h>

int main() {
    void* handle = dlopen("lib/libgeos.so", RTLD_LAZY);
    if (!handle) {
        fprintf(stderr, "dlopen failed: %s\n", dlerror());
        return 1;
    }
    printf("dlopen succeeded!\n");
    dlclose(handle);
    return 0;
}

$ ./test_dlopen
Segmentation fault (core dumped)

After rebuilding the lib in Debug, when firing gdb, it seems like this happens during loading of ITM instructions:

$ gdb --args ./test_dlopen
(gdb) run
Starting program: /home/ubuntu/test_dlopen
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/s390x-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
do_lookup_x (undef_name=undef_name@entry=0x3fffd5899f0 "_ITM_deregisterTMCloneTable", new_hash=new_hash@entry=2247416213, old_hash=old_hash@entry=0x3ffffffe2e8, ref=0x3fffd582240, result=0x3ffffffe2f8, scope=0x2aa00003580, i=<optimized out>, version=0x0, flags=9, skip=0x0, type_class=0, undef_map=0x2aa00003d70) at dl-lookup.c:453
453	dl-lookup.c: No such file or directory.

ldd doesn't show any broken links:

$ ldd lib/libgeos.so
	linux-vdso64.so.1 (0x000003ff941fe000)
	librt.so.1 => /lib/s390x-linux-gnu/librt.so.1 (0x000003ff93a80000)
	libm.so.6 => /lib/s390x-linux-gnu/libm.so.6 (0x000003ff93980000)
	libc.so.6 => /lib/s390x-linux-gnu/libc.so.6 (0x000003ff93780000)
	/lib/ld64.so.1 (0x000003ff94180000)

Looking at library references, we can see that libgeos.so references deregisterTMCloneTable:

$ readelf --dyn-syms -W lib/libgeos.so | grep _ITM_deregisterTMCloneTable
 33408: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _ITM_deregisterTMCloneTable

And that it is not provided by libgcc, but by libitm:

$ readelf -Ws /lib/s390x-linux-gnu/libgcc_s.so.1 | grep _ITM_deregisterTMCloneTable
     4: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _ITM_deregisterTMCloneTable

$ readelf -Ws /lib/s390x-linux-gnu/libitm.so.1 | grep _ITM_deregisterTMCloneTable
    181: 000000000000b6b0   240 FUNC    GLOBAL DEFAULT   12 _ITM_deregisterTMCloneTable@@LIBITM_1.0

Forcing a link to libitm.so when building the library with CMAKE_SHARED_LINKER_FLAGS=-litm, helps us show that we don't have a mismatch in the library version, but doesn't help with our current problem:

$ ldd lib/libgeos.so
	linux-vdso64.so.1 (0x000003ff923fe000)
	librt.so.1 => /lib/s390x-linux-gnu/librt.so.1 (0x000003ff91100000)
	libitm.so.1 => /lib/s390x-linux-gnu/libitm.so.1 (0x000003ff91080000)
	libm.so.6 => /lib/s390x-linux-gnu/libm.so.6 (0x000003ff90f80000)
	libc.so.6 => /lib/s390x-linux-gnu/libc.so.6 (0x000003ff90d80000)
	/lib/ld64.so.1 (0x000003ff92380000)

$ readelf --dyn-syms -W lib/libgeos.so | grep ITM_deregister
  9409: 0000000000000000     0 FUNC    WEAK   DEFAULT  UND _ITM_deregisterTMCloneTable@LIBITM_1.0 (5)

$ gdb --args ./test_dlopen
(gdb) run
Starting program: /home/ubuntu/test_dlopen
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/s390x-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
do_lookup_x (undef_name=undef_name@entry=0x3fffc9899f0 "_ITM_deregisterTMCloneTable", new_hash=new_hash@entry=2247416213, old_hash=old_hash@entry=0x3ffffffe2e8, ref=0x3fffc982240, result=0x3ffffffe2f8, scope=0x2aa00003580, i=<optimized out>, version=0x0, flags=9, skip=0x0, type_class=0, undef_map=0x2aa00004290) at dl-lookup.c:453
453	dl-lookup.c: No such file or directory.

I did try to pass CMAKE_CXX_FLAGS=-fno-gnu-tm to disable transactional memory, but it doesn't seem to stick in the final library, as readelf still references _ITM_*.

[golgeek]

Comparing with a locally built library on an s390x machine, there are some differences in the shared linked libraries returned by ldd, but nothing substantial:

$ ldd lib/libgeos.linux.local.3.11.5.so
	linux-vdso64.so.1 (0x000003ff86cfe000)
	libstdc++.so.6 => /lib/s390x-linux-gnu/libstdc++.so.6 (0x000003ff86680000)
	libm.so.6 => /lib/s390x-linux-gnu/libm.so.6 (0x000003ff86580000)
	libgcc_s.so.1 => /lib/s390x-linux-gnu/libgcc_s.so.1 (0x000003ff86500000)
	libc.so.6 => /lib/s390x-linux-gnu/libc.so.6 (0x000003ff86300000)
	/lib/ld64.so.1 (0x000003ff86c80000)

$ readelf --dyn-syms -W lib/libgeos.linux.local.3.11.5.so | grep ITM_deregister
   125: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _ITM_deregisterTMCloneTable

$ for i in $(ldd lib/libgeos.linux.local.3.11.5.so | awk -F'=> ' '/=>/ {split($2, a, " "); print a[1]}'); do
> echo "$i:" && readelf --dyn-syms -W $i | grep ITM_deregister;
> done
/lib/s390x-linux-gnu/libstdc++.so.6:
    65: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _ITM_deregisterTMCloneTable
/lib/s390x-linux-gnu/libm.so.6:
     4: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _ITM_deregisterTMCloneTable
/lib/s390x-linux-gnu/libgcc_s.so.1:
     4: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _ITM_deregisterTMCloneTable
/lib/s390x-linux-gnu/libc.so.6:

[golgeek]

Update: patchelf is to blame!

Our current build system relies on patchelf 0.10, and this version somehow messes up the final .so files produced, and bumping patchelf to latest (0.18.0) on the builder fixes the issue.

One thing I still want to test is if we still need patchelf in the build pipeline or if this is an old requirement.