Skip to content

Commit c4fd68c

Browse files
Link the unwinder statically so the Linux artifacts drop libgcc_s.so.1
The released Linux artifacts carry a DT_NEEDED on libgcc_s.so.1. `gcc_s.` is not one of the stems musl's loader resolves internally, so on Alpine it is a real file lookup, and it fails on any image without the `libgcc` package: UnsatisfiedLinkError: .../libnetty_tcnative_linux_aarch_64.so: Error loading shared library libgcc_s.so.1: No such file or directory Invariant 1 permitted that entry on the premise that "Alpine satisfies it with the libgcc package, which the JDK images install". The premise is false. Alpine JDK images disagree: eclipse-temurin:21-jdk-alpine ships libgcc, but amazoncorretto:21-alpine and many vendor images do not. Both architectures are affected. Worse, the verification could not have caught it. Dockerfile.alpine pins eclipse-temurin -- the one base that happens to carry libgcc -- and then runs `apk add binutils`, which depends on libgcc and would drag it in even on a base without it. -static-libgcc was recorded in docs and in a pom comment as a dead end, on the grounds that the C driver leaves the _Unwind_* symbols pulled in by libstdc++.a undefined. That conclusion came from a link that never received the flag. GNU libtool 2.4.6 builds a shared library through archive_cmds, which expands only $libobjs $deplibs $compiler_flags; -l* is parsed into deplibs, but -static-lib* falls into func_mode_link's catch-all `-* | +*)` branch and reaches only the program link path. The generated libnetty_tcnative.la shows it directly: dependency_libs=' ... -lssl -lcrypto -l:libstdc++.a -lrt -lpthread -ldl' -l:libstdc++.a survived; -static-libstdc++, passed on the same LDFLAGS line, is gone -- so the `-l:` form was what linked libstdc++ statically all along, and -static-libstdc++ was dead weight. Changes: - link -l:libgcc.a -l:libgcc_eh.a after -l:libstdc++.a in both Linux profiles (boringssl-static-default and linux-aarch64; they duplicate the whole native build, so a change to one does not apply to the other), and drop the -static-libstdc++ that libtool was discarding. libgcc_eh.a defines the _Unwind_* family; with them resolved the driver's trailing --as-needed -lgcc_s records no DT_NEEDED. Measured identical on GCC 4.9.4, 9.5.0 (x86_64) and 10.5.0 (aarch64), bracketing the shipping devtoolset-9 and ARM GCC 10.2.1: 11 undefined _Unwind_* and libgcc_s.so.1 present before, 0 and absent after. - check_musl_compat.sh: stop allowing libgcc_s.so.1 for the static profile, and add a fatal check for undefined _Unwind_* symbols. The second is what makes the first safe -- it distinguishes "linked statically" from "linked against nothing", so a future build whose flags stop reaching the link fails loudly instead of shipping a library missing its unwinder. Uses '^$' rather than '' for the empty allowlist, because `grep -Eq ""` matches everything and would have silently disabled the DT_NEEDED check. - make the Alpine verification able to detect this class of bug at all. binutils is installed as the virtual package .elftools and removed, with libgcc, before the execution checks when MUSL_DROP_ELFTOOLS=1. That is necessary but not sufficient: temurin's JDK itself depends on libgcc, so add nolibgcc CI legs on amazoncorretto:21-alpine, whose JDK does not. Verified those legs reproduce the failure above on the current artifact and pass on the fixed one. - ci-deploy-github-packages.yml: widen the push paths filter, which listed only the publishing machinery, so a pom or native-source change could not trigger a republish. - document the libtool flag-filtering trap as a general rule, and record that it also explains why -fsanitize=address never reaches the link in the boringssl-static-asan profile (not fixed here). On aarch64 this pulls libgcc's init_have_lse_atomics into the library as an .init_array constructor referencing __getauxval -- a Class C symbol, which SIGSEGVs the JVM inside dlopen rather than raising UnsatisfiedLinkError. It is satisfied by the existing weak, default-visibility fallback in musl_compat.c, and check_musl_compat.sh already fails the build if that regresses. Confirmed defined, not undefined, in a matching link. See #907, #152
1 parent 357a406 commit c4fd68c

9 files changed

Lines changed: 282 additions & 27 deletions

File tree

‎.github/workflows/ci-build.yml‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -161,34 +161,60 @@ jobs:
161161
jars: build-centos6-x86_64-jars
162162
variant: bare
163163
extra-pkgs: ""
164+
drop-elftools: "1"
164165
build-service: runtime-setup
165166
run-service: verify
166167
- setup: alpine-aarch64
167168
os: ubuntu-24.04-arm
168169
jars: build-centos7-aarch64-jars
169170
variant: bare
170171
extra-pkgs: ""
172+
drop-elftools: "1"
173+
build-service: runtime-setup
174+
run-service: verify
175+
# The invariant leg: an Alpine JDK that ships no `libgcc` package of its own, so
176+
# `apk del .elftools` really does leave libgcc absent. eclipse-temurin's JDK depends on
177+
# libgcc, so the bare legs above cannot prove the artifact loads without it.
178+
- setup: alpine-x86_64-nolibgcc
179+
os: ubuntu-24.04
180+
jars: build-centos6-x86_64-jars
181+
variant: nolibgcc
182+
extra-pkgs: ""
183+
drop-elftools: "1"
184+
jdk-image: amazoncorretto:21-alpine
185+
build-service: runtime-setup
186+
run-service: verify
187+
- setup: alpine-aarch64-nolibgcc
188+
os: ubuntu-24.04-arm
189+
jars: build-centos7-aarch64-jars
190+
variant: nolibgcc
191+
extra-pkgs: ""
192+
drop-elftools: "1"
193+
jdk-image: amazoncorretto:21-alpine
171194
build-service: runtime-setup
172195
run-service: verify
173196
- setup: alpine-x86_64-gcompat
174197
os: ubuntu-24.04
175198
jars: build-centos6-x86_64-jars
176199
variant: gcompat
177200
extra-pkgs: gcompat
201+
drop-elftools: "0"
178202
build-service: runtime-setup
179203
run-service: verify
180204
- setup: alpine-aarch64-gcompat
181205
os: ubuntu-24.04-arm
182206
jars: build-centos7-aarch64-jars
183207
variant: gcompat
184208
extra-pkgs: gcompat
209+
drop-elftools: "0"
185210
build-service: runtime-setup
186211
run-service: verify
187212
- setup: glibc-control-x86_64
188213
os: ubuntu-24.04
189214
jars: build-centos6-x86_64-jars
190215
variant: glibc-control
191216
extra-pkgs: ""
217+
drop-elftools: "0"
192218
build-service: control-runtime-setup
193219
run-service: control-verify
194220

@@ -207,11 +233,14 @@ jobs:
207233
env:
208234
MUSL_VARIANT: ${{ matrix.variant }}
209235
MUSL_EXTRA_PKGS: ${{ matrix.extra-pkgs }}
236+
MUSL_JDK_IMAGE: ${{ matrix.jdk-image }}
210237
run: docker compose -f docker/docker-compose.alpine.yaml build ${{ matrix.build-service }}
211238

212239
- name: Verify the artifact under ${{ matrix.variant }}
213240
env:
214241
MUSL_VARIANT: ${{ matrix.variant }}
215242
MUSL_EXTRA_PKGS: ${{ matrix.extra-pkgs }}
243+
MUSL_DROP_ELFTOOLS: ${{ matrix.drop-elftools }}
244+
MUSL_JDK_IMAGE: ${{ matrix.jdk-image }}
216245
MUSL_JARS_DIR: musl-verify-jars
217246
run: docker compose -f docker/docker-compose.alpine.yaml run --rm ${{ matrix.run-service }}

‎.github/workflows/ci-deploy-github-packages.yml‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,9 +8,17 @@ on:
88
# deliberately lives on the test branch only. Trigger on push to that branch instead.
99
push:
1010
branches: [ musl-compat ]
11+
# Anything that changes the produced artifact must be able to trigger a republish, not just the
12+
# publishing machinery itself -- a pom or native-source change with the old filter silently did
13+
# nothing.
1114
paths:
1215
- '.github/workflows/ci-deploy-github-packages.yml'
13-
- '.github/scripts/retarget_staging_index.sh'
16+
- '.github/scripts/**'
17+
- 'scripts/**'
18+
- 'pom.xml'
19+
- 'boringssl-static/pom.xml'
20+
- 'openssl-dynamic/**'
21+
- 'docker/**'
1422

1523
# Usable once/if this workflow is also present on the default branch.
1624
workflow_dispatch:

‎.github/workflows/ci-pr.yml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -214,13 +214,36 @@ jobs:
214214
jars: build-pr-centos6-x86_64-jars
215215
variant: bare
216216
extra-pkgs: ""
217+
drop-elftools: "1"
217218
build-service: runtime-setup
218219
run-service: verify
219220
- setup: alpine-aarch64
220221
os: ubuntu-24.04-arm
221222
jars: build-pr-centos7-aarch64-jars
222223
variant: bare
223224
extra-pkgs: ""
225+
drop-elftools: "1"
226+
build-service: runtime-setup
227+
run-service: verify
228+
# The invariant leg: an Alpine JDK that ships no `libgcc` package of its own, so
229+
# `apk del .elftools` really does leave libgcc absent. eclipse-temurin's JDK depends on
230+
# libgcc, so the bare legs above cannot prove the artifact loads without it.
231+
- setup: alpine-x86_64-nolibgcc
232+
os: ubuntu-24.04
233+
jars: build-pr-centos6-x86_64-jars
234+
variant: nolibgcc
235+
extra-pkgs: ""
236+
drop-elftools: "1"
237+
jdk-image: amazoncorretto:21-alpine
238+
build-service: runtime-setup
239+
run-service: verify
240+
- setup: alpine-aarch64-nolibgcc
241+
os: ubuntu-24.04-arm
242+
jars: build-pr-centos7-aarch64-jars
243+
variant: nolibgcc
244+
extra-pkgs: ""
245+
drop-elftools: "1"
246+
jdk-image: amazoncorretto:21-alpine
224247
build-service: runtime-setup
225248
run-service: verify
226249
# Control: anything failing on Alpine must pass here, or the check is at fault rather
@@ -230,6 +253,7 @@ jobs:
230253
jars: build-pr-centos6-x86_64-jars
231254
variant: glibc-control
232255
extra-pkgs: ""
256+
drop-elftools: "0"
233257
build-service: control-runtime-setup
234258
run-service: control-verify
235259

@@ -248,11 +272,14 @@ jobs:
248272
env:
249273
MUSL_VARIANT: ${{ matrix.variant }}
250274
MUSL_EXTRA_PKGS: ${{ matrix.extra-pkgs }}
275+
MUSL_JDK_IMAGE: ${{ matrix.jdk-image }}
251276
run: docker compose -f docker/docker-compose.alpine.yaml build ${{ matrix.build-service }}
252277

253278
- name: Verify the artifact under ${{ matrix.variant }}
254279
env:
255280
MUSL_VARIANT: ${{ matrix.variant }}
256281
MUSL_EXTRA_PKGS: ${{ matrix.extra-pkgs }}
282+
MUSL_DROP_ELFTOOLS: ${{ matrix.drop-elftools }}
283+
MUSL_JDK_IMAGE: ${{ matrix.jdk-image }}
257284
MUSL_JARS_DIR: musl-verify-jars
258285
run: docker compose -f docker/docker-compose.alpine.yaml run --rm ${{ matrix.run-service }}

‎boringssl-static/pom.xml‎

Lines changed: 28 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -599,15 +599,26 @@
599599
<echo message="Setup flags to use during linking" />
600600

601601
<!-- If we compile on linux we need to adjust the ldflags to correct link libstdc++ -->
602-
<!-- NOTE: -static-libgcc was tried here to drop the DT_NEEDED on libgcc_s.so.1
603-
(not a musl-reserved name). It is a no-op in this build: the library is linked
604-
with the C driver, so the _Unwind_* symbols pulled in by libstdc++.a stay
605-
undefined and libgcc_s.so.1 remains genuinely required. Alpine satisfies it
606-
with the `libgcc` package, which the JDK images already install. -->
602+
<!-- The libgcc archives must come *after* libstdc++.a: it is what pulls in the
603+
_Unwind_* symbols, and libgcc_eh.a is what defines them. Linking them
604+
statically is what keeps libgcc_s.so.1 out of DT_NEEDED - `gcc_s.` is not a
605+
musl-reserved stem, so on Alpine that entry is a real file lookup and fails
606+
outright on any image without the `libgcc` package (Amazon Corretto and many
607+
vendor JDK images ship none; eclipse-temurin does, which is why this went
608+
unnoticed). scripts/check_musl_compat.sh asserts both halves.
609+
610+
NOTE: spell this as `-l:` and never `-static-libgcc`. GNU libtool 2.4.6 links
611+
the shared library via archive_cmds, which expands only $libobjs $deplibs
612+
$compiler_flags; `-l*` is parsed into deplibs, but `-static-lib*` falls into
613+
func_mode_link's catch-all `-* | +*)` branch and reaches only the *program*
614+
link path, so gcc never sees it. That is why -static-libgcc looked like a
615+
no-op here. `-static-libstdc++` was dropped from this line for the same
616+
reason - it was silently discarded, and -l:libstdc++.a is what actually
617+
linked libstdc++ statically. See docs/musl-compatibility.md. -->
607618
<if>
608619
<equals arg1="${os.detected.name}" arg2="linux" />
609620
<then>
610-
<property name="hawtjniLdflags" value="-L${boringsslHome} -lssl -lcrypto -static-libstdc++ -l:libstdc++.a" />
621+
<property name="hawtjniLdflags" value="-L${boringsslHome} -lssl -lcrypto -l:libstdc++.a -l:libgcc.a -l:libgcc_eh.a" />
611622
</then>
612623
<else>
613624
<property name="hawtjniLdflags" value="${ldflags}" />
@@ -1095,7 +1106,17 @@
10951106
<configureArg>--libdir=${project.build.directory}/native-build/target/lib</configureArg>
10961107
<configureArg>CFLAGS=-O3 -Werror -fno-omit-frame-pointer -fvisibility=hidden -Wunused -Wno-unused-value</configureArg>
10971108
<configureArg>CPPFLAGS=-DHAVE_OPENSSL -I${boringsslSourceDir}/include</configureArg>
1098-
<configureArg>LDFLAGS=-L${boringsslHome} -lssl -lcrypto -static-libstdc++ -l:libstdc++.a</configureArg>
1109+
<!-- Keep in sync with the boringssl-static-default profile: the two Linux
1110+
profiles duplicate the whole native build, so a change to one does not apply
1111+
to the other. The libgcc archives must follow libstdc++.a, and must be
1112+
spelled `-l:` rather than -static-libgcc because libtool drops the latter -
1113+
see the comment on hawtjniLdflags in that profile. On aarch64 this also pulls
1114+
libgcc's init_have_lse_atomics constructor into the library, whose
1115+
__getauxval reference is satisfied by musl_compat.c; check_musl_compat.sh
1116+
fails the build if that ever stops being true, because an unresolved symbol
1117+
in an ELF constructor SIGSEGVs the JVM inside dlopen rather than raising
1118+
UnsatisfiedLinkError. -->
1119+
<configureArg>LDFLAGS=-L${boringsslHome} -lssl -lcrypto -l:libstdc++.a -l:libgcc.a -l:libgcc_eh.a</configureArg>
10991120
<configureArg>--host=aarch64-linux-gnu</configureArg>
11001121
<configureArg>CC=aarch64-none-linux-gnu-gcc</configureArg>
11011122
</configureArgs>

‎docker/Dockerfile.alpine‎

Lines changed: 20 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -5,17 +5,29 @@
55
# work on musl. Nothing here runs Maven, so the os.detected.classifier enforcer rules never come
66
# into play.
77
#
8-
# The base is deliberately a stock JDK image with no compatibility shim: it carries musl, libgcc
9-
# and libstdc++ and no gcompat or libc6-compat, so /lib/libc.so.6 does not exist. That is what
10-
# "bare" below means, and invariant 1 in docs/musl-compatibility.md permits libgcc_s.so.1
11-
# precisely because Alpine's libgcc package provides it. If a future base image starts shipping
12-
# gcompat, the bare variant silently stops testing anything -- verify.sh prints the installed
13-
# compat packages on every run so that shows up in the log.
8+
# The base is deliberately a stock JDK image with no compatibility shim: no gcompat and no
9+
# libc6-compat, so /lib/libc.so.6 does not exist. That is what "bare" below means. If a future base
10+
# image starts shipping gcompat, the bare variant silently stops testing anything -- verify.sh
11+
# prints the installed compat packages on every run so that shows up in the log.
12+
#
13+
# The default base does ship libgcc and libstdc++, and that is NOT representative: Alpine JDK
14+
# images differ on it (Amazon Corretto and various vendor images carry no `libgcc` package), which
15+
# is precisely how a libgcc_s.so.1 DT_NEEDED shipped unnoticed for eight releases.
16+
#
17+
# Two knobs are needed to test that honestly, because there are two ways libgcc sneaks in:
18+
# - MUSL_DROP_ELFTOOLS=1 makes verify.sh uninstall the binutils below, which depend on libgcc;
19+
# - jdk_image must name a JDK that does not itself depend on libgcc. eclipse-temurin does, so on
20+
# that base `apk del` cannot remove it and the leg proves nothing about this invariant.
21+
# amazoncorretto:21-alpine does not, and still provides javac and jar.
22+
# See docs/musl-compatibility.md §4b.
1423
ARG jdk_image=eclipse-temurin:21-jdk-alpine
1524
FROM ${jdk_image}
1625

17-
# binutils for readelf/nm. It provides no libc symbols, so it cannot mask a missing one.
18-
RUN apk add --no-cache binutils
26+
# binutils for readelf/nm. It provides no libc symbols, so it cannot mask a missing one -- but it
27+
# does pull `libgcc` in as a dependency, which would mask exactly the failure this image exists to
28+
# detect. Installed as a virtual package so verify.sh can remove it, and libgcc with it, once the
29+
# ELF inspection is done and only the runtime checks remain.
30+
RUN apk add --no-cache --virtual .elftools binutils
1931

2032
# Set per variant rather than baked in, so each package set is measured independently.
2133
# Empty is the invariant; "gcompat" exists to make a bare failure diagnostic, because a fix that

‎docker/docker-compose.alpine.yaml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,12 +15,19 @@ services:
1515
dockerfile: docker/Dockerfile.alpine
1616
args:
1717
EXTRA_PKGS: "${MUSL_EXTRA_PKGS:-}"
18+
# eclipse-temurin's JDK depends on libgcc, so `apk del` cannot take it away there and that
19+
# base can never prove the artifact loads without it. Point this at an Alpine JDK that
20+
# carries no libgcc (amazoncorretto:21-alpine) for the leg that tests exactly that.
21+
jdk_image: "${MUSL_JDK_IMAGE:-eclipse-temurin:21-jdk-alpine}"
1822

1923
common: &common
2024
image: netty-tcnative-alpine:${MUSL_VARIANT:-bare}
2125
depends_on: [runtime-setup]
2226
environment:
2327
- MUSL_VARIANT=${MUSL_VARIANT:-bare}
28+
# Set to 1 on the bare legs: uninstalls binutils, and libgcc with it, before the runtime
29+
# checks. Leave unset for the gcompat legs, whose point is to load with the shim present.
30+
- MUSL_DROP_ELFTOOLS=${MUSL_DROP_ELFTOOLS:-0}
2431
volumes:
2532
- ..:/code:delegated
2633
working_dir: /code

‎docker/musl-verify/verify.sh‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -128,6 +128,25 @@ else
128128
fi
129129
rm -f "$UND" "$AVAIL"
130130

131+
# ------------------------------------------------------- drop the ELF tools (and libgcc with them)
132+
# Everything above needs binutils; nothing below does. binutils depends on libgcc, so on an image
133+
# whose JDK does not itself require libgcc, installing it silently satisfies a libgcc_s.so.1
134+
# DT_NEEDED and the bare leg stops testing what it claims to. Removing it here means the execution
135+
# checks -- the ones that actually decide the outcome -- run against the package set a stock Alpine
136+
# JDK image really has. Guarded on apk so the Debian glibc-control image skips this untouched.
137+
if [ "${MUSL_DROP_ELFTOOLS:-0}" = "1" ] && command -v apk >/dev/null 2>&1; then
138+
echo "-- dropping ELF tools so the runtime checks see no libgcc"
139+
apk del .elftools >/dev/null 2>&1 || echo " WARN: could not remove the .elftools virtual package"
140+
echo " compat now: $(apk info 2>/dev/null | grep -Ex 'gcompat|libc6-compat|libgcc|libstdc\+\+' | sort | tr '\n' ' ')"
141+
for lib in /usr/lib/libgcc_s.so.1 /usr/lib/libstdc++.so.6; do
142+
if [ -e "$lib" ]; then
143+
echo " note: $lib is still present (the JDK itself depends on it)"
144+
else
145+
echo " ok: $lib is absent"
146+
fi
147+
done
148+
fi
149+
131150
# ---------------------------------------------------------------- execute
132151
echo "-- load and initialize"
133152
CLASSES="$WORK/classes"

0 commit comments

Comments
 (0)