Skip to content

Commit be639e2

Browse files
chenBrightchenBright
committed
Fix(build): hide libunwind's _Unwind_* symbols under Bazel to fix bthread_tracer crash
1 parent d8ddff8 commit be639e2

16 files changed

Lines changed: 1912 additions & 2 deletions

File tree

‎.bazelignore‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,8 @@
1-
./example/build_with_bazel
1+
./example/build_with_bazel
2+
3+
# `registry/` is brpc's self-maintained Bzlmod registry. Its overlay
4+
# BUILD.bazel files reference sources from the libunwind tarball that is
5+
# only materialized at module resolution time, not in the source tree.
6+
# Without this entry, `bazel build //...` would try to evaluate them as
7+
# regular packages and fail with "missing input file ...".
8+
./registry

‎.licenserc.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -218,4 +218,7 @@ header:
218218
# Fuzzing seed
219219
- 'test/fuzzing/fuzz_*_seed_corpus/*'
220220

221+
# bazel-central-registry
222+
- 'registry/**'
223+
221224
comment: on-failure

‎LICENSE‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -959,3 +959,41 @@ copyright The Chromium Authors and licensed under the 3-clause BSD license:
959959
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
960960
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
961961
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
962+
963+
--------------------------------------------------------------------------------
964+
965+
registry/modules/libunwind/**: licensed under the following terms:
966+
967+
Forked from the Bazel Central Registry (BCR):
968+
https://github.com/bazelbuild/bazel-central-registry/tree/main/modules/libunwind
969+
970+
The upstream files are distributed under the Apache License, Version 2.0,
971+
the same license as brpc itself (the full text of which is reproduced at
972+
the top of this LICENSE file). brpc has modified the forked files; per the
973+
Apache License 2.0 §4(b), the modifications are summarized below:
974+
975+
- registry/modules/libunwind/<version>/MODULE.bazel
976+
- registry/modules/libunwind/<version>/overlay/MODULE.bazel
977+
Append the suffix `.brpc-no-unwind` to the `version` field, marking
978+
these as brpc's variant of the libunwind module.
979+
980+
- registry/modules/libunwind/<version>/overlay/BUILD.bazel
981+
Add the `hide_unwind_symbols` config_setting (gated by
982+
`--define=libunwind_hide_unwind_symbols=true`); when the switch is
983+
on, drop `src/unwind/*.c` (the GCC `_Unwind_*` ABI compatibility
984+
layer) from `unwind_srcs` so the resulting libunwind does not export
985+
`_Unwind_*` symbols. See docs/cn/bthread_tracer.md for the
986+
rationale.
987+
988+
- registry/modules/libunwind/<version>/source.json
989+
Update overlay file SHA-256 hashes to match the modified
990+
BUILD.bazel / MODULE.bazel.
991+
992+
- registry/modules/libunwind/metadata.json
993+
Replace the `versions` array with brpc's renamed versions.
994+
995+
The above only governs the build/source files brpc redistributes inside
996+
registry/modules/libunwind/. The libunwind source code itself, downloaded
997+
by Bazel from the URL pinned in source.json, remains governed by the
998+
libunwind project's own license; see https://github.com/libunwind/libunwind
999+
for details.

‎bazel/config/BUILD.bazel‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -136,6 +136,7 @@ config_setting(
136136
define_values = {
137137
"with_bthread_tracer": "true",
138138
},
139+
visibility = ["//visibility:public"],
139140
)
140141

141142
config_setting(

‎docs/cn/bthread_tracer.md‎

Lines changed: 143 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -58,7 +58,7 @@ jump_stack是bthread挂起或者运行的必经之路，也是STB的拦截点。
5858

5959
# 使用方法
6060

61-
1. 下载安装libunwind和abseil-cpp。
61+
1. 下载安装libunwind和abseil-cpp。**注意：libunwind 必须从源码编译，不要使用系统包管理器安装的 `libunwind-dev` / `libunwind-devel`**，否则会触发本文末尾「[已知问题：libunwind 与 libgcc_s 的 `_Unwind_*` 符号冲突](#已知问题libunwind-与-libgcc_s-的-_unwind_-符号冲突)」中描述的崩溃。bazel 构建可以跳过此步，直接使用 brpc 仓库自维护的 libunwind 版本。
6262
2. 给config_brpc.sh增加`--with-bthread-tracer`选项或者给cmake增加`-DWITH_BTHREAD_TRACER=ON`选项或者给bazel（Bzlmod模式）增加`--define with_bthread_tracer=true`选项。
6363
3. 访问服务的内置服务：`http://ip:port/bthreads/<bthread_id>?st=1`或者代码里调用`bthread::stack_trace()`函数。
6464
4. 如果希望追踪pthread的调用栈，在对应pthread上调用`bthread::init_for_pthread_stack_trace()`函数获取一个伪bthread_t，然后使用步骤3即可获取pthread调用栈。
@@ -73,6 +73,148 @@ jump_stack是bthread挂起或者运行的必经之路，也是STB的拦截点。
7373
#5 0x00007fdbbfa58dc0 bthread::TaskGroup::task_runner()
7474
```
7575

76+
# 已知问题
77+
78+
## libunwind 与 libgcc_s 的 `_Unwind_*` 符号冲突
79+
80+
### 现象
81+
82+
启用 bthread tracer 后，可能在 `bthread_exit` / `pthread_exit` 或者 C++ 异常处理路径上偶发段错误，类似如下调用栈：
83+
84+
```text
85+
#0 0x0000000000000000 in ?? ()
86+
#1 _ULx86_64_dwarf_find_proc_info () in libunwind.so
87+
#2 fetch_proc_info () in libunwind.so
88+
#3 _ULx86_64_dwarf_make_proc_info () in libunwind.so
89+
#4 _ULx86_64_get_proc_info () in libunwind.so
90+
#5 __libunwind_Unwind_GetLanguageSpecificData () in libunwind.so
91+
#6 __gxx_personality_v0 () in libstdc++.so.6
92+
#7 ?? () in libgcc_s.so.1
93+
#8 _Unwind_ForcedUnwind () in libgcc_s.so.1
94+
#9 __GI___pthread_unwind () in libc / libpthread
95+
#10 __do_cancel () in libc / libpthread
96+
#11 __GI___pthread_exit () in libc / libpthread
97+
```
98+
99+
### 根因
100+
101+
libunwind 的 `src/unwind/*.c` 实现了 GCC 的 `_Unwind_*` ABI 兼容层（`_Unwind_GetLanguageSpecificData`、`_Unwind_ForcedUnwind`、`_Unwind_Resume` 等），与 `libgcc_s.so.1` 提供同名的全局符号。当 libunwind 以**共享库**形式被链接，并且在最终二进制的 `DT_NEEDED` 列表中位置比 `libgcc_s.so.1` 靠前时，运行时动态链接器 `ld.so` 会把 `pthread_exit` / 异常处理触发的 `_Unwind_*` 调用解析到 libunwind 中的 DWARF 实现。该实现需要的内部上下文在 `pthread_exit` 路径上未被 brpc 初始化好，从而触发空指针访问。
102+
103+
这是一个 ELF **运行时符号解析顺序**问题，与编译器（GCC / Clang）无关 —— Clang 默认运行时同样使用 `libstdc++ + libgcc_s`，会复现完全一致的崩溃。
104+
105+
### 解决方案
106+
107+
> **重要：不要使用系统包管理器安装的 libunwind**（例如 `apt install libunwind-dev`、`yum install libunwind-devel`）。多数发行版打包的 `libunwind.so` 仍把 `_Unwind_*` 暴露在动态符号表中，会触发本节描述的崩溃。
108+
>
109+
> 必须使用**从源码编译的 libunwind**。上游 `./configure` + `make` 默认会通过 `-Wl,--version-script` 把 `_Unwind_*` 标为 local，不导出到动态符号表，从而避免冲突。
110+
111+
下表汇总了三种构建方式的推荐方案：
112+
113+
| 构建方式 | 推荐方案 |
114+
|---|---|
115+
| `config_brpc.sh` + `make` | 从源码编译并安装 libunwind，把头文件与库目录显式传给 `config_brpc.sh` |
116+
| `cmake` | 从源码编译并安装 libunwind，把头文件与库目录显式传给 `cmake` |
117+
| `bazel`（Bzlmod） | 直接使用 brpc 仓库自维护的 libunwind 版本（已默认配置好，零额外操作） |
118+
119+
### `make`（`config_brpc.sh`）
120+
121+
源码编译并安装 libunwind 到一个独立目录（避免污染系统目录），然后让 `config_brpc.sh` 显式从该目录查找 libunwind。下面的步骤与 brpc CI [`.github/actions/init-ut-make-config/action.yml`](../../.github/actions/init-ut-make-config/action.yml:16-19) 完全一致：
122+
123+
```bash
124+
# 1) 源码编译 libunwind（推荐 v1.8.1 或以上版本）
125+
git clone https://github.com/libunwind/libunwind.git
126+
cd libunwind && git checkout tags/v1.8.1
127+
mkdir -p /libunwind
128+
autoreconf -i
129+
./configure --prefix=/libunwind
130+
make -j$(nproc) && make install
131+
cd ..
132+
133+
# 2) 让 config_brpc.sh 使用 /libunwind 下的头文件与库（不要让它自动找到系统的 libunwind-dev）
134+
cd brpc
135+
sh config_brpc.sh \
136+
--with-bthread-tracer \
137+
--headers="/libunwind/include /usr/include" \
138+
--libs="/libunwind/lib /usr/lib /usr/lib64"
139+
make -j$(nproc)
140+
```
141+
142+
构建完成后可用以下命令确认 libunwind.so 没有导出 `_Unwind_*`：
143+
144+
```bash
145+
nm -D /libunwind/lib/libunwind.so | grep ' T _Unwind_' \
146+
&& echo "WARN: _Unwind_* exported" \
147+
|| echo "OK: _Unwind_* hidden"
148+
```
149+
150+
### `cmake`
151+
152+
[`CMakeLists.txt`](../../CMakeLists.txt:90-100) 通过 `find_library(... NAMES unwind unwind-x86_64)` 查找 libunwind。同样需要先源码编译 libunwind 到独立前缀，再用 `CMAKE_PREFIX_PATH` 让 cmake 优先在该前缀下查找：
153+
154+
```bash
155+
# 1) 源码编译 libunwind（同 make 章节）
156+
git clone https://github.com/libunwind/libunwind.git
157+
cd libunwind && git checkout tags/v1.8.1
158+
mkdir -p /libunwind
159+
autoreconf -i
160+
./configure --prefix=/libunwind
161+
make -j$(nproc) && make install
162+
cd ..
163+
164+
# 2) 让 cmake 在 /libunwind 下优先查找头文件和库
165+
cd brpc
166+
mkdir build && cd build
167+
cmake -DWITH_BTHREAD_TRACER=ON \
168+
-DCMAKE_PREFIX_PATH=/libunwind \
169+
..
170+
make -j$(nproc)
171+
```
172+
173+
> 提示：如果系统已经装了 `libunwind-dev`，`find_library` 仍可能优先匹配到 `/usr/lib`。可在 cmake 命令上额外指定
174+
> `-DLIBUNWIND_LIB=/libunwind/lib/libunwind.so -DLIBUNWIND_X86_64_LIB=/libunwind/lib/libunwind-x86_64.so -DLIBUNWIND_INCLUDE_PATH=/libunwind/include`
175+
> 强制走自编译版本，避免系统包混入。
176+
177+
### `bazel`（Bzlmod）
178+
179+
Bazel 默认 fastbuild 模式会为每个 `cc_library` 生成中间 `.so`（例如 `libexternal_Slibunwind~_Slibunwind.so`），把 libunwind 强制变成一个独立共享库，其 `_Unwind_*` 符号会被导出抢占 `libgcc_s.so.1` —— **问题极易复现**。
180+
181+
brpc 仓库已经在 [`registry/modules/libunwind/`](../../registry/modules/libunwind/) 自维护了一份 libunwind 的 Bzlmod overlay，并通过 [`.bazelrc`](../../.bazelrc) 中的 `--registry=file://%workspace%/registry` 让它覆盖默认 BCR 中的 libunwind。版本号采用 `<base>.brpc-no-unwind` 后缀（例如 `1.8.3.brpc-no-unwind`），用以区别于 BCR 上的同基版本，并直观体现"由 brpc 维护，不导出 `_Unwind_*` 符号"。该 overlay 增加了一个开关：
182+
183+
```
184+
--define=libunwind_hide_unwind_symbols=true
185+
```
186+
187+
开启后，libunwind 的 `src/unwind/*.c`（即 GCC `_Unwind_*` 兼容层）整体不参与编译，等效于上游 autoconf `--version-script` 的隐藏效果。brpc 只使用 libunwind 的 `unw_*` 原生 API（`unw_getcontext`、`unw_init_local`、`unw_step` 等），不依赖 `_Unwind_*` 兼容层，因此该开关安全无副作用。
188+
189+
`.bazelrc` 已默认在 `build:test` / `test` 配置下打开此开关：
190+
191+
```
192+
build:test --define libunwind_hide_unwind_symbols=true
193+
test --define libunwind_hide_unwind_symbols=true
194+
```
195+
196+
**因此 bazel 用户无需任何额外操作**，按文档使用方法第 2 步加上 `--define=with_bthread_tracer=true` 即可：
197+
198+
```bash
199+
# 测试场景，.bazelrc 中 test 配置已自动带上 hide 开关
200+
bazel test //test:bthread_unittest
201+
202+
# 非测试构建（生产部署），需要显式同时带上两个 define
203+
bazel build --define=with_bthread_tracer=true \
204+
--define=libunwind_hide_unwind_symbols=true \
205+
//...
206+
```
207+
208+
> **特别注意**：如果在生产构建中只启用 `--define=with_bthread_tracer=true` 而漏掉 `--define=libunwind_hide_unwind_symbols=true`，binary 在 `pthread_exit` / 异常路径上会概率性崩溃。
209+
210+
构建后可用以下命令验证 libunwind 共享库没有导出 `_Unwind_*`：
211+
212+
```bash
213+
nm -D bazel-bin/external/_solib_*/libexternal*libunwind*.so 2>/dev/null \
214+
| grep ' T _Unwind_' || echo "OK: no _Unwind_* exported by libunwind.so"
215+
```
216+
217+
76218
# 相关flag
77219

78220
- `signal_trace_timeout_ms`：信号追踪模式的超时时间，默认为50ms。

‎registry/modules/libunwind/1.8.1.brpc-no-unwind/MODULE.bazel‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# Forked from the Bazel Central Registry:
2+
# https://github.com/bazelbuild/bazel-central-registry/blob/main/modules/libunwind/1.8.1/MODULE.bazel
3+
# Distributed under the Apache License, Version 2.0. See the `registry/modules/
4+
# libunwind/**` entry near the bottom of brpc's LICENSE file.
5+
#
6+
# brpc modification (Apache License 2.0 §4(b)):
7+
# - `version` suffixed with `.brpc-no-unwind` to distinguish this brpc
8+
# variant from the upstream BCR version.
9+
10+
module(
11+
name = "libunwind",
12+
version = "1.8.1.brpc-no-unwind",
13+
bazel_compatibility = [">=7.2.1"], # need support for "overlay" directory
14+
compatibility_level = 0,
15+
)
16+
17+
bazel_dep(name = "bazel_skylib", version = "1.7.1")
18+
bazel_dep(name = "platforms", version = "0.0.11")
19+
bazel_dep(name = "rules_cc", version = "0.1.1")
20+
bazel_dep(name = "xz", version = "5.4.5.bcr.5")
21+
bazel_dep(name = "zlib", version = "1.3.1.bcr.5")

0 commit comments

Comments
 (0)