debug: repoint the fixed leg at the thread-lifetime commit #2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # x86_64 confirmation soak for the plugin dlopen/dlclose SIGSEGV. | |
| # | |
| # The root cause was captured on Linux/aarch64 (core dump: a pthread-key TLS destructor inside a | |
| # dlclose'd plugin image, called by __nptl_deallocate_tsd when the thread that armed it exits). The | |
| # fix (busbar fix/1.6.0-mysql-e2e-segv, RTLD_NODELETE) is class-level: nothing is ever unmapped. | |
| # | |
| # THE CONTROL LEG IS THE POINT. At the observed ~3% per-run rate, a green fixed leg at any N this | |
| # workflow can afford is weak evidence on its own (150 clean runs is ~1% likely by luck, 60 is ~16%). | |
| # What this soak is actually for is confirming that the UNFIXED x86_64 crash carries the same | |
| # __nptl_deallocate_tsd signature as the aarch64 one — i.e. that CI's wild crash and the diagnosed | |
| # crash are the same defect. A red control with that backtrace is the result worth having. | |
| name: segv soak (x86_64, control vs fixed) | |
| on: | |
| workflow_dispatch: {} | |
| push: | |
| branches: [debug/1.6.0-segv-soak-x86] | |
| jobs: | |
| soak: | |
| runs-on: ubuntu-latest | |
| strategy: | |
| fail-fast: false | |
| matrix: | |
| include: | |
| # UNFIXED: the exact pin the wild crashes were seen on. | |
| - leg: control | |
| busbar_ref: eac13fa3fecdd21ab3530751016a0c956e7e27b5 | |
| # FIXED: the thread-lifetime commit on fix/1.6.0-plugin-thread-lifetime. Plugin code runs | |
| # only on loader-owned workers that never retire, so no thread carrying plugin TLS ever | |
| # exits. (The withdrawn RTLD_NODELETE leg is deliberately gone — it returned the wrong | |
| # plugin's image and must never be a comparison point again.) | |
| - leg: fixed | |
| busbar_ref: 225e9e176ac17d6f147174aa40d82bc2172440a6 | |
| name: ${{ matrix.leg }} | |
| services: | |
| mysql: | |
| image: mysql:8@sha256:b3b90af2a6552ae30c266fdb7d5dd55f3afb72404bb78d37fe8a23eb857fd3fb | |
| env: | |
| MYSQL_ROOT_PASSWORD: busbar | |
| MYSQL_USER: busbar | |
| MYSQL_PASSWORD: busbar | |
| MYSQL_DATABASE: busbar_test | |
| ports: | |
| - 3306:3306 | |
| options: >- | |
| --health-cmd "mysqladmin ping -h localhost -ubusbar -pbusbar" --health-interval 10s --health-timeout 5s --health-retries 10 | |
| steps: | |
| - name: Checkout the plugin | |
| uses: actions/checkout@v7 | |
| with: | |
| path: plugin | |
| - name: Checkout busbar at this leg's pin | |
| uses: actions/checkout@v7 | |
| with: | |
| repository: GetBusbar/busbar | |
| ref: ${{ matrix.busbar_ref }} | |
| path: busbarAI | |
| - name: Show which loader this leg actually built against | |
| run: | | |
| echo "leg=${{ matrix.leg }} busbar_ref=${{ matrix.busbar_ref }}" | |
| # The one line that distinguishes the two legs, printed so a green run cannot be | |
| # mistaken for a leg that silently checked out the wrong tree. | |
| echo -n "ffi_thread references in the loader: " | |
| grep -c ffi_thread busbarAI/crates/plugin-loader/src/lib.rs || echo "0 (unfixed loader)" | |
| - uses: dtolnay/rust-toolchain@stable | |
| - name: mysql — set strict sql_mode | |
| run: mysql -h 127.0.0.1 -uroot -pbusbar -e "SET GLOBAL sql_mode='STRICT_ALL_TABLES,NO_ENGINE_SUBSTITUTION';" | |
| - name: core dumps + gdb | |
| run: | | |
| sudo apt-get update -qq && sudo apt-get install -y -qq gdb | |
| sudo mkdir -p /tmp/cores && sudo chmod 1777 /tmp/cores | |
| echo "/tmp/cores/core.%e.%p" | sudo tee /proc/sys/kernel/core_pattern | |
| - name: build the e2e test binary | |
| working-directory: plugin | |
| run: cargo test -p busbar-store-mysql-plugin --test e2e --no-run | |
| env: | |
| BUSBAR_TEST_MYSQL_URL: mysql://busbar:busbar@127.0.0.1:3306/busbar_test | |
| # Runs the built binary DIRECTLY at --test-threads=8 rather than through `cargo test`. That is | |
| # the shape that reproduced locally at ~3%/run; default parallelism is what produced the 25 | |
| # clean attempts. The file-drop case is skipped because it shells out to a full `cargo build | |
| # --release -p busbar` and is unrelated to the dlopen/dlclose path under test. | |
| - name: soak — 150 runs at --test-threads=8 | |
| working-directory: plugin | |
| env: | |
| BUSBAR_TEST_MYSQL_URL: mysql://busbar:busbar@127.0.0.1:3306/busbar_test | |
| RUST_BACKTRACE: full | |
| run: | | |
| set +e | |
| ulimit -c unlimited | |
| BIN=$(find target/debug/deps -maxdepth 1 -name "e2e-*" -type f -executable | head -1) | |
| echo "BIN=$BIN" | |
| CRASHES=0 | |
| for i in $(seq 1 150); do | |
| "$BIN" --test-threads=8 --skip load_and_exercise_mysql_plugin_via_file_drop >/tmp/run.log 2>&1 | |
| rc=$? | |
| if [ "$rc" != "0" ]; then | |
| CRASHES=$((CRASHES+1)) | |
| echo "=== run $i exited $rc ===" | |
| tail -20 /tmp/run.log | |
| CORE=$(ls -t /tmp/cores/core.* 2>/dev/null | head -1) | |
| if [ -n "$CORE" ] && [ "$CRASHES" -le 2 ]; then | |
| echo "=== backtrace from $CORE ===" | |
| gdb --batch -ex "bt" -ex "info sharedlibrary" -ex "thread apply all bt 6" "$BIN" "$CORE" 2>&1 | head -60 | |
| fi | |
| fi | |
| done | |
| echo "RESULT leg=${{ matrix.leg }} crashes=$CRASHES / 150" | |
| # Never fail the job on a crash: the control leg is EXPECTED to crash, and a red job would | |
| # make the interesting leg look like infrastructure breakage. The count is the result. | |
| exit 0 |