Skip to content

Segfault in sweep_off_heap during minor GC racing a concurrent module load (JIT, OTP 27.3.4.15) #11522

Description

@KagemniKarimu

Describe the bug

beam.smp segfaults (general protection fault) in sweep_off_heap during a minor GC while another scheduler is concurrently committing a module load (finish_loading_1erts_start_staging_code_ix). The GC'd process's off-heap chain contains garbage pointers whose values resemble x86-64 instruction bytes, suggesting overlap between process heap memory and JIT-generated code memory.

We have hit this 5 times over two weeks in CI while running a large Elixir test suite under coverage (mix test --cover). The workload characteristics:

  • ~1,300 application modules are cover-compiled (purged and reloaded) at suite start, and ExUnit loads test files during the run, so module-load traffic is constant while other schedulers execute JIT code.
  • The crash consistently lands in test files that churn large numbers of refc binaries in tight loops (thousands of generated names/UUIDs inserted into Postgres), i.e. the processes whose minor GCs sweep the longest off-heap chains.
  • It only reproduces on freshly provisioned VMs (GitHub Actions ubuntu-latest runners): about 1 in 20 cold runs crash, while 80 consecutive repeat runs on already-warmed VMs (same commit, same seed available) produced zero crashes.

This looks like a sibling of (or an incomplete fix for) OTP-19599 ("rare bug that could cause an emulator crash after unloading a module or erasing a persistent_term") — the version we run already contains that fix.

Backtrace

Core captured with matching binaries; gdb run on the same machine.

dmesg:

traps: erts_sched_1[3466] general protection fault ip:55b632c35c54 sp:7ff242da1840 error:0 in beam.smp[435c54,55b632c00000+158000]

Crashing scheduler:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  sweep_off_heap (p=p@entry=0x7ff236ab9f78, fullsweep=fullsweep@entry=0) at beam/erl_gc.c:3182
        refc_binary = 0x81a48c0fceff413b
        prev = 0x7ff1db6427c0
        insert_old_here = 0x7ff236aba180
        oheap = 0x7ff1db404fb8 ","
        oheap_sz = 54176
        bin_vheap = 32
        shrink_ncandidates = 0
        shrink_nactive = 1
        shrink_unresolved_end = 0x7ff1db6427b0
        br = 0x8948fe4f0f485f5c
#1  0x000055b632c381ae in do_minor (p=p@entry=0x7ff236ab9f78, live_hf_end=live_hf_end@entry=0xfffffffffffffff8, mature=<optimized out>, mature_size=mature_size@entry=3072, new_sz=4185, objv=objv@entry=0x7ff242da1d40, nobj=1) at beam/erl_gc.c:1825
#2  0x000055b632c3b55a in minor_collection (recl=<synthetic pointer>, ygen_usage=2539, nobj=1, objv=0x7ff242da1d40, need=25, live_hf_end=0xfffffffffffffff8, p=0x7ff236ab9f78) at beam/erl_gc.c:1509
#3  garbage_collect (p=p@entry=0x7ff236ab9f78, live_hf_end=live_hf_end@entry=0xfffffffffffffff8, need=need@entry=21, objv=objv@entry=0x7ff242da1d40, nobj=nobj@entry=1, fcalls=fcalls@entry=3526, max_young_gen_usage=0) at beam/erl_gc.c:803
#4  0x000055b632c3c590 in erts_garbage_collect_nobump (p=0x7ff236ab9f78, need=21, objv=0x7ff242da1d40, nobj=1, fcalls=3526) at beam/erl_gc.c:942
#5  0x00007ff23bbffafd in ?? ()        <- JIT-generated code (anonymous executable mapping)
#6  0x0000000000000000 in ?? ()

Note the values of refc_binary (0x81a48c0fceff413b) and br (0x8948fe4f0f485f5c): both decode as plausible x86-64 instruction byte sequences rather than heap addresses.

Scheduler concurrently committing a module load at the moment of the crash:

Thread 18 (Thread 0x7ff237bfe6c0 (LWP 3467)):
#0  0x000055b632c2b804 in hash_put (h=h@entry=0x55b63324e940 <export_tables+128>, tmpl=0x7ff1ef2a4be8) at beam/hash.c:242
#1  0x000055b632c2bd5d in index_put_entry (t=t@entry=0x55b63324e940 <export_tables+128>, tmpl=<optimized out>) at beam/index.c:76
#2  0x000055b632c2e1ad in export_start_staging () at beam/export.c:520
#3  0x000055b632b06f8e in erts_start_staging_code_ix (num_new=num_new@entry=1) at beam/code_ix.c:104
#4  0x000055b632aeaf8f in finish_loading_1 (A__p=0x55b637a69918, BIF__ARGS=<optimized out>, A__I=<optimized out>) at beam/beam_bif_load.c:418
#5  0x00007ff23bbff63c in ?? ()
#6  0x0000000000000000 in ?? ()

All other threads are idle (poll/cond waits) or in unrelated syscalls.

To Reproduce

No standalone reproducer yet — statistics from our CI:

  • mix test --partitions 4 --cover --warnings-as-errors over a ~6,600-test Elixir suite; the crashing partition takes ~5 minutes.
  • ~5% of runs crash on a freshly provisioned GitHub Actions VM; 0 of 80 repeat runs crashed on warmed VMs (same commit; one run replayed the exact seed of a previous crash without reproducing, so it is a timing race, not input-determined).
  • Loaded NIFs (crypto, asn1, a Rustler NIF) were idle at crash time in the captured core; earlier crashes happened in test code with no NIF involvement on the crashing process.

Expected behavior

No emulator crash.

Affected versions

  • Erlang/OTP 27.3.4.15 (erts 15.2.7.11), x86_64-unknown-linux-gnu, JIT flavor, from the erlef/setup-beam builds. Elixir 1.19.5.
  • Observed on Ubuntu 24.04 (GitHub ubuntu-latest) runners.
  • First observed 2026-08-08; we cannot say whether older versions are affected (the suite/workload is recent).

Additional context

We kept the full 800 MB core file, the exact beam.smp, and the loaded .so files, and can share them privately or run further gdb/etp-macro analysis on the core on request.

Metadata

Metadata

Assignees

No one assigned

    Labels

    team:VMAssigned to OTP team VM

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions