NumPy crash that only happens with uv's Python 3.14 on Linux

[ngoldbaum]

See numpy/numpy#30899 (comment) for full context.

If I set up a new Python environment with uv's Python build and run a NumPy script that triggers NumPy's temporary elision codepaths, I get a segfault inside of the backtrace C function provided by libunwind (although the segfault is in libgcc).

uv venv .venv314 --python 3.14
source .venv314/bin/activate
uv pip install numpy
python test.py

Where test.py has the following content:

import numpy as np
d = np.random.rand(512, 512).astype(np.float32)
condition = False
condition |= d == np.float32(65534.0)

The traceback from gdb:

(gdb) r test.py
Starting program: /home/goldbaum/.venv314/bin/python test.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff78cba10 in ?? () from /lib/x86_64-linux-gnu/libgcc_s.so.1
(gdb) bt
#0  0x00007ffff78cba10 in ?? () from /lib/x86_64-linux-gnu/libgcc_s.so.1
#1  0x00007ffff78cd91e in _Unwind_Backtrace () from /lib/x86_64-linux-gnu/libgcc_s.so.1
#2  0x00007ffff7d35eb3 in __GI___backtrace (array=<optimized out>, size=10) at ./debug/backtrace.c:78
#3  0x00007ffff6d39948 in check_callers (cannot=0x7fffffffcf54) at ../numpy/_core/src/multiarray/temp_elide.c:171
#4  0x00007ffff6d3a2b5 in can_elide_temp (olhs=0x7ffff628a1f0, orhs=0x152b7b8 <_Py_FalseStruct>, cannot=0x7fffffffcf54)
    at ../numpy/_core/src/multiarray/temp_elide.c:347
#5  0x00007ffff6d3a3a4 in try_binary_elide (m1=0x152b7b8 <_Py_FalseStruct>, m2=0x7ffff628a1f0,
    inplace_op=0x7ffff6d23cc8 <array_inplace_bitwise_or>, res=0x7fffffffcf88, commutative=1)
    at ../numpy/_core/src/multiarray/temp_elide.c:373
#6  0x00007ffff6d23568 in array_bitwise_or (m1=0x152b7b8 <_Py_FalseStruct>, m2=0x7ffff628a1f0)
    at ../numpy/_core/src/multiarray/number.c:491
#7  0x0000000001ada90f in PyNumber_InPlaceOr.warm ()
#8  0x0000000000000014 in ?? ()
#9  0x0000000000000014 in ?? ()
#10 0x00007ffff7e16020 in ?? ()
#11 0x000000000152b7b9 in _Py_FalseStruct ()
#12 0x00007ffff7a6672e in ?? ()
#13 0x0000000001801394 in _TAIL_CALL_BINARY_OP.llvm.10362591704542007599 ()
#14 0x00000000018890ed in _PyEval_Vector ()
#15 0x0000000001888de7 in PyEval_EvalCode ()
#16 0x00000000018e4c73 in run_mod.llvm ()
#17 0x0000000001a3b9c7 in pyrun_file ()
#18 0x0000000001a3979b in _PyRun_SimpleFileObject ()
#19 0x0000000001a39682 in _PyRun_AnyFileObject ()
#20 0x0000000001a391f6 in pymain_run_file_obj ()
#21 0x0000000001a390f4 in pymain_run_file ()
#22 0x00000000019b0f91 in Py_RunMain ()
#23 0x00000000019ae9c3 in pymain_main.llvm ()
#24 0x00000000019ae7ad in main ()

See the linked NumPy issue for some vague speculation from me about why this is happening.

[ngoldbaum]

NumPy's temporary elision code has a storied history. It does lead to real performance benefits for a lot of uses of NumPy. And in practice it doesn't cause any issues. Except when it does, like in this case.

The basic idea is that if NumPy is able to determine that an array operation would use a temporary array object created only for the operation, NumPy instead elides the temporary and does the operation in-place.

The trouble is that it's only safe to do this if the object is a temporary created by the Python interpreter for Python code. It's not safe to do it for an operation that uses a temporary in extension code. And it's impossible to tell the difference from inside NumPy.

So, NumPy cheats. For really big operations above some threshold size, NumPy scans the C call stack to evaluate whether frames above the operation are in NumPy itself or in CPython. If it is, it has to be a temporary and it's safe to elide. If it isn't, the temporary must be created by extensions outside of NumPy or CPython, so we don't elide.

I'd really rather not have this silly C call stack scanning thing in such a load-bearing spot to enable runtime optimizations, but that ship sailed more than a decade ago unfortunately.

[ngoldbaum]

A friend who works on gcc managed to reproduce this with a debug build of gcc, so I know where this is crashing in libgcc now:

numpy/numpy#30899 (comment)

I'd really appreciate some input from a maintainer about what to do here. Should this be fixed in python-build-standalone, or should NumPy somehow detect uv's Python and disable this code path? I'd prefer not to do that but I also don't want users to hit segfaults.

[jjhelmus]

I am able to replicate this with CPython 3.14 from python-build-standalone. There is no segfault when running in a 3.13 environment. I'm suspicious if this is caused by 3.14 being built as a tail calling interpreter. I'm going to do some additional tests to debug this issue.

[zanieb]

cc @Fidget-Spinner

[jjhelmus]

I build a few different variations of CPython 3.14 locally and tested in with the reproducer.

The segfault occurs when CPython is built with BOLT enabled. Not sure why 3.13 is not effected as that too is built with BOLT.

[colesbury]

A few other data points, in case their helpful:

• The NumPy crash doesn't repro with 3.14t, just 3.14. They are built with most of the same configuration options except--enable-experimental-jit=yes-off vs. --disable-gil, but I don't think it's the JIT.
• I've noticed previously that gdb shows corrupted frames for 3.14, but not 3.14t, which fits with backtrace() crashing.

Personally, I'm suspicious of BOLT, maybe in combination with other options. I tried repro'ing the crash with CPython built from source, but ran into llvm/llvm-project#163263. How are you successfully building with BOLT?

[jjhelmus]

c.f. #934 for another difference in behavior in NumPy when using python-build-standalone. In this case a RuntimeWarning when using frompyfunc

[jjhelmus]

How are you successfully building with BOLT?

The project uses LLVM 21 and applies some patches to CPython in order to get BOLT working. I'm suspicious if some of these patches are incomplete.

[jjhelmus]

Of the patches applied related to BOLT:

• patch-configure-bolt-skip-funcs.patch is needed for the built to succeed.
  • With just this patch there is no segfault.
  • The test code from Heinsenbug seen against numpy + python-build-standalone builds on Linux #934 give a runtime warning.
• patch-configure-bolt-apply-flags-128514.patch
  • With this an the above patch there is a segfault.
• patch-configure-bolt-icf-safe.patch
  • Need above patches to apply + build, have not tested completely.

[ngoldbaum]

In case it helps, my friend with a libgcc debug build did a little more debugging and found the crash is happening in x86_64_fallback_frame_state: https://gcc.gnu.org/git/?p=gcc.git;a=blob;f=libgcc/config/i386/linux-unwind.h;h=1018c22ee1e3de4b96bed39f6ba625745b64f5f4;hb=a40655524e12497378baeb3cc9fb4de12da8ca08#l44. This function is inlined into uw_frame_state_for, which is why it doesn't show up in the gdb backtrace. He also finds that it happens on the 6th iteration of the for loop.

Unfortunately it's not straightforward to see where exactly the crash is happening due to inlining in their build.

[jjhelmus]

The segfault is removed if the strategy used to partition blocks into fragments is set to only consider hot and cold fragments rather than hot, warn and cold (--split-strategy=profile2/cdsplit).

I'm not certain if with the cdsplit strategy if BOLT is producing invalid code paths or if NumPy's elision is responsible.

[ngoldbaum]

if NumPy's elision is responsible.

NumPy is calling the backtrace function. I'd be surprised if that was the problem. However, if BOLT is somehow breaking frame unwinding in the CPython build then this problem is something I could see happening, since CPython shows up in the stack trace requested by NumPy.

[jjhelmus]

I'm seeing different frames when breaking on the check_callers function depending on the BOLT split-strategy

Using profile2:

0  0x00007ffff6a29550 in check_callers () from /home/jhelmus/debugging/pbs/991_numpy/venv_3_1_2_profile2/lib/python3.14/site-packages/numpy/_core/_multiarray_umath.cpython-314-x86_64-linux-gnu.so
#1  0x00007ffff6a29cac in try_binary_elide ()
   from /home/jhelmus/debugging/pbs/991_numpy/venv_3_1_2_profile2/lib/python3.14/site-packages/numpy/_core/_multiarray_umath.cpython-314-x86_64-linux-gnu.so
#2  0x00007ffff6a1108e in array_bitwise_or ()
   from /home/jhelmus/debugging/pbs/991_numpy/venv_3_1_2_profile2/lib/python3.14/site-packages/numpy/_core/_multiarray_umath.cpython-314-x86_64-linux-gnu.so
#3  0x0000000001864026 in binary_op1 (v=0x152f7b8 <_Py_FalseStruct>, w=0x7fffcc9b4990, op_slot=120) at Objects/abstract.c:972
#4  binary_iop1 (v=0x152f7b8 <_Py_FalseStruct>, w=0x7fffcc9b4990, iop_slot=224, op_slot=120) at Objects/abstract.c:1237
#5  binary_iop (v=0x152f7b8 <_Py_FalseStruct>, w=0x7fffcc9b4990, iop_slot=224, op_slot=120, op_name=<optimized out>) at Objects/abstract.c:1253
#6  PyNumber_InPlaceOr (v=0x152f7b8 <_Py_FalseStruct>, w=0x7fffcc9b4990) at Objects/abstract.c:1285
#7  0x0000000001801c94 in _TAIL_CALL_BINARY_OP (frame=0x7ffff7fb0020, stack_pointer=0x7ffff7fb0080, tstate=0x15e3928 <_PyRuntime+315648>, next_instr=0x7ffff74860ee, oparg=20)
    at Python/generated_cases.c.h:62
#8  0x00000000018d0c63 in _TAIL_CALL_start_frame (frame=0x7ffff7fb0020, stack_pointer=0x7ffff7fb0080, tstate=0x152f7b9 <_Py_FalseStruct+1>, next_instr=0x7ffff74860ee, oparg=-10820)
    at Python/generated_cases.c.h:12524

Using cdsplit

#0  check_callers (cannot=0x7fffffffd5f4) at ../numpy/_core/src/multiarray/temp_elide.c:132
#1  0x00007ffff673a2b5 in can_elide_temp (olhs=0x7ffff5c85dd0, orhs=0x152f7b8 <_Py_FalseStruct>, cannot=0x7fffffffd5f4) at ../numpy/_core/src/multiarray/temp_elide.c:347
#2  0x00007ffff673a3a4 in try_binary_elide (m1=0x152f7b8 <_Py_FalseStruct>, m2=0x7ffff5c85dd0, inplace_op=0x7ffff6723cc8 <array_inplace_bitwise_or>, res=0x7fffffffd628, commutative=1)
    at ../numpy/_core/src/multiarray/temp_elide.c:373
#3  0x00007ffff6723568 in array_bitwise_or (m1=0x152f7b8 <_Py_FalseStruct>, m2=0x7ffff5c85dd0) at ../numpy/_core/src/multiarray/number.c:491
#4  0x0000000001ada827 in binary_op1 (v=0x152f7b8 <_Py_FalseStruct>, w=0x7ffff5c85dd0, op_slot=120) at Objects/abstract.c:972
#5  binary_iop1 (v=0x152f7b8 <_Py_FalseStruct>, w=0x7ffff5c85dd0, iop_slot=224, op_slot=120) at Objects/abstract.c:1237
#6  binary_iop (v=0x152f7b8 <_Py_FalseStruct>, w=0x7ffff5c85dd0, iop_slot=224, op_slot=120, op_name=<optimized out>) at Objects/abstract.c:1253
#7  PyNumber_InPlaceOr (v=0x152f7b8 <_Py_FalseStruct>, w=0x7ffff5c85dd0) at Objects/abstract.c:1285
#8  0x0000000000000014 in ?? ()
#9  0x0000000000000014 in ?? ()
#10 0x00007ffff7fb0020 in ?? ()
#11 0x000000000152f7b9 in _Py_FalseStruct ()
#12 0x00007ffff74860ee in ?? ()
#13 0x0000000001801a2c in _TAIL_CALL_BINARY_OP (frame=0xe14b7dc6ec5d6a00, stack_pointer=0x7ffff7fb0080, tstate=0x15e3928 <_PyRuntime+315648>, next_instr=0x1ada827 <PyNumber_InPlaceOr+2624935>,
    oparg=-10640) at Python/generated_cases.c.h:62
#14 0x00000000018970a3 in _TAIL_CALL_start_frame (frame=0xe14b7dc6ec5d6a00, stack_pointer=0x7ffff7fb0080, tstate=0x7fffffffd6b0, next_instr=0x1ada827 <PyNumber_InPlaceOr+2624935>, oparg=-10764)
    at Python/generated_cases.c.h:12524

Frames 8-12 looks suspicious.