Open
Description
I run into this by doing
import awkward as ak
ak.jax.register_and_check()
array = ak.Array([[1.1], [2.2], [], [3.3], [], [], [4.4], [5.5]], backend="jax")
ak.firsts(array).to_list()when jax is installed and made to use the local cuda installation pip install --upgrade "jax[cuda12-local]" or when letting it install the nvidia packages with pip install --upgrade "jax[cuda12]". Basically jax-cuda12-pjrt and jax-cuda12-plugin need to present in the env.
(ak-uproot) ➜ awkward git:(ikrommyd/fix-jax-backend) ✗ pip list | grep jax
jax 0.5.3
jax-cuda12-pjrt 0.5.3
jax-cuda12-plugin 0.5.3
jaxlib 0.5.3
I'm pasting below my gdb trace by placing the reproducer in a script and doing gdb --args python bug.py
(ak-uproot) ➜ awkward git:(ikrommyd/fix-jax-backend) ✗ gdb --args python bug.py
GNU gdb (Ubuntu 15.0.50.20240403-0ubuntu1) 15.0.50.20240403-git
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from python...
(gdb) run
Starting program: /home/iason/miniforge3/envs/ak-uproot/bin/python bug.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff45ff6c0 (LWP 58237)]
[New Thread 0x7ffff3dfe6c0 (LWP 58238)]
[New Thread 0x7ffff13ff6c0 (LWP 58239)]
[New Thread 0x7fffee9ff6c0 (LWP 58240)]
[New Thread 0x7fffebfff6c0 (LWP 58241)]
[New Thread 0x7fffe95ff6c0 (LWP 58242)]
[New Thread 0x7fffe4bff6c0 (LWP 58243)]
[New Thread 0x7fffe43fe6c0 (LWP 58244)]
[New Thread 0x7fffdf9ff6c0 (LWP 58245)]
[New Thread 0x7fffdcfff6c0 (LWP 58246)]
[New Thread 0x7fffdc7fe6c0 (LWP 58247)]
[New Thread 0x7fffd9dff6c0 (LWP 58248)]
[New Thread 0x7fffd73ff6c0 (LWP 58249)]
[New Thread 0x7fffd49ff6c0 (LWP 58250)]
[New Thread 0x7fffcffff6c0 (LWP 58251)]
[New Thread 0x7fffc81ff6c0 (LWP 58272)]
[New Thread 0x7fffc4efd6c0 (LWP 58290)]
[New Thread 0x7fffa37ff6c0 (LWP 58291)]
[New Thread 0x7fffa2ffe6c0 (LWP 58292)]
[New Thread 0x7fffa27fd6c0 (LWP 58293)]
[New Thread 0x7fffa1ffc6c0 (LWP 58294)]
[New Thread 0x7fffa17fb6c0 (LWP 58295)]
[New Thread 0x7fffa0ffa6c0 (LWP 58296)]
[New Thread 0x7fff8bfff6c0 (LWP 58297)]
[New Thread 0x7fff8b7fe6c0 (LWP 58298)]
[New Thread 0x7fff8affd6c0 (LWP 58299)]
[New Thread 0x7fff8a7fc6c0 (LWP 58300)]
[New Thread 0x7fff89ffb6c0 (LWP 58301)]
[New Thread 0x7fff897fa6c0 (LWP 58302)]
[New Thread 0x7fff88ff96c0 (LWP 58303)]
[New Thread 0x7fff67fff6c0 (LWP 58304)]
[New Thread 0x7fff677fe6c0 (LWP 58305)]
[New Thread 0x7fff66ffd6c0 (LWP 58306)]
[New Thread 0x7fff667fc6c0 (LWP 58307)]
[New Thread 0x7fff65ffb6c0 (LWP 58308)]
[New Thread 0x7fff657fa6c0 (LWP 58309)]
[New Thread 0x7fff64ff96c0 (LWP 58310)]
[New Thread 0x7fff43fff6c0 (LWP 58311)]
[New Thread 0x7fff3bfff6c0 (LWP 58312)]
[New Thread 0x7fff437fe6c0 (LWP 58313)]
[New Thread 0x7fff42ffd6c0 (LWP 58314)]
[New Thread 0x7fff427fc6c0 (LWP 58315)]
[New Thread 0x7fff41ffb6c0 (LWP 58316)]
[New Thread 0x7fff417fa6c0 (LWP 58317)]
[New Thread 0x7fff40ff96c0 (LWP 58318)]
[New Thread 0x7fff3b7fe6c0 (LWP 58319)]
[New Thread 0x7fff3affd6c0 (LWP 58320)]
[New Thread 0x7fff3a7fc6c0 (LWP 58321)]
[New Thread 0x7fff39ffb6c0 (LWP 58322)]
[New Thread 0x7fff019ff6c0 (LWP 58326)]
[New Thread 0x7fff011fe6c0 (LWP 58327)]
[New Thread 0x7fff009fd6c0 (LWP 58328)]
[Thread 0x7fff019ff6c0 (LWP 58326) exited]
[New Thread 0x7fff019ff6c0 (LWP 58329)]
[New Thread 0x7ffefadde6c0 (LWP 58330)]
[New Thread 0x7ffefa5dd6c0 (LWP 58331)]
[New Thread 0x7ffef9ddc6c0 (LWP 58332)]
[New Thread 0x7ffef95db6c0 (LWP 58333)]
[New Thread 0x7ffef8dda6c0 (LWP 58334)]
[New Thread 0x7ffee1fff6c0 (LWP 58335)]
[New Thread 0x7ffee17fe6c0 (LWP 58336)]
[New Thread 0x7ffee0ffd6c0 (LWP 58337)]
[New Thread 0x7ffec3fff6c0 (LWP 58338)]
[New Thread 0x7ffec37fe6c0 (LWP 58339)]
[New Thread 0x7ffec2ffd6c0 (LWP 58340)]
[New Thread 0x7ffec27fc6c0 (LWP 58341)]
[New Thread 0x7ffec1ffb6c0 (LWP 58342)]
[New Thread 0x7ffec17fa6c0 (LWP 58343)]
[New Thread 0x7ffec0ff96c0 (LWP 58344)]
[New Thread 0x7ffea3fff6c0 (LWP 58345)]
[New Thread 0x7ffea37fe6c0 (LWP 58346)]
[New Thread 0x7ffea2ffd6c0 (LWP 58347)]
[New Thread 0x7ffea27fc6c0 (LWP 58348)]
[Thread 0x7ffea27fc6c0 (LWP 58348) exited]
[New Thread 0x7ffea27fc6c0 (LWP 58349)]
[New Thread 0x7ffea1ffb6c0 (LWP 58350)]
[Detaching after vfork from child process 58351]
[Detaching after vfork from child process 58352]
[Detaching after vfork from child process 58353]
[Detaching after vfork from child process 58354]
[Detaching after vfork from child process 58355]
[Detaching after vfork from child process 58356]
[Detaching after vfork from child process 58357]
[Detaching after vfork from child process 58358]
[Detaching after vfork from child process 58359]
[Detaching after vfork from child process 58360]
[Detaching after vfork from child process 58361]
[Detaching after vfork from child process 58362]
[Detaching after vfork from child process 58363]
[Detaching after vfork from child process 58364]
[Detaching after vfork from child process 58365]
[Detaching after vfork from child process 58366]
[Detaching after vfork from child process 58367]
[Detaching after vfork from child process 58368]
[Detaching after vfork from child process 58369]
[Detaching after vfork from child process 58370]
[Detaching after vfork from child process 58371]
[Detaching after vfork from child process 58372]
Thread 1 "python" received signal SIGSEGV, Segmentation fault.
0x00007ffff346dd20 in awkward_IndexedArray64_numnull ()
from /home/iason/miniforge3/envs/ak-uproot/lib/python3.12/site-packages/awkward_cpp/lib/libawkward-cpu-kernels.so
(gdb) bt
#0 0x00007ffff346dd20 in awkward_IndexedArray64_numnull ()
from /home/iason/miniforge3/envs/ak-uproot/lib/python3.12/site-packages/awkward_cpp/lib/libawkward-cpu-kernels.so
#1 0x00007ffff73c7d8a in ffi_call_unix64 () from /home/iason/miniforge3/envs/ak-uproot/lib/python3.12/lib-dynload/../../libffi.so.8
#2 0x00007ffff73c71cd in ffi_call_int () from /home/iason/miniforge3/envs/ak-uproot/lib/python3.12/lib-dynload/../../libffi.so.8
#3 0x00007ffff73c791d in ffi_call () from /home/iason/miniforge3/envs/ak-uproot/lib/python3.12/lib-dynload/../../libffi.so.8
#4 0x00007ffff7b816a3 in _call_function_pointer (argtypecount=<optimized out>, argcount=3, resmem=0x7fffffffcb70,
restype=<optimized out>, atypes=<optimized out>, avalues=<optimized out>, pProc=0x7ffff346dd20 <awkward_IndexedArray64_numnull>,
flags=<optimized out>) at /usr/local/src/conda/python-3.12.10/Modules/_ctypes/callproc.c:932
#5 _ctypes_callproc (pProc=pProc@entry=0x7ffff346dd20 <awkward_IndexedArray64_numnull>, argtuple=argtuple@entry=0x7fffc51f1200,
flags=<optimized out>, argtypes=<optimized out>, restype=<optimized out>, checker=0x0)
at /usr/local/src/conda/python-3.12.10/Modules/_ctypes/callproc.c:1274
#6 0x00007ffff7b8b241 in PyCFuncPtr_call (self=<optimized out>, inargs=<optimized out>, kwds=<optimized out>)
at /usr/local/src/conda/python-3.12.10/Modules/_ctypes/_ctypes.c:4219
#7 0x0000555555781225 in _PyObject_Call (tstate=0x555555bdbe90 <_PyRuntime+458992>, callable=0x7fffee123590, args=0x7fffc51f1200,
kwargs=<optimized out>) at /usr/local/src/conda/python-3.12.10/Objects/call.c:367
#8 0x000055555566771c in PyCFunction_Call (kwargs=0x0, args=0x7fffc51f1200, callable=0x7fffee123590)
at /usr/local/src/conda/python-3.12.10/Objects/call.c:387
#9 _PyEval_EvalFrameDefault (tstate=tstate@entry=0x555555bdbe90 <_PyRuntime+458992>, frame=frame@entry=0x7ffff7fb2330,
throwflag=throwflag@entry=0) at Python/bytecodes.c:3263
#10 0x0000555555755c4e in _PyEval_EvalFrame (throwflag=0, frame=0x7ffff7fb2330, tstate=0x555555bdbe90 <_PyRuntime+458992>)
at /usr/local/src/conda/python-3.12.10/Include/internal/pycore_ceval.h:89
#11 _PyEval_Vector (kwnames=0x0, argcount=<optimized out>, args=0x7fffffffd050, locals=0x0, func=0x7ffff0b37c40,
tstate=0x555555bdbe90 <_PyRuntime+458992>) at /usr/local/src/conda/python-3.12.10/Python/ceval.c:1685
#12 _PyFunction_Vectorcall (kwnames=0x0, nargsf=<optimized out>, stack=0x7fffffffd050, func=0x7ffff0b37c40)
at /usr/local/src/conda/python-3.12.10/Objects/call.c:419
#13 _PyObject_FastCallDictTstate (tstate=0x555555bdbe90 <_PyRuntime+458992>, callable=0x7ffff0b37c40, args=0x7fffffffd050,
nargsf=<optimized out>, kwargs=<optimized out>) at /usr/local/src/conda/python-3.12.10/Objects/call.c:133
#14 0x000055555577e7a9 in _PyObject_Call_Prepend (tstate=tstate@entry=0x555555bdbe90 <_PyRuntime+458992>,
callable=callable@entry=0x7ffff0b37c40, obj=obj@entry=0x7fffa073aa80, args=args@entry=0x7fffa0767f80, kwargs=kwargs@entry=0x0)
at /usr/local/src/conda/python-3.12.10/Objects/call.c:508
#15 0x000055555584b2fb in slot_tp_call (self=0x7fffa073aa80, args=0x7fffa0767f80, kwds=0x0)
at /usr/local/src/conda/python-3.12.10/Objects/typeobject.c:8791
#16 0x000055555575326c in _PyObject_MakeTpCall (tstate=0x555555bdbe90 <_PyRuntime+458992>, callable=0x7fffa073aa80,
args=0x7ffff7fb2308, nargs=<optimized out>, keywords=0x0) at /usr/local/src/conda/python-3.12.10/Objects/call.c:240
#17 0x0000555555666b4f in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=0x7ffff7fb2268, throwflag=<optimized out>)
at Python/bytecodes.c:2715
#18 0x00005555558044f1 in PyEval_EvalCode (co=co@entry=0x7ffff740cb20, globals=globals@entry=0x7ffff75f55c0,
locals=locals@entry=0x7ffff75f55c0) at /usr/local/src/conda/python-3.12.10/Python/ceval.c:580
#19 0x000055555583eafa in run_eval_code_obj (tstate=tstate@entry=0x555555bdbe90 <_PyRuntime+458992>, co=co@entry=0x7ffff740cb20,
globals=globals@entry=0x7ffff75f55c0, locals=locals@entry=0x7ffff75f55c0)
at /usr/local/src/conda/python-3.12.10/Python/pythonrun.c:1757
#20 0x00005555558397b5 in run_mod (mod=mod@entry=0x555555c7f968, filename=filename@entry=0x7ffff74380f0,
globals=globals@entry=0x7ffff75f55c0, locals=locals@entry=0x7ffff75f55c0, flags=flags@entry=0x7fffffffd550,
arena=arena@entry=0x7ffff751bc90) at /usr/local/src/conda/python-3.12.10/Python/pythonrun.c:1778
#21 0x0000555555836850 in pyrun_file (fp=fp@entry=0x555555c1a060, filename=filename@entry=0x7ffff74380f0, start=start@entry=257,
globals=globals@entry=0x7ffff75f55c0, locals=locals@entry=0x7ffff75f55c0, closeit=closeit@entry=1, flags=0x7fffffffd550)
at /usr/local/src/conda/python-3.12.10/Python/pythonrun.c:1674
#22 0x000055555583656e in _PyRun_SimpleFileObject (fp=0x555555c1a060, filename=0x7ffff74380f0, closeit=1, flags=0x7fffffffd550)
at /usr/local/src/conda/python-3.12.10/Python/pythonrun.c:459
#23 0x00005555558362a4 in _PyRun_AnyFileObject (fp=0x555555c1a060, filename=filename@entry=0x7ffff74380f0, closeit=closeit@entry=1,
flags=flags@entry=0x7fffffffd550) at /usr/local/src/conda/python-3.12.10/Python/pythonrun.c:78
#24 0x00005555558331c2 in pymain_run_file_obj (skip_source_first_line=0, filename=0x7ffff74380f0, program_name=0x7ffff74381b0)
at /usr/local/src/conda/python-3.12.10/Modules/main.c:361
#25 pymain_run_file (config=0x555555b7ea70 <_PyRuntime+77008>) at /usr/local/src/conda/python-3.12.10/Modules/main.c:380
#26 pymain_run_python (exitcode=0x7fffffffd524) at /usr/local/src/conda/python-3.12.10/Modules/main.c:634
#27 Py_RunMain () at /usr/local/src/conda/python-3.12.10/Modules/main.c:714
#28 0x00005555557ee5d7 in Py_BytesMain (argc=<optimized out>, argv=<optimized out>)
at /usr/local/src/conda/python-3.12.10/Modules/main.c:768
#29 0x00007ffff7c2a1ca in __libc_start_call_main (main=main@entry=0x5555557ee520 <main>, argc=argc@entry=2,
argv=argv@entry=0x7fffffffd7b8) at ../sysdeps/nptl/libc_start_call_main.h:58
#30 0x00007ffff7c2a28b in __libc_start_main_impl (main=0x5555557ee520 <main>, argc=2, argv=0x7fffffffd7b8, init=<optimized out>,
fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffd7a8) at ../csu/libc-start.c:360
#31 0x00005555557ee47d in _start ()