We can’t get a core dump due to “resource limits”:

Sep 09 10:40:02 jupiter kernel: CPU 1/KVM[1545351]: segfault at 14 ip 000056473f97ca24 sp 00007f550f076eb0 error 4 in .qemu-system-x86_64-wrapped[56473f771000+652000] likely on CPU 20 (core 4, socket 0)
Sep 09 10:40:02 jupiter kernel: Code: 48 83 c4 58 89 d8 5b 5d 41 5c 41 5d 41 5e 41 5f c3 0f 1f 44 00 00 89 f0 48 8b 8d 40 83 00 00 4c 8d 04 40 49 c1 e0 03 4c 01 c1 <8b> 41 14 85 c0 0f 84 11 01 00 00 83 c0 01 89 41 14 41 80 bd d1 01
Sep 09 10:40:02 jupiter systemd-coredump[1545634]: Resource limits disable core dumping for process 1545297 (.qemu-system-x8).
Sep 09 10:40:02 jupiter systemd-coredump[1545634]: [🡕] Process 1545297 (.qemu-system-x8) of user 301 terminated abnormally without generating a coredump.

This is true even if ulimit and systemd-coredump are configured as follows:

$ ulimit -a | rg core
-c: core file size (blocks)         unlimited

systemd.coredump.extraConfig = ''
  Storage=external
  Compress=yes
  ProcessSizeMax=200G
  ExternalSizeMax=200G
  JournalSizeMax=0
  MaxUse=0
  KeepFree=0
'';

Per this blog post, this is because the qemu process had its limit set to zero:

$ rg core /proc/$(pidof qemu-system-x86_64)/limits
6:Max core file size        0                    0                    bytes
% sudo prlimit --core=-1 -p $(pidof qemu-system-x86_64)
[sudo] password for delan:
$ rg core /proc/$(pidof qemu-system-x86_64)/limits
6:Max core file size        unlimited            unlimited            bytes

We get the core dump summary below. It’s for process 1545846, but it’s not clear to me which thread crashed:

Sep 09 10:40:22 jupiter systemd-coredump[1546111]: [🡕] Process 1545846 (.qemu-system-x8) of user 301 dumped core.

                                                   Module libdigestmd5.so.3 without build-id.
                                                   Module libgs2.so.3 without build-id.
                                                   Module liblogin.so.3 without build-id.
                                                   Module libdb-5.3.so without build-id.
                                                   Module libsasldb.so.3 without build-id.
                                                   Module libcrammd5.so.3 without build-id.
                                                   Module libgssapiv2.so.3 without build-id.
                                                   Module libcrypt.so.2 without build-id.
                                                   Module libplain.so.3 without build-id.
                                                   Module libscram.so.3 without build-id.
                                                   Module libanonymous.so.3 without build-id.
                                                   Module libotp.so.3 without build-id.
                                                   Module libmp3lame.so.0 without build-id.
                                                   Module libmpg123.so.0 without build-id.
                                                   Module libogg.so.0 without build-id.
                                                   Module libvorbisenc.so.2 without build-id.
                                                   Module libvorbis.so.0 without build-id.
                                                   Module libFLAC.so.12 without build-id.
                                                   Module libgpg-error.so.0 without build-id.
                                                   Module libdatrie.so.1 without build-id.
                                                   Module libbrotlicommon.so.1 without build-id.
                                                   Module libkeyutils.so.1 without build-id.
                                                   Module libkrb5support.so.0 without build-id.
                                                   Module libcom_err.so.3 without build-id.
                                                   Module libk5crypto.so.3 without build-id.
                                                   Module libkrb5.so.3 without build-id.
                                                   Module libsndfile.so.1 without build-id.
                                                   Module libvulkan.so.1 without build-id.
                                                   Module libsharpyuv.so.0 without build-id.
                                                   Module libdeflate.so.0 without build-id.
                                                   Module libLerc.so.4 without build-id.
                                                   Module liblzma.so.5 without build-id.
                                                   Module libgcrypt.so.20 without build-id.
                                                   Module libicudata.so.73 without build-id.
                                                   Module libxml2.so.2 without build-id.
                                                   Module libjson-glib-1.0.so.0 without build-id.
                                                   Module libgraphite2.so.3 without build-id.
                                                   Module libthai.so.0 without build-id.
                                                   Module libXdmcp.so.6 without build-id.
                                                   Module libXau.so.6 without build-id.
                                                   Module libgstbase-1.0.so.0 without build-id.
                                                   Module libunwind.so.8 without build-id.
                                                   Module libbrotlidec.so.1 without build-id.
                                                   Module libgssapi_krb5.so.2 without build-id.
                                                   Module libpsl.so.5 without build-id.
                                                   Module libssh2.so.1 without build-id.
                                                   Module libnghttp2.so.14 without build-id.
                                                   Module libsamplerate.so.0 without build-id.
                                                   Module libcelt0.so.2 without build-id.
                                                   Module libpulsecommon-17.0.so without build-id.
                                                   Module libnspr4.so without build-id.
                                                   Module libplc4.so without build-id.
                                                   Module libplds4.so without build-id.
                                                   Module libnssutil3.so without build-id.
                                                   Module libnss3.so without build-id.
                                                   Module libsmime3.so without build-id.
                                                   Module libssl3.so without build-id.
                                                   Module libgfxstream_backend.so without build-id.
                                                   Module libva-drm.so.2 without build-id.
                                                   Module libva.so.2 without build-id.
                                                   Module libwebp.so.7 without build-id.
                                                   Module libwebpdemux.so.2 without build-id.
                                                   Module libtiff.so.6 without build-id.
                                                   Module libsystemd.so.0 without build-id.
                                                   Module libicuuc.so.73 without build-id.
                                                   Module libxcb-shm.so.0 without build-id.
                                                   Module libxcb-render.so.0 without build-id.
                                                   Module libxcb.so.1 without build-id.
                                                   Module libXrender.so.1 without build-id.
                                                   Module libfreetype.so.6 without build-id.
                                                   Module libXinerama.so.1 without build-id.
                                                   Module libXcomposite.so.1 without build-id.
                                                   Module libXdamage.so.1 without build-id.
                                                   Module libxkbcommon.so.0 without build-id.
                                                   Module libtracker-sparql-3.0.so.0 without build-id.
                                                   Module libfribidi.so.0 without build-id.
                                                   Module libfontconfig.so.1 without build-id.
                                                   Module libpangoft2-1.0.so.0 without build-id.
                                                   Module libharfbuzz.so.0 without build-id.
                                                   Module libpango-1.0.so.0 without build-id.
                                                   Module libpangocairo-1.0.so.0 without build-id.
                                                   Module libxcb-randr.so.0 without build-id.
                                                   Module libexpat.so.1 without build-id.
                                                   Module libdrm.so.2 without build-id.
                                                   Module libpcre2-8.so.0 without build-id.
                                                   Module libffi.so.8 without build-id.
                                                   Module libselinux.so.1 without build-id.
                                                   Module libatomic.so.1 without build-id.
                                                   Module libXss.so.1 without build-id.
                                                   Module libXrandr.so.2 without build-id.
                                                   Module libXfixes.so.3 without build-id.
                                                   Module libXi.so.6 without build-id.
                                                   Module libXcursor.so.1 without build-id.
                                                   Module libXext.so.6 without build-id.
                                                   Module libgcc_s.so.1 without build-id.
                                                   Module libcap.so.2 without build-id.
                                                   Module libgmp.so.10 without build-id.
                                                   Module libhogweed.so.6 without build-id.
                                                   Module libnettle.so.8 without build-id.
                                                   Module libtasn1.so.6 without build-id.
                                                   Module libunistring.so.5 without build-id.
                                                   Module libidn2.so.0 without build-id.
                                                   Module libp11-kit.so.0 without build-id.
                                                   Module libstdc++.so.6 without build-id.
                                                   Module libsasl2.so.3 without build-id.
                                                   Module liblz4.so.1 without build-id.
                                                   Module liborc-0.4.so.0 without build-id.
                                                   Module libgstapp-1.0.so.0 without build-id.
                                                   Module libgstreamer-1.0.so.0 without build-id.
                                                   Module libopus.so.0 without build-id.
                                                   Module libbz2.so.1 without build-id.
                                                   Module libaio.so.1 without build-id.
                                                   Module libiscsi.so.10 without build-id.
                                                   Module libpipewire-0.3.so.0 without build-id.
                                                   Module libjack.so.0 without build-id.
                                                   Module libpulse.so.0 without build-id.
                                                   Module libasound.so.2 without build-id.
                                                   Module libusbredirparser.so.1 without build-id.
                                                   Module libcacard.so.0 without build-id.
                                                   Module librutabaga_gfx_ffi.so.0 without build-id.
                                                   Module libSDL2_image-2.0.so.0 without build-id.
                                                   Module libX11.so.6 without build-id.
                                                   Module libvte-2.91.so.0 without build-id.
                                                   Module libncursesw.so.6 without build-id.
                                                   Module libvdeplug.so.3 without build-id.
                                                   Module libzstd.so.1 without build-id.
                                                   Module libnuma.so.1 without build-id.
                                                   Module libseccomp.so.2 without build-id.
                                                   Module libSDL2-2.0.so.0 without build-id.
                                                   Module liblzo2.so.2 without build-id.
                                                   Module libsnappy.so.1 without build-id.
                                                   Module libusb-1.0.so.0 without build-id.
                                                   Module libudev.so.1 without build-id.
                                                   Module libfdt.so.1 without build-id.
                                                   Module libjpeg.so.62 without build-id.
                                                   Module libz.so.1 without build-id.
                                                   Module libpng16.so.16 without build-id.
                                                   Module libgnutls.so.30 without build-id.
                                                   Module libspice-server.so.1 without build-id.
                                                   Module libcapstone.so.5 without build-id.
                                                   Module libepoxy.so.0 without build-id.
                                                   Stack trace of thread 1545895:
                                                   #0  0x0000556bf0e25a24 kvm_virtio_pci_vector_use_one (.qemu-system-x86_64-wrapped + 0x536a24)
                                                   #1  0x0000556bf10183e0 memory_region_write_accessor (.qemu-system-x86_64-wrapped + 0x7293e0)
                                                   #2  0x0000556bf1017ca6 access_with_adjusted_size (.qemu-system-x86_64-wrapped + 0x728ca6)
                                                   #3  0x0000556bf1017fd1 memory_region_dispatch_write (.qemu-system-x86_64-wrapped + 0x728fd1)
                                                   #4  0x0000556bf101fa5c flatview_write_continue (.qemu-system-x86_64-wrapped + 0x730a5c)
                                                   #5  0x0000556bf101fcd3 flatview_write (.qemu-system-x86_64-wrapped + 0x730cd3)
                                                   #6  0x0000556bf1022fe9 address_space_rw (.qemu-system-x86_64-wrapped + 0x733fe9)
                                                   #7  0x0000556bf1077498 kvm_cpu_exec (.qemu-system-x86_64-wrapped + 0x788498)
                                                   #8  0x0000556bf10786e5 kvm_vcpu_thread_fn (.qemu-system-x86_64-wrapped + 0x7896e5)
                                                   #9  0x0000556bf120af58 qemu_thread_start (.qemu-system-x86_64-wrapped + 0x91bf58)
                                                   #10 0x00007fd370a85272 start_thread (libc.so.6 + 0x8e272)
                                                   #11 0x00007fd370b00c24 __clone (libc.so.6 + 0x109c24)

                                                   Stack trace of thread 1545900:
                                                   #0  0x00007fd370b00a8d writev (libc.so.6 + 0x109a8d)
                                                   #1  0x00007fd372a752ab _ZL16stream_writev_cbP9RedStreamPK5ioveci (libspice-server.so.1 + 0x4f2ab)
                                                   #2  0x00007fd372a62320 _ZN16RedChannelClient15handle_outgoingEv (libspice-server.so.1 + 0x3c320)
                                                   #3  0x00007fd372a62497 _ZN16RedChannelClient4sendEv (libspice-server.so.1 + 0x3c497)
                                                   #4  0x00007fd372a62723 _ZN16RedChannelClient4pushEv (libspice-server.so.1 + 0x3c723)
                                                   #5  0x00007fd372a628f2 _ZN16RedChannelClient14handle_messageEtjPv (libspice-server.so.1 + 0x3c8f2)
                                                   #6  0x00007fd372a61739 _ZN16RedChannelClient15handle_incomingEv (libspice-server.so.1 + 0x3b739)
                                                   #7  0x00007fd372a6283d _ZL24red_channel_client_eventiiP16RedChannelClient (libspice-server.so.1 + 0x3c83d)
                                                   #8  0x00007fd372a4d39d spice_watch_dispatch (libspice-server.so.1 + 0x2739d)
                                                   #9  0x00007fd372054e39 g_main_dispatch (libglib-2.0.so.0 + 0x5be39)
                                                   #10 0x00007fd372057fd7 g_main_context_iterate_unlocked.isra.0 (libglib-2.0.so.0 + 0x5efd7)
                                                   #11 0x00007fd37205888f g_main_loop_run (libglib-2.0.so.0 + 0x5f88f)
                                                   #12 0x00007fd372a77239 _ZL15red_worker_mainPv (libspice-server.so.1 + 0x51239)
                                                   #13 0x00007fd370a85272 start_thread (libc.so.6 + 0x8e272)
                                                   #14 0x00007fd370b00c24 __clone (libc.so.6 + 0x109c24)

                                                   Stack trace of thread 1545894:
                                                   #0  0x00007fd370afc9cf ioctl (libc.so.6 + 0x1059cf)
                                                   #1  0x0000556bf1076d8f kvm_vcpu_ioctl (.qemu-system-x86_64-wrapped + 0x787d8f)
                                                   #2  0x0000556bf1077280 kvm_cpu_exec (.qemu-system-x86_64-wrapped + 0x788280)
                                                   #3  0x0000556bf10786e5 kvm_vcpu_thread_fn (.qemu-system-x86_64-wrapped + 0x7896e5)
                                                   #4  0x0000556bf120af58 qemu_thread_start (.qemu-system-x86_64-wrapped + 0x91bf58)
                                                   #5  0x00007fd370a85272 start_thread (libc.so.6 + 0x8e272)
                                                   #6  0x00007fd370b00c24 __clone (libc.so.6 + 0x109c24)

                                                   Stack trace of thread 1545897:
                                                   #0  0x00007fd370afc9cf ioctl (libc.so.6 + 0x1059cf)
                                                   #1  0x0000556bf1076d8f kvm_vcpu_ioctl (.qemu-system-x86_64-wrapped + 0x787d8f)
                                                   #2  0x0000556bf1077280 kvm_cpu_exec (.qemu-system-x86_64-wrapped + 0x788280)
                                                   #3  0x0000556bf10786e5 kvm_vcpu_thread_fn (.qemu-system-x86_64-wrapped + 0x7896e5)
                                                   #4  0x0000556bf120af58 qemu_thread_start (.qemu-system-x86_64-wrapped + 0x91bf58)
                                                   #5  0x00007fd370a85272 start_thread (libc.so.6 + 0x8e272)
                                                   #6  0x00007fd370b00c24 __clone (libc.so.6 + 0x109c24)

                                                   Stack trace of thread 1545896:
                                                   #0  0x00007fd370afc9cf ioctl (libc.so.6 + 0x1059cf)
                                                   #1  0x0000556bf1076d8f kvm_vcpu_ioctl (.qemu-system-x86_64-wrapped + 0x787d8f)
                                                   #2  0x0000556bf1077280 kvm_cpu_exec (.qemu-system-x86_64-wrapped + 0x788280)
                                                   #3  0x0000556bf10786e5 kvm_vcpu_thread_fn (.qemu-system-x86_64-wrapped + 0x7896e5)
                                                   #4  0x0000556bf120af58 qemu_thread_start (.qemu-system-x86_64-wrapped + 0x91bf58)
                                                   #5  0x00007fd370a85272 start_thread (libc.so.6 + 0x8e272)
                                                   #6  0x00007fd370b00c24 __clone (libc.so.6 + 0x109c24)

                                                   Stack trace of thread 1545846:
                                                   #0  0x00007fd370af3b86 ppoll (libc.so.6 + 0xfcb86)
                                                   #1  0x0000556bf1220e1e qemu_poll_ns (.qemu-system-x86_64-wrapped + 0x931e1e)
                                                   #2  0x0000556bf121e71e main_loop_wait (.qemu-system-x86_64-wrapped + 0x92f71e)
                                                   #3  0x0000556bf0e64133 qemu_main_loop (.qemu-system-x86_64-wrapped + 0x575133)
                                                   #4  0x0000556bf1081456 qemu_default_main (.qemu-system-x86_64-wrapped + 0x792456)
                                                   #5  0x00007fd370a2110e __libc_start_call_main (libc.so.6 + 0x2a10e)
                                                   #6  0x00007fd370a211c9 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2a1c9)
                                                   #7  0x0000556bf0c26625 _start (.qemu-system-x86_64-wrapped + 0x337625)

                                                   Stack trace of thread 1545893:
                                                   #0  0x00007fd370af362f __poll (libc.so.6 + 0xfc62f)
                                                   #1  0x00007fd372057f2f g_main_context_iterate_unlocked.isra.0 (libglib-2.0.so.0 + 0x5ef2f)
                                                   #2  0x00007fd37205888f g_main_loop_run (libglib-2.0.so.0 + 0x5f88f)
                                                   #3  0x0000556bf10bb909 iothread_run (.qemu-system-x86_64-wrapped + 0x7cc909)
                                                   #4  0x0000556bf120af58 qemu_thread_start (.qemu-system-x86_64-wrapped + 0x91bf58)
                                                   #5  0x00007fd370a85272 start_thread (libc.so.6 + 0x8e272)
                                                   #6  0x00007fd370b00c24 __clone (libc.so.6 + 0x109c24)

                                                   Stack trace of thread 1545869:
                                                   #0  0x00007fd370acd0c5 clock_nanosleep@GLIBC_2.2.5 (libc.so.6 + 0xd60c5)
                                                   #1  0x00007fd370ad7da7 __nanosleep (libc.so.6 + 0xe0da7)
                                                   #2  0x00007fd372086e8f g_usleep (libglib-2.0.so.0 + 0x8de8f)
                                                   #3  0x0000556bf1214eec call_rcu_thread (.qemu-system-x86_64-wrapped + 0x925eec)
                                                   #4  0x0000556bf120af58 qemu_thread_start (.qemu-system-x86_64-wrapped + 0x91bf58)
                                                   #5  0x00007fd370a85272 start_thread (libc.so.6 + 0x8e272)
                                                   #6  0x00007fd370b00dec __clone3 (libc.so.6 + 0x109dec)

                                                   Stack trace of thread 1545870:
                                                   #0  0x00007fd370a81c5e __futex_abstimed_wait_common (libc.so.6 + 0x8ac5e)
                                                   #1  0x00007fd370a847d5 pthread_cond_timedwait@@GLIBC_2.3.2 (libc.so.6 + 0x8d7d5)
                                                   #2  0x0000556bf120b0e1 qemu_cond_timedwait_ts (.qemu-system-x86_64-wrapped + 0x91c0e1)
                                                   #3  0x0000556bf120bc80 qemu_cond_timedwait_impl (.qemu-system-x86_64-wrapped + 0x91cc80)
                                                   #4  0x0000556bf12200e4 worker_thread (.qemu-system-x86_64-wrapped + 0x9310e4)
                                                   #5  0x0000556bf120af58 qemu_thread_start (.qemu-system-x86_64-wrapped + 0x91bf58)
                                                   #6  0x00007fd370a85272 start_thread (libc.so.6 + 0x8e272)
                                                   #7  0x00007fd370b00c24 __clone (libc.so.6 + 0x109c24)
                                                   ELF object binary architecture: AMD x86-64

The segfault happened in kvm_virtio_pci_vector_use_one, on thread 1545895:

$ sudo coredumpctl debug
[...]
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000556bf0e25a24 in kvm_virtio_pci_vector_use_one ()
[Current thread is 1 (Thread 0x7fd363fff6c0 (LWP 1545895))]
(gdb) where
#0  0x0000556bf0e25a24 in kvm_virtio_pci_vector_use_one ()
#1  0x0000556bf10183e0 in memory_region_write_accessor ()
#2  0x0000556bf1017ca6 in access_with_adjusted_size ()
#3  0x0000556bf1017fd1 in memory_region_dispatch_write ()
#4  0x0000556bf101fa5c in flatview_write_continue ()
#5  0x0000556bf101fcd3 in flatview_write ()
#6  0x0000556bf1022fe9 in address_space_rw ()
#7  0x0000556bf1077498 in kvm_cpu_exec ()
#8  0x0000556bf10786e5 in kvm_vcpu_thread_fn ()
#9  0x0000556bf120af58 in qemu_thread_start ()
#10 0x00007fd370a85272 in start_thread () from /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libc.so.6
#11 0x00007fd370b00c24 in clone () from /nix/store/dbcw19dshdwnxdv5q2g6wldj6syyvq7l-glibc-2.39-52/lib/libc.so.6

Add support for macOS runners #3

Description

Activity

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions