kernel panic when installing k3s

[matteyeux]

elemental-toolkit version:

v2.3.0-dev+gbc64b5a

CPU architecture, OS, and Version:

Linux installer 6.8.0-53-generic #55-Ubuntu SMP PREEMPT_DYNAMIC Fri Jan 17 15:37:52 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Describe the bug

Starting k3s server with flannel backend (or calico backend) triggers a kernel panic.

[  404.883906] BUG: kernel NULL pointer dereference, address: 0000000000000000
[  404.884500] #PF: supervisor read access in kernel mode
[  404.884891] #PF: error_code(0x0000) - not-present page
[  404.885271] PGD 107641067 P4D 107641067 PUD 108c0d067 PMD 0 
[  404.885693] Oops: 0000 [#1] PREEMPT SMP NOPTI
[  404.886029] CPU: 1 PID: 2829 Comm: conntrack Not tainted 6.8.0-53-generic #55-Ubuntu
[  404.886601] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 0.0.0 02/06/2015
[  404.887190] RIP: 0010:strlen+0x0/0x30
[  404.887465] Code: 48 89 f8 31 d2 31 f6 31 ff c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 <80> 3f 00 74 16 48 89 f8 48 83 c0 01 80 38 00 75 f7 48 29 f8 31 ff
[  404.888814] RSP: 0018:ffffc00780dc75d8 EFLAGS: 00010246
[  404.889204] RAX: 0000000000000000 RBX: ffff9993cdae9f00 RCX: 0000000000000000
[  404.889726] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[  404.890244] RBP: ffffc00780dc7610 R08: 0000000000000000 R09: 0000000000000000
[  404.890765] R10: ffff9993c6389090 R11: 0000000000000000 R12: 0000000000000000
[  404.891346] R13: ffff9993c638908c R14: ffff9993c6389048 R15: ffffc00780dc762c
[  404.891873] FS:  0000000000431538(0000) GS:ffff9996e0280000(0000) knlGS:0000000000000000
[  404.892460] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  404.892886] CR2: 0000000000000000 CR3: 0000000103d5c006 CR4: 0000000000770ef0
[  404.893411] PKRU: 55555554
[  404.893628] Call Trace:
[  404.893844]  <TASK>
[  404.894010]  ? show_regs+0x6d/0x80
[  404.894271]  ? __die+0x24/0x80
[  404.894501]  ? page_fault_oops+0x99/0x1b0
[  404.894807]  ? do_user_addr_fault+0x2e9/0x670
[  404.895130]  ? exc_page_fault+0x83/0x1b0
[  404.895426]  ? asm_exc_page_fault+0x27/0x30
[  404.895757]  ? __pfx_strlen+0x10/0x10
[  404.896029]  ? ctnetlink_dump_secctx.isra.0+0x7a/0xf0 [nf_conntrack_netlink]
[  404.896553]  ctnetlink_fill_info+0x297/0x480 [nf_conntrack_netlink]
[  404.897008]  ctnetlink_dump_table+0x27a/0x5f0 [nf_conntrack_netlink]
[  404.897674]  netlink_dump+0x1bd/0x350
[  404.898128]  ? kmalloc_trace+0x139/0x360
[  404.898609]  __netlink_dump_start+0x1e4/0x310
[  404.899114]  ctnetlink_get_conntrack+0x233/0x2c0 [nf_conntrack_netlink]
[  404.899783]  ? __pfx_ctnetlink_start+0x10/0x10 [nf_conntrack_netlink]
[  404.900435]  ? __pfx_ctnetlink_dump_table+0x10/0x10 [nf_conntrack_netlink]
[  404.901123]  ? __pfx_ctnetlink_done+0x10/0x10 [nf_conntrack_netlink]
[  404.901851]  nfnetlink_rcv_msg+0x347/0x420 [nfnetlink]
[  404.902683]  ? __pfx_nfnetlink_rcv_msg+0x10/0x10 [nfnetlink]
[  404.903293]  netlink_rcv_skb+0x5a/0x110
[  404.903776]  nfnetlink_rcv+0x6c/0x180 [nfnetlink]
[  404.904324]  netlink_unicast+0x24a/0x390
[  404.904996]  netlink_sendmsg+0x214/0x470
[  404.905607]  __sys_sendto+0x21b/0x230
[  404.906065]  __x64_sys_sendto+0x24/0x40
[  404.906545]  x64_sys_call+0x1b2d/0x25a0
[  404.907004]  do_syscall_64+0x7f/0x180
[  404.907498]  ? __count_memcg_events+0x6b/0x120
[  404.908004]  ? count_memcg_events.constprop.0+0x2a/0x50
[  404.908558]  ? handle_mm_fault+0xad/0x380
[  404.909029]  ? do_user_addr_fault+0x333/0x670
[  404.909504]  ? irqentry_exit_to_user_mode+0x7b/0x260
[  404.910033]  ? irqentry_exit+0x43/0x50
[  404.910489]  ? clear_bhb_loop+0x15/0x70
[  404.911087]  ? clear_bhb_loop+0x15/0x70
[  404.911570]  ? clear_bhb_loop+0x15/0x70
[  404.911993]  entry_SYSCALL_64_after_hwframe+0x78/0x80
[  404.912500] RIP: 0033:0x41998b
[  404.912877] Code: 0f 05 48 83 f8 da 75 08 4c 89 c0 48 89 d6 0f 05 c3 48 89 f8 4d 89 c2 48 89 f7 4d 89 c8 48 89 d6 4c 8b 4c 24 08 48 89 ca 0f 05 <c3> e9 e1 ff ff ff 56 e8 4d 00 00 00 48 85 c0 74 09 48 89 c7 59 e9
[  404.914368] RSP: 002b:00007ffc9ab6b4d8 EFLAGS: 00000206 ORIG_RAX: 000000000000002c
[  404.915072] RAX: ffffffffffffffda RBX: 0000728e59508240 RCX: 000000000041998b
[  404.915767] RDX: 0000000000000014 RSI: 00007ffc9ab6c500 RDI: 0000000000000003
[  404.916445] RBP: 00007ffc9ab6d540 R08: 0000000000425298 R09: 000000000000000c
[  404.917368] R10: 0000000000000000 R11: 0000000000000206 R12: 00007ffc9ab6c500
[  404.918136] R13: 0000000000001000 R14: 00007ffc9ab6b500 R15: 0000000000000002
[  404.918868]  </TASK>
[  404.919224] Modules linked in: nf_conntrack_netlink nft_log nft_limit xt_limit xt_NFLOG nfnetlink_log xt_physdev veth vxlan ip6_udp_tunnel udp_tunnel xt_multiport ipt_REJECT nf_reject_ipv4 xt_addrtype xt_nat xt_tcpudp xt_MASQUERADE xt_mark xt_nfacct ip_set xt_conntrack xt_comment nft_chain_nat nft_compat nf_tables iptable_filter iptable_nat nf_nat br_netfilter bridge stp llc nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nfnetlink_acct intel_rapl_msr intel_rapl_common intel_uncore_frequency_common intel_pmc_core intel_vsec pmt_telemetry pmt_class kvm_intel ppdev kvm irqbypass rapl virtio_gpu joydev virtio_dma_buf psmouse i2c_piix4 parport_pc input_leds parport mac_hid binfmt_misc sch_fq_codel efi_pstore nfnetlink dmi_sysfs ip_tables x_tables overlay isofs cfg80211 crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel sha256_ssse3 sha1_ssse3 floppy e1000 serio_raw pata_acpi btrfs blake2b_generic xor raid6_pq libcrc32c dm_mirror dm_region_hash dm_log qemu_fw_cfg autofs4 aesni_intel crypto_simd
[  404.919282]  cryptd
[  404.927263] CR2: 0000000000000000
[  404.927766] ---[ end trace 0000000000000000 ]---
[  405.314647] RIP: 0010:strlen+0x0/0x30
[  405.315275] Code: 48 89 f8 31 d2 31 f6 31 ff c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 <80> 3f 00 74 16 48 89 f8 48 83 c0 01 80 38 00 75 f7 48 29 f8 31 ff
[  405.316869] RSP: 0018:ffffc00780dc75d8 EFLAGS: 00010246
[  405.317516] RAX: 0000000000000000 RBX: ffff9993cdae9f00 RCX: 0000000000000000
[  405.318295] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[  405.319069] RBP: ffffc00780dc7610 R08: 0000000000000000 R09: 0000000000000000
[  405.319844] R10: ffff9993c6389090 R11: 0000000000000000 R12: 0000000000000000
[  405.320615] R13: ffff9993c638908c R14: ffff9993c6389048 R15: ffffc00780dc762c
[  405.321389] FS:  0000000000431538(0000) GS:ffff9996e0280000(0000) knlGS:0000000000000000
[  405.322233] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  405.322914] CR2: 0000000000000000 CR3: 0000000103d5c006 CR4: 0000000000770ef0
[  405.323730] PKRU: 55555554
[  405.324209] Kernel panic - not syncing: Fatal exception in interrupt
[  405.325036] Kernel Offset: 0x2a400000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[  405.682559] Rebooting in 10 seconds..

To Reproduce
Use the orange example and build the ISO and boot it.

Download and run k3s server :

$ curl -LO https://github.com/k3s-io/k3s/releases/download/v1.31.5%2Bk3s1/k3s
$ chmod +x k3s
$ ./k3s server --disable traefik --snapshotter=native

Kernel panic

Expected behavior
I expected it to not kernel panic. I tested on a real ubuntu server ISO (not built with elemental-toolkit) and it works smoothly.

Logs

Additional context

I was able to start k3s with --flannel-backend=None (so no network) and it works, but it's pretty much useless. : ./k3s server --disable traefik --snapshotter=native --flannel-backend=None

By reading the documentation, I thought k3s was supported, but it seems to work only on openSUSE Tumbleweed (I have no issue with the green example).

The kernel panic is not a normal behaviour for the kernel, so there is something wrong with it that I want to debug, but hard to understand what's wrong with k3s and elemental-toolkit based OS.

[frelon]

I'm reading from the issue that you boot the ISO and try to download and run k3s directly without installing to a disk first, is that correct?

[matteyeux]

Yes. I'm doing the same thing using the other example based on OpenSuse and it works perfectly

[kkaempf]

Sorry, @matteyeux , I'm seeing Ubuntu here and k3s - not sure how this is related to Elemental.

But we can dig a bit deeper. Can you do the following please

1. try it on a plain Ubuntu (no Elemental)
2. run k3s with strace

[matteyeux]

not sure how this is related to Elemental.

Well, as I said in the Expected behavior  I tested with a normal ubuntu server image and it worked, but not with the ISO built with elemental. That's why I am opening an issue here.

1. Already done
2. Didn't think about running strace, will do it tomorrow.

Thanks

[kkaempf]

Thanks for the feedback @matteyeux !
Interesting that it works on plain Ubuntu but not on "immutable Ubuntu".
strace might find to narrow down the root cause.