Open
Description
gemini process was oom-killed in https://argus.scylladb.com/test/2873b203-404e-492c-957e-8cf49830f0f5/runs?additionalRuns[]=b54477a2-75ef-4244-91bb-21e2f3cd7556 :
2024-05-30T12:18:59.007+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: node_exporter invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_
adj=0
2024-05-30T12:18:59.007+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: CPU: 0 PID: 4210 Comm: node_exporter Not tainted 6.5.0-1018-aws scylladb/scylla-cluster-tests#18~22.04.1-Ubuntu
2024-05-30T12:18:59.007+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: Hardware name: Amazon EC2 c6i.xlarge/, BIOS 1.0 10/16/2017
2024-05-30T12:18:59.007+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: Call Trace:
2024-05-30T12:18:59.007+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: <TASK>
2024-05-30T12:18:59.007+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: dump_stack_lvl+0x48/0x70
2024-05-30T12:18:59.007+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: dump_stack+0x10/0x20
2024-05-30T12:18:59.007+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: dump_header+0x50/0x270
2024-05-30T12:18:59.007+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: oom_kill_process+0x10c/0x1b0
2024-05-30T12:18:59.007+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: out_of_memory+0x109/0x350
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: __alloc_pages_may_oom+0x114/0x1e0
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: __alloc_pages_slowpath.constprop.0+0x4b1/0x9a0
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: __alloc_pages+0x313/0x340
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: alloc_pages+0x95/0x1a0
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: folio_alloc+0x1b/0x60
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: filemap_alloc_folio+0x31/0x40
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: __filemap_get_folio+0xf8/0x260
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: filemap_fault+0x4d1/0x800
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: __do_fault+0x36/0x150
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: do_read_fault+0xfc/0x170
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: do_fault+0xf3/0x170
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: handle_pte_fault+0x7b/0x190
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: __handle_mm_fault+0x3c8/0x730
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: handle_mm_fault+0xe9/0x360
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: do_user_addr_fault+0x227/0x680
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: exc_page_fault+0x84/0x190
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: asm_exc_page_fault+0x27/0x30
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: RIP: 0033:0x459890
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: Code: Unable to access opcode bytes at 0x459866.
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: RSP: 002b:000000c00000f218 EFLAGS: 00010202
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: RAX: 000000000001726a RBX: 0000000000000000 RCX: 00000000010bf360
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: RDX: 00000000000001c4 RSI: 0000000000003d2d RDI: 00000000000001c5
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: RBP: 000000c00000f228 R08: 0000000000f4e320 R09: 0000000000f4f148
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: R10: 000000c00034a820 R11: 0000000000000000 R12: 000000c000108408
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: R13: 0000000001110fe0 R14: 000000c000006000 R15: 000000000000000a
2024-05-30T12:18:59.008+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: </TASK>
2024-05-30T12:18:59.009+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: Mem-Info:
2024-05-30T12:18:59.009+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: active_anon:1889266 inactive_anon:25485 isolated_anon:0
active_file:4 inactive_file:0 isolated_file:0
unevictable:8683 dirty:0 writeback:0
slab_reclaimable:6565 slab_unreclaimable:13927
mapped:2241 shmem:247 pagetables:4619
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:25041 free_pcp:52 free_cma:0
B isolated(anon):0kB isolated(file):0kB mapped:8964kB dirty:0kB writeback:0kB shmem:988kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 550912kB writeback_tmp:0kB kernel_stack:3360kB pagetables:18476kB sec_pageta
bles:0kB all_unreclaimable? no
2024-05-30T12:18:59.009+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: Node 0 DMA free:14336kB boost:0kB min:128kB low:160kB high:192kB reserved_highatomic:0KB active_anon:0kB
inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
2024-05-30T12:18:59.009+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: lowmem_reserve[]: 0 2984 7760 7760 7760
2024-05-30T12:18:59.009+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: Node 0 DMA32 free:44876kB boost:0kB min:25940kB low:32424kB high:38908kB reserved_highatomic:0KB active_a
non:3012160kB inactive_anon:16kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:3129252kB managed:3063716kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
2024-05-30T12:18:59.009+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: lowmem_reserve[]: 0 0 4775 4775 4775
2024-05-30T12:18:59.009+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: Node 0 Normal free:40952kB boost:0kB min:41508kB low:51884kB high:62260kB reserved_highatomic:0KB active_
anon:4544904kB inactive_anon:101924kB active_file:800kB inactive_file:0kB unevictable:34732kB writepending:0kB present:5095424kB managed:4898636kB mlocked:27616kB bounce:0kB free_pcp:208kB local_pcp:144kB free_cm
a:0kB
2024-05-30T12:18:59.009+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: lowmem_reserve[]: 0 0 0 0 0
2024-05-30T12:18:59.009+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB (M) 3*4096kB (M) =
14336kB
2024-05-30T12:18:59.009+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: Node 0 DMA32: 754*4kB (UM) 649*8kB (UM) 529*16kB (UM) 357*32kB (UM) 186*64kB (UM) 42*128kB (UM) 0*256kB 0
*512kB 0*1024kB 0*2048kB 0*4096kB = 45376kB
2024-05-30T12:18:59.009+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: Node 0 Normal: 662*4kB (UME) 314*8kB (ME) 352*16kB (ME) 217*32kB (ME) 136*64kB (ME) 64*128kB (UME) 17*256
kB (E) 5*512kB (UME) 0*1024kB 0*2048kB 0*4096kB = 41544kB
2024-05-30T12:18:59.009+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
2024-05-30T12:18:59.009+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
2024-05-30T12:18:59.009+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: 4332 total pagecache pages
2024-05-30T12:18:59.009+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: 0 pages in swap cache
2024-05-30T12:18:59.009+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: Free swap = 0kB
2024-05-30T12:18:59.009+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: Total swap = 0kB
2024-05-30T12:18:59.009+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: 2060167 pages RAM
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: 0 pages HighMem/MovableOnly
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: 65739 pages reserved
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !WARNING | kernel: 0 pages hwpoisoned
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: Tasks state (memory values in pages):
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ 128] 0 128 16178 1047 106496 0 -250 systemd-journal
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ 168] 0 168 72328 6848 110592 0 -1000 multipathd
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ 172] 0 172 2973 1111 65536 0 -1000 systemd-udevd
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ 351] 100 351 4065 992 69632 0 0 systemd-network
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ 353] 101 353 6418 1848 86016 0 0 systemd-resolve
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ 444] 0 444 704 480 49152 0 0 acpid
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ 448] 0 448 1822 576 53248 0 0 cron
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ 449] 102 449 2175 992 57344 0 -900 dbus-daemon
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ 456] 0 456 20692 800 65536 0 0 irqbalance
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ 457] 0 457 8271 3008 106496 0 0 networkd-dispat
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ 459] 0 459 328886 2131 204800 0 0 amazon-ssm-agen
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ 470] 0 470 3879 960 77824 0 0 systemd-logind
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ 475] 0 475 1555 512 49152 0 0 agetty
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ 478] 0 478 1544 480 49152 0 0 agetty
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ 482] 114 482 4723 743 65536 0 0 chronyd
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ 516] 0 516 27525 2944 122880 0 0 unattended-upgr
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ 517] 0 517 58868 1148 94208 0 0 polkitd
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ 520] 114 520 2641 433 65536 0 0 chronyd
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ 1496] 0 1496 74000 1632 167936 0 0 packagekitd
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ 1542] 0 1542 385428 3511 274432 0 -900 snapd
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ 2800] 0 2800 3860 1248 73728 0 -1000 sshd
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ 3291] 1000 3291 4235 1120 73728 0 0 systemd
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ 3292] 1000 3292 42604 1496 98304 0 0 (sd-pam)
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ 3397] 0 3397 105873 1781 163840 0 0 syslog-ng
2024-05-30T12:18:59.010+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ 4210] 998 4210 310475 1839 126976 0 0 node_exporter
2024-05-30T12:18:59.011+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ 5530] 0 5530 24364 2894 94208 0 0 dirmngr
2024-05-30T12:18:59.011+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ 5533] 0 5533 20454 524 57344 0 0 gpg-agent
2024-05-30T12:18:59.011+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ 7685] 0 7685 468247 3441 294912 0 -999 containerd
2024-05-30T12:18:59.011+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ 7778] 0 7778 533762 8394 438272 0 -500 dockerd
2024-05-30T12:18:59.011+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ 8264] 0 8264 4266 1344 69632 0 0 sshd
2024-05-30T12:18:59.011+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ 8309] 1000 8309 4301 1152 69632 0 0 sshd
2024-05-30T12:18:59.011+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ 8393] 0 8393 4267 1344 77824 0 0 sshd
2024-05-30T12:18:59.011+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ 8438] 1000 8438 4302 1146 77824 0 0 sshd
2024-05-30T12:18:59.011+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ 8457] 0 8457 309118 718 114688 0 -998 containerd-shim
2024-05-30T12:18:59.011+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ 8478] 0 8478 1100 32 53248 0 0 tail
2024-05-30T12:18:59.011+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ 8492] 1000 8492 2975 832 69632 0 0 sudo
2024-05-30T12:18:59.011+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ 8493] 0 8493 424983 2290 221184 0 0 docker
2024-05-30T12:18:59.011+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: [ 8510] 0 8510 2049685 1877425 15196160 0 0 gemini
2024-05-30T12:18:59.011+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !INFO | kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=node_exporter.service,mems_allowed=0,global_oom,task_memcg=/system.slice/docker-9211e23c8eb758d31b282cf584797f768dabe6b2825eee667953454456f12fee.scope,task=gemini,pid=8510,uid=0
2024-05-30T12:18:59.011+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !ERR | kernel: Out of memory: Killed process 8510 (gemini) total-vm:8198740kB, anon-rss:7509700kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:14840kB oom_score_adj:0
2024-05-30T12:18:59.011+00:00 gemini-with-nemesis-3h-normal-6-0-loader-node-b54477a2-1 !NOTICE | systemd[1]: docker-9211e23c8eb758d31b282cf584797f768dabe6b2825eee667953454456f12fee.scope: A process of this unit has been killed by the OOM killer.
c6i.xlarge instance type (default for a loader in SCT) has 4 CPUs and 8GB of RAM. We need to switch to an instance type with more RAM or add a swap for gemini-3h-with-nemesis test.