kernel "page allocation failure" occurs

[mura890]

Mountpoint for Amazon S3 version

mount-s3 1.19.0

AWS Region

ap-northeast-1

Describe the running environment

I am running an S3 bucket in the same account mounted on EC2, OS: Rocky Linux release 8.10 (Green Obsidian).
The instance type is c6in.2xlarge.

Mountpoint options

/usr/bin/mount-s3 prod-S3-bucket --prefix archives/ /mnt/applications/prod/archives --uid 1001 --gid 1001 --file-mode 0660 --dir-mode 0770 --allow-delete --allow-other --allow-overwrite --log-directory /var/log/mount-s3/ --metadata-ttl 10

What happened?

A kernel “page allocation failure” occurred when copying 16GB and 17GB video content to an S3 bucket mounted via mount-s3. (mount-s3 did not terminate abnormally.)

This has never happened before on an instance that has been running continuously for a long time.

If there is a known cause or countermeasure for this phenomenon, please let me know. Thank you in advance.

Translated with DeepL.com (free version)

Relevant log output

Feb  9 08:09:39 prod-server kernel: AwsEventLoop 4: page allocation failure: order:0, mode:0x484020(GFP_ATOMIC|__GFP_COMP), nodemask=(null),cpuset=/,mems_allowed=0
Feb  9 08:09:39 prod-server kernel: CPU: 6 PID: 544614 Comm: AwsEventLoop 4 Kdump: loaded Not tainted 4.18.0-553.30.1.el8_10.x86_64 #1
Feb  9 08:09:39 prod-server kernel: Hardware name: Amazon EC2 c6in.2xlarge/, BIOS 1.0 10/16/2017
Feb  9 08:09:39 prod-server kernel: Call Trace:
Feb  9 08:09:39 prod-server kernel: <IRQ>
Feb  9 08:09:39 prod-server kernel: dump_stack+0x41/0x60
Feb  9 08:09:39 prod-server kernel: warn_alloc.cold.127+0x7b/0x108
Feb  9 08:09:39 prod-server kernel: __alloc_pages_slowpath+0xcb2/0xcd0
Feb  9 08:09:39 prod-server kernel: ? tcp_v4_rcv+0xd18/0xd90
Feb  9 08:09:39 prod-server kernel: __alloc_pages_nodemask+0x2e2/0x330
Feb  9 08:09:39 prod-server kernel: new_slab+0x3f4/0x4f0
Feb  9 08:09:39 prod-server kernel: ? tcp_v4_early_demux+0xfb/0x160
Feb  9 08:09:39 prod-server kernel: ___slab_alloc+0x3a3/0x950
Feb  9 08:09:39 prod-server kernel: ? __alloc_pages_slowpath+0x166/0xcd0
Feb  9 08:09:39 prod-server kernel: ? napi_skb_cache_get+0x47/0x60
Feb  9 08:09:39 prod-server kernel: kmem_cache_alloc_bulk+0x153/0x300
Feb  9 08:09:39 prod-server kernel: napi_skb_cache_get+0x47/0x60
Feb  9 08:09:39 prod-server kernel: __napi_build_skb+0x14/0x70
Feb  9 08:09:39 prod-server kernel: napi_build_skb+0x11/0x80
Feb  9 08:09:39 prod-server kernel: ena_alloc_skb+0x23/0xa0 [ena]
Feb  9 08:09:39 prod-server kernel: ena_clean_rx_irq+0x5cc/0xd40 [ena]
Feb  9 08:09:39 prod-server kernel: ? ena_io_poll+0x183/0x6e0 [ena]
Feb  9 08:09:39 prod-server kernel: ? kmem_cache_free+0x2d6/0x300
Feb  9 08:09:39 prod-server kernel: ena_io_poll+0x30a/0x6e0 [ena]
Feb  9 08:09:39 prod-server kernel: __napi_poll+0x2d/0x130
Feb  9 08:09:39 prod-server kernel: net_rx_action+0x252/0x320
Feb  9 08:09:39 prod-server kernel: __do_softirq+0xdc/0x2cf
Feb  9 08:09:39 prod-server kernel: irq_exit_rcu+0xc6/0xd0
Feb  9 08:09:39 prod-server kernel: irq_exit+0xa/0x10
Feb  9 08:09:39 prod-server kernel: do_IRQ+0x7f/0xd0
Feb  9 08:09:39 prod-server kernel: common_interrupt+0xf/0xf
Feb  9 08:09:39 prod-server kernel: </IRQ>
Feb  9 08:09:39 prod-server kernel: RIP: 0010:__page_cache_release+0x24/0x230
Feb  9 08:09:39 prod-server kernel: Code: 66 0f 1f 44 00 00 0f 1f 44 00 00 41 57 41 56 41 55 41 54 55 53 48 89 fb 48 83 ec 20 65 48 8b 04 25 28 00 00 00 48 89 44 24 18 <31> c0 48 8b 57 08 48 8d 42 ff 83 e2 01 48 0f 44 c7 48 8b 00 a8 20
Feb  9 08:09:39 prod-server kernel: RSP: 0018:ffffa2ca0497fc60 EFLAGS: 00000286 ORIG_RAX: ffffffffffffffd9
Feb  9 08:09:39 prod-server kernel: RAX: c56ad8386f330c00 RBX: ffffd2d74b834580 RCX: 0000000000000030
Feb  9 08:09:39 prod-server kernel: RDX: dead0000000000ff RSI: ffff938b1f1b98f0 RDI: ffffd2d74b834580
Feb  9 08:09:39 prod-server kernel: RBP: 000000000000000b R08: ffffd2d744f59208 R09: ffff938b2efd4000
Feb  9 08:09:39 prod-server kernel: R10: 0000000000000000 R11: ffffd2d744faeb88 R12: ffff9388518e9400
Feb  9 08:09:39 prod-server kernel: R13: 0000000000000002 R14: ffff938848aecec0 R15: 0000000000000000
Feb  9 08:09:39 prod-server kernel: ? free_unref_page_commit+0x91/0x110
Feb  9 08:09:39 prod-server kernel: __put_page+0x2f/0x50
Feb  9 08:09:39 prod-server kernel: skb_release_data+0x9f/0x190
Feb  9 08:09:39 prod-server kernel: __kfree_skb+0x13/0x20
Feb  9 08:09:39 prod-server kernel: tcp_recvmsg+0x7a9/0xa90
Feb  9 08:09:39 prod-server kernel: ? sock_poll+0x61/0xe0
Feb  9 08:09:39 prod-server kernel: ? ep_item_poll.isra.19+0x40/0xc0
Feb  9 08:09:39 prod-server kernel: inet_recvmsg+0x5b/0xd0
Feb  9 08:09:39 prod-server kernel: sock_read_iter+0x92/0x100
Feb  9 08:09:39 prod-server kernel: new_sync_read+0x10f/0x160
Feb  9 08:09:39 prod-server kernel: vfs_read+0x91/0x150
Feb  9 08:09:39 prod-server kernel: ksys_read+0x4f/0xb0
Feb  9 08:09:39 prod-server kernel: do_syscall_64+0x5b/0x1a0
Feb  9 08:09:39 prod-server kernel: entry_SYSCALL_64_after_hwframe+0x66/0xcb
Feb  9 08:09:39 prod-server kernel: RIP: 0033:0x7fa648c12ad4
Feb  9 08:09:39 prod-server kernel: Code: c3 0f 1f 44 00 00 41 54 49 89 d4 55 48 89 f5 53 89 fb 48 83 ec 10 e8 7b fc ff ff 4c 89 e2 48 89 ee 89 df 41 89 c0 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 38 44 89 c7 48 89 44 24 08 e8 b7 fc ff ff 48
Feb  9 08:09:39 prod-server kernel: RSP: 002b:00007fa647c95470 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
Feb  9 08:09:39 prod-server kernel: RAX: ffffffffffffffda RBX: 0000000000000020 RCX: 00007fa648c12ad4
Feb  9 08:09:39 prod-server kernel: RDX: 0000000000004000 RSI: 00007fa6301fa390 RDI: 0000000000000020
Feb  9 08:09:39 prod-server kernel: RBP: 00007fa6301fa390 R08: 0000000000000000 R09: 00000000199be844
Feb  9 08:09:39 prod-server kernel: R10: 00000000199be844 R11: 0000000000000246 R12: 0000000000004000
Feb  9 08:09:39 prod-server kernel: R13: 0000000000000000 R14: 00007fa647c95570 R15: 00007fa647c95508
Feb  9 08:09:39 prod-server kernel: Mem-Info:
Feb  9 08:09:39 prod-server kernel: active_anon:656114 inactive_anon:679646 isolated_anon:0#012 active_file:1022595 inactive_file:1262020 isolated_file:0#012 unevictable:0 dirty:140 writeback:0#012 slab_reclaimable:119934 slab_unreclaimable:40229#012 mapped:62257 shmem:42828 pagetables:8912 bounce:0#012 free:114480 free_pcp:1688 free_cma:0
Feb  9 08:09:39 prod-server kernel: Node 0 active_anon:2624456kB inactive_anon:2718584kB active_file:4090120kB inactive_file:5048080kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:249028kB dirty:560kB writeback:0kB shmem:171312kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 3837952kB writeback_tmp:0kB kernel_stack:28912kB pagetables:35648kB all_unreclaimable? no
Feb  9 08:09:39 prod-server kernel: Node 0 DMA free:13312kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Feb  9 08:09:39 prod-server kernel: lowmem_reserve[]: 0 2660 15392 15392 15392
Feb  9 08:09:39 prod-server kernel: Node 0 DMA32 free:84556kB min:11664kB low:14580kB high:17496kB active_anon:664528kB inactive_anon:422716kB active_file:255048kB inactive_file:1284944kB unevictable:0kB writepending:92kB present:3129252kB managed:2801572kB mlocked:0kB bounce:0kB free_pcp:6280kB local_pcp:224kB free_cma:0kB
Feb  9 08:09:39 prod-server kernel: lowmem_reserve[]: 0 0 12732 12732 12732
Feb  9 08:09:39 prod-server kernel: Node 0 Normal free:397604kB min:181500kB low:195460kB high:209420kB active_anon:1959928kB inactive_anon:2296028kB active_file:3835332kB inactive_file:3763416kB unevictable:0kB writepending:468kB present:13352960kB managed:13047068kB mlocked:0kB bounce:0kB free_pcp:940kB local_pcp:168kB free_cma:0kB
Feb  9 08:09:39 prod-server kernel: lowmem_reserve[]: 0 0 0 0 0
Feb  9 08:09:39 prod-server kernel: Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 2*2048kB (UM) 2*4096kB (M) = 13312kB
Feb  9 08:09:39 prod-server kernel: Node 0 DMA32: 1398*4kB (UMEH) 1420*8kB (UMEH) 1346*16kB (UMEH) 618*32kB (UMEH) 254*64kB (UMEH) 60*128kB (UMH) 9*256kB (UMH) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 84504kB
Feb  9 08:09:39 prod-server kernel: Node 0 Normal: 6586*4kB (UME) 43265*8kB (UMEH) 945*16kB (UMEH) 176*32kB (UMEH) 18*64kB (UEH) 0*128kB 0*256kB 0*512kB 1*1024kB (H) 1*2048kB (H) 0*4096kB = 397440kB
Feb  9 08:09:39 prod-server kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Feb  9 08:09:39 prod-server kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Feb  9 08:09:39 prod-server kernel: 2338330 total pagecache pages
Feb  9 08:09:39 prod-server kernel: 10949 pages in swap cache
Feb  9 08:09:39 prod-server kernel: Swap cache stats: add 15336328, delete 15325808, find 10448715/11777434
Feb  9 08:09:39 prod-server kernel: Free swap  = 16042236kB
Feb  9 08:09:39 prod-server kernel: Total swap = 18431996kB
Feb  9 08:09:39 prod-server kernel: 4124551 pages RAM
Feb  9 08:09:39 prod-server kernel: 0 pages HighMem/MovableOnly
Feb  9 08:09:39 prod-server kernel: 158551 pages reserved
Feb  9 08:09:39 prod-server kernel: 0 pages hwpoisoned
Feb  9 08:09:39 prod-server kernel: SLUB: Unable to allocate memory on node -1, gfp=0x480020(GFP_ATOMIC)
Feb  9 08:09:39 prod-server kernel:  cache: skbuff_head_cache, object size: 256, buffer size: 256, default order: 1, min order: 0
Feb  9 08:09:39 prod-server kernel:  node 0: slabs: 65, objs: 2080, free: 1163
Feb  9 08:09:45 prod-server kernel: AwsEventLoop 1: page allocation failure: order:0, mode:0x480020(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0
Feb  9 08:09:45 prod-server kernel: swapper/6: page allocation failure: order:0, mode:0x480020(GFP_ATOMIC), nodemask=(null),cpuset=/,mems_allowed=0
Feb  9 08:09:45 prod-server kernel: CPU: 3 PID: 544611 Comm: AwsEventLoop 1 Kdump: loaded Not tainted 4.18.0-553.30.1.el8_10.x86_64 #1
Feb  9 08:09:45 prod-server kernel: AwsEventLoop 2: page allocation failure: order:0, mode:0x484020(GFP_ATOMIC|__GFP_COMP), nodemask=(null),cpuset=/,mems_allowed=0
Feb  9 08:09:45 prod-server kernel: kswapd0: page allocation failure: order:0, mode:0x484020(GFP_ATOMIC|__GFP_COMP), nodemask=(null),cpuset=/,mems_allowed=0
Feb  9 08:09:46 prod-server kernel: Hardware name: Amazon EC2 c6in.2xlarge/, BIOS 1.0 10/16/2017
Feb  9 08:09:46 prod-server kernel: Call Trace:
Feb  9 08:09:46 prod-server kernel: <IRQ>
Feb  9 08:09:46 prod-server kernel: dump_stack+0x41/0x60
Feb  9 08:09:46 prod-server kernel: warn_alloc.cold.127+0x7b/0x108
Feb  9 08:09:46 prod-server kernel: pool-358-thread: page allocation failure: order:0, mode:0x480020(GFP_ATOMIC), nodemask=(null),cpuset=
Feb  9 08:09:46 prod-server kernel: __alloc_pages_slowpath+0xcb2/0xcd0
Feb  9 08:09:46 prod-server kernel: /,mems_allowed=0
Feb  9 08:09:46 prod-server kernel: __alloc_pages_nodemask+0x2e2/0x330
Feb  9 08:09:46 prod-server kernel: __napi_alloc_skb+0x14c/0x190
Feb  9 08:09:46 prod-server kernel: ena_alloc_skb+0x48/0xa0 [ena]
Feb  9 08:09:46 prod-server kernel: ena_clean_rx_irq+0x27a/0xd40 [ena]
Feb  9 08:09:46 prod-server kernel: 
Feb  9 08:09:46 prod-server kernel: ? inet_current_timestamp+0x40/0x80
Feb  9 08:09:46 prod-server kernel: ? napi_gro_complete.constprop.164+0xc1/0xd0
Feb  9 08:09:46 prod-server kernel: ? napi_gro_flush+0x68/0x90
Feb  9 08:09:46 prod-server kernel: ena_io_poll+0x30a/0x6e0 [ena]
Feb  9 08:09:46 prod-server kernel: __napi_poll+0x2d/0x130
Feb  9 08:09:46 prod-server kernel: net_rx_action+0x252/0x320
Feb  9 08:09:46 prod-server kernel: ? __napi_schedule_irqoff+0x58/0x80
Feb  9 08:09:46 prod-server kernel: __do_softirq+0xdc/0x2cf
Feb  9 08:09:46 prod-server kernel: irq_exit_rcu+0xc6/0xd0
Feb  9 08:09:46 prod-server kernel: irq_exit+0xa/0x10
Feb  9 08:09:46 prod-server kernel: do_IRQ+0x7f/0xd0
Feb  9 08:09:46 prod-server kernel: common_interrupt+0xf/0xf
Feb  9 08:09:46 prod-server kernel: </IRQ>
Feb  9 08:09:46 prod-server kernel: RIP: 0033:0x7fa64835e7b2
Feb  9 08:09:46 prod-server kernel: Code: e7 62 e1 fe 28 7f 07 62 e1 fe 28 7f 47 01 48 81 fa 80 00 00 00 76 89 48 8d 4c 17 80 62 e1 fe 28 7f 40 02 62 e1 fe 28 7f 40 03 <48> 83 ef 80 48 39 f9 76 29 48 83 e7 c0 90 62 e1 fd 28 7f 07 62 e1
Feb  9 08:09:46 prod-server kernel: RSP: 002b:00007fa644c8f3b8 EFLAGS: 00000202 ORIG_RAX: ffffffffffffffdb
Feb  9 08:09:46 prod-server kernel: RAX: 00007fa6283c4f85 RBX: 00007fa62801bba0 RCX: 00007fa6283c5305
Feb  9 08:09:46 prod-server kernel: RDX: 0000000000000400 RSI: 0000000000000000 RDI: 00007fa6283c4f85
Feb  9 08:09:46 prod-server kernel: RBP: 00007fa6283c4f85 R08: ffffffffffffffe0 R09: 00007fa6283c4f85
Feb  9 08:09:46 prod-server kernel: R10: 00007fa628436970 R11: 00007fa628248900 R12: 0000000000000000
Feb  9 08:09:46 prod-server kernel: R13: 00007fa644c8f400 R14: 00007fa644c8f400 R15: 00007fa62801bba0
Feb  9 08:09:46 prod-server kernel: CPU: 7 PID: 544612 Comm: AwsEventLoop 2 Kdump: loaded Not tainted 4.18.0-553.30.1.el8_10.x86_64 #1
Feb  9 08:09:46 prod-server kernel: Mem-Info:
Feb  9 08:09:46 prod-server kernel: Hardware name: Amazon EC2 c6in.2xlarge/, BIOS 1.0 10/16/2017
Feb  9 08:09:46 prod-server kernel: Call Trace:
Feb  9 08:09:46 prod-server kernel: <IRQ>
Feb  9 08:09:46 prod-server kernel: dump_stack+0x41/0x60
Feb  9 08:09:46 prod-server kernel: warn_alloc.cold.127+0x7b/0x108
Feb  9 08:09:46 prod-server kernel: __alloc_pages_slowpath+0xcb2/0xcd0
Feb  9 08:09:46 prod-server kernel: __alloc_pages_nodemask+0x2e2/0x330
Feb  9 08:09:46 prod-server kernel: active_anon:656116 inactive_anon:681057 isolated_anon:0#012 active_file:997946 inactive_file:1353477 isolated_file:32#012 unevictable:0 dirty:18 writeback:65#012 slab_reclaimable:118635 slab_unreclaimable:40136#012 mapped:62777 shmem:42828 pagetables:9076 bounce:0#012 free:41733 free_pcp:1718 free_cma:0
Feb  9 08:09:46 prod-server kernel: new_slab+0x3f4/0x4f0
Feb  9 08:09:46 prod-server kernel: ___slab_alloc+0x3a3/0x950
Feb  9 08:09:46 prod-server kernel: ? napi_skb_cache_get+0x47/0x60
Feb  9 08:09:46 prod-server kernel: ? ip_protocol_deliver_rcu+0x2c/0x1d0
Feb  9 08:09:46 prod-server kernel: Node 0 active_anon:2624464kB inactive_anon:2724228kB active_file:3991784kB inactive_file:5413908kB unevictable:0kB isolated(anon):0kB isolated(file):128kB mapped:251108kB dirty:72kB writeback:260kB shmem:171312kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 3833856kB writeback_tmp:0kB kernel_stack:28944kB pagetables:36304kB all_unreclaimable? no
Feb  9 08:09:46 prod-server kernel: ? ip_local_deliver_finish+0x4d/0x60
Feb  9 08:09:46 prod-server kernel: ? ip_local_deliver+0x6b/0xf0
Feb  9 08:09:46 prod-server kernel: ? tcp_v4_early_demux+0xfb/0x160
Feb  9 08:09:46 prod-server kernel: kmem_cache_alloc_bulk+0x153/0x300
Feb  9 08:09:46 prod-server kernel: napi_skb_cache_get+0x47/0x60
Feb  9 08:09:46 prod-server kernel: __napi_build_skb+0x14/0x70
Feb  9 08:09:46 prod-server kernel: Node 0 
Feb  9 08:09:46 prod-server kernel: napi_build_skb+0x11/0x80
Feb  9 08:09:46 prod-server kernel: ena_alloc_skb+0x23/0xa0 [ena]
Feb  9 08:09:46 prod-server kernel: ena_clean_rx_irq+0x5cc/0xd40 [ena]
Feb  9 08:09:46 prod-server kernel: ? inet_current_timestamp+0x40/0x80
Feb  9 08:09:46 prod-server kernel: DMA free:13312kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Feb  9 08:09:46 prod-server kernel: ? napi_gro_complete.constprop.164+0xc1/0xd0
Feb  9 08:09:46 prod-server kernel: ? napi_gro_flush+0x68/0x90
Feb  9 08:09:46 prod-server kernel: ena_io_poll+0x30a/0x6e0 [ena]
Feb  9 08:09:46 prod-server kernel: ? __wake_up_common+0x7a/0x190
Feb  9 08:09:46 prod-server kernel: __napi_poll+0x2d/0x130
Feb  9 08:09:46 prod-server kernel: lowmem_reserve[]:
Feb  9 08:09:46 prod-server kernel: net_rx_action+0x252/0x320
Feb  9 08:09:46 prod-server kernel: ? __napi_schedule_irqoff+0x58/0x80
Feb  9 08:09:46 prod-server kernel: __do_softirq+0xdc/0x2cf
Feb  9 08:09:46 prod-server kernel: irq_exit_rcu+0xc6/0xd0
Feb  9 08:09:46 prod-server kernel: irq_exit+0xa/0x10
Feb  9 08:09:46 prod-server kernel: do_IRQ+0x7f/0xd0
Feb  9 08:09:46 prod-server kernel: common_interrupt+0xf/0xf
Feb  9 08:09:46 prod-server kernel: 0
Feb  9 08:09:46 prod-server kernel: </IRQ>
Feb  9 08:09:46 prod-server kernel: RIP: 0010:copy_user_enhanced_fast_string+0xe/0x40
Feb  9 08:09:46 prod-server kernel: Code: 89 d1 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 31 c0 0f 01 ca c3 cc cc cc cc 0f 1f 00 0f 01 cb 83 fa 40 0f 82 70 ff ff ff 89 d1 <f3> a4 31 c0 0f 01 ca c3 cc cc cc cc 66 0f 1f 44 00 00 83 f8 12 74
Feb  9 08:09:46 prod-server kernel: RSP: 0018:ffffa2ca0496fbd8 EFLAGS: 00050202 ORIG_RAX: ffffffffffffffdc
Feb  9 08:09:46 prod-server kernel: RAX: 00007fa62c14f27e RBX: ffffa2ca0496fde8 RCX: 0000000000000234
Feb  9 08:09:46 prod-server kernel: RDX: 0000000000000334 RSI: ffff938701678140 RDI: 00007fa62c14f04a
Feb  9 08:09:46 prod-server kernel: 2660
Feb  9 08:09:46 prod-server kernel: RBP: 0000000000000334 R08: 0000000000000000 R09: ffffffffba239e00
Feb  9 08:09:46 prod-server kernel: R10: 0000000000000000 R11: ffff938713173fa0 R12: 00000000000016be
Feb  9 08:09:46 prod-server kernel: R13: ffff938701678040 R14: 0000000000000334 R15: ffffa2ca0496fe48
Feb  9 08:09:46 prod-server kernel: ? skb_kill_datagram+0x70/0x70
Feb  9 08:09:46 prod-server kernel: copyout+0x22/0x30
Feb  9 08:09:46 prod-server kernel: _copy_to_iter+0xa0/0x3b0
Feb  9 08:09:46 prod-server kernel: ? skb_kill_datagram+0x70/0x70
Feb  9 08:09:46 prod-server kernel: ? __check_object_size+0xac/0x173
Feb  9 08:09:46 prod-server kernel: 15392
Feb  9 08:09:46 prod-server kernel: __skb_datagram_iter+0x1a6/0x310
Feb  9 08:09:46 prod-server kernel: 15392
Feb  9 08:09:46 prod-server kernel: ? skb_kill_datagram+0x70/0x70
Feb  9 08:09:46 prod-server kernel: skb_copy_datagram_iter+0x3b/0x90
Feb  9 08:09:46 prod-server kernel: tcp_recvmsg+0x224/0xa90
Feb  9 08:09:46 prod-server kernel: ? sock_poll+0x61/0xe0
Feb  9 08:09:46 prod-server kernel: ? ep_item_poll.isra.19+0x40/0xc0
Feb  9 08:09:46 prod-server kernel: 15392
Feb  9 08:09:46 prod-server kernel: inet_recvmsg+0x5b/0xd0
Feb  9 08:09:46 prod-server kernel: 
Feb  9 08:09:46 prod-server kernel: sock_read_iter+0x92/0x100
Feb  9 08:09:46 prod-server kernel: Node 0 
Feb  9 08:09:46 prod-server kernel: new_sync_read+0x10f/0x160
Feb  9 08:09:46 prod-server kernel: vfs_read+0x91/0x150
Feb  9 08:09:46 prod-server kernel: ksys_read+0x4f/0xb0
Feb  9 08:09:46 prod-server kernel: do_syscall_64+0x5b/0x1a0
Feb  9 08:09:46 prod-server kernel: DMA32 free:65920kB min:11664kB low:14580kB high:17496kB active_anon:664528kB inactive_anon:325036kB active_file:245388kB inactive_file:1386168kB unevictable:0kB writepending:92kB present:3129252kB managed:2801572kB mlocked:0kB bounce:0kB free_pcp:5884kB local_pcp:324kB free_cma:0kB
Feb  9 08:09:46 prod-server kernel: entry_SYSCALL_64_after_hwframe+0x66/0xcb
Feb  9 08:09:46 prod-server kernel: RIP: 0033:0x7fa648c12ad4
Feb  9 08:09:46 prod-server kernel: Code: c3 0f 1f 44 00 00 41 54 49 89 d4 55 48 89 f5 53 89 fb 48 83 ec 10 e8 7b fc ff ff 4c 89 e2 48 89 ee 89 df 41 89 c0 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 38 44 89 c7 48 89 44 24 08 e8 b7 fc ff ff 48
Feb  9 08:09:46 prod-server kernel: RSP: 002b:00007fa627ffe470 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
Feb  9 08:09:46 prod-server kernel: RAX: ffffffffffffffda RBX: 0000000000000033 RCX: 00007fa648c12ad4
Feb  9 08:09:46 prod-server kernel: RDX: 0000000000004000 RSI: 00007fa62c14dbc0 RDI: 0000000000000033
Feb  9 08:09:46 prod-server kernel: RBP: 00007fa62c14dbc0 R08: 0000000000000000 R09: 0000556a0002b8a5
Feb  9 08:09:46 prod-server kernel: lowmem_reserve[]:
Feb  9 08:09:46 prod-server kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000004000
Feb  9 08:09:46 prod-server kernel: 0
Feb  9 08:09:46 prod-server kernel: R13: 0000000000000000 R14: 00007fa627ffe570 R15: 00007fa627ffe508
Feb  9 08:09:46 prod-server kernel: CPU: 2 PID: 1446867 Comm: pool-358-thread Kdump: loaded Not tainted 4.18.0-553.30.1.el8_10.x86_64 #1
Feb  9 08:09:46 prod-server kernel: Hardware name: Amazon EC2 c6in.2xlarge/, BIOS 1.0 10/16/2017
Feb  9 08:09:46 prod-server kernel: Call Trace:
Feb  9 08:09:46 prod-server kernel: <IRQ>
Feb  9 08:09:46 prod-server kernel: SLUB: Unable to allocate memory on node -1, gfp=0x480020(GFP_ATOMIC)
Feb  9 08:09:46 prod-server kernel: dump_stack+0x41/0x60
Feb  9 08:09:46 prod-server kernel: warn_alloc.cold.127+0x7b/0x108
Feb  9 08:09:46 prod-server kernel: __alloc_pages_slowpath+0xcb2/0xcd0
Feb  9 08:09:46 prod-server kernel: __alloc_pages_nodemask+0x2e2/0x330
Feb  9 08:09:46 prod-server kernel: __napi_alloc_skb+0x14c/0x190
Feb  9 08:09:46 prod-server kernel: ena_alloc_skb+0x48/0xa0 [ena]
Feb  9 08:09:46 prod-server kernel: ena_clean_rx_irq+0x27a/0xd40 [ena]
Feb  9 08:09:46 prod-server kernel: ? wait_task_inactive+0x101/0x1b0
Feb  9 08:09:46 prod-server kernel: ? enqueue_entity+0x102/0x6b0
Feb  9 08:09:46 prod-server kernel: ena_io_poll+0x30a/0x6e0 [ena]
Feb  9 08:09:46 prod-server kernel: ? task_tick_fair+0x81/0xad0
Feb  9 08:09:46 prod-server kernel: ? tick_sched_do_timer+0x50/0x50
Feb  9 08:09:46 prod-server kernel: __napi_poll+0x2d/0x130
Feb  9 08:09:46 prod-server kernel: net_rx_action+0x252/0x320
Feb  9 08:09:46 prod-server kernel: ? __napi_schedule_irqoff+0x58/0x80
Feb  9 08:09:46 prod-server kernel: __do_softirq+0xdc/0x2cf
Feb  9 08:09:46 prod-server kernel: irq_exit_rcu+0xc6/0xd0
Feb  9 08:09:46 prod-server kernel: irq_exit+0xa/0x10
Feb  9 08:09:46 prod-server kernel: do_IRQ+0x7f/0xd0
Feb  9 08:09:46 prod-server kernel: common_interrupt+0xf/0xf
Feb  9 08:09:46 prod-server kernel: </IRQ>
Feb  9 08:09:46 prod-server kernel: RIP: 0010:free_unref_page_list+0x11d/0x1a0
Feb  9 08:09:46 prod-server kernel: Code: 77 28 49 c7 47 28 00 00 00 00 0f 1f 44 00 00 4c 89 f6 4c 89 ff 41 83 c4 01 e8 2f e4 ff ff 41 83 fc 20 75 c2 48 8b 3c 24 57 9d <0f> 1f 44 00 00 9c 58 0f 1f 44 00 00 48 89 04 24 fa 66 0f 1f 44 00
Feb  9 08:09:46 prod-server kernel: RSP: 0000:ffffa2ca0a51f578 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffd9
Feb  9 08:09:46 prod-server kernel: RAX: 0000000000000000 RBX: 00000000003750b5 RCX: ffff938b2efd4c50
Feb  9 08:09:46 prod-server kernel: RDX: ffffd2d74e670408 RSI: ffffa2ca0a51f528 RDI: 0000000000000246
Feb  9 08:09:46 prod-server kernel: RBP: ffffa2ca0a51f608 R08: ffffd2d74f936a48 R09: ffffd2d74f936508
Feb  9 08:09:46 prod-server kernel: R10: 0000000000000000 R11: 0000000000000009 R12: 0000000000000020
Feb  9 08:09:46 prod-server kernel: R13: ffffa2ca0a51f610 R14: 00000000003750b5 R15: ffffd2d74dd42d40
Feb  9 08:09:46 prod-server kernel: shrink_page_list+0x1e0/0xce0
Feb  9 08:09:46 prod-server kernel: shrink_inactive_list+0x1d3/0x440
Feb  9 08:09:46 prod-server kernel: shrink_lruvec+0x472/0x6c0
Feb  9 08:09:46 prod-server kernel: ? do_shrink_slab+0x3a/0x330
Feb  9 08:09:46 prod-server kernel: shrink_node+0x23f/0x710
Feb  9 08:09:46 prod-server kernel: do_try_to_free_pages+0xd8/0x4c0
Feb  9 08:09:46 prod-server kernel: try_to_free_pages+0xf3/0x1d0
Feb  9 08:09:46 prod-server kernel: __alloc_pages_slowpath+0x3e2/0xcd0
Feb  9 08:09:46 prod-server kernel: ? __kmalloc+0x113/0x250
Feb  9 08:09:46 prod-server kernel: __alloc_pages_nodemask+0x2e2/0x330
Feb  9 08:09:46 prod-server kernel: xfs_buf_alloc_pages+0xd1/0x1d0 [xfs]
Feb  9 08:09:46 prod-server kernel: xfs_buf_get_map+0x2e7/0x4a0 [xfs]
Feb  9 08:09:46 prod-server kernel: xfs_buf_read_map+0x53/0x320 [xfs]
Feb  9 08:09:46 prod-server kernel: ? kmem_cache_alloc+0x13f/0x280
Feb  9 08:09:46 prod-server kernel: ? xfs_imap_to_bp+0x4b/0x70 [xfs]
Feb  9 08:09:46 prod-server kernel: xfs_trans_read_buf_map+0x1cf/0x360 [xfs]
Feb  9 08:09:46 prod-server kernel: ? xfs_imap_to_bp+0x4b/0x70 [xfs]
Feb  9 08:09:46 prod-server kernel: xfs_imap_to_bp+0x4b/0x70 [xfs]
Feb  9 08:09:46 prod-server kernel: xfs_trans_log_inode+0x171/0x2a0 [xfs]
Feb  9 08:09:46 prod-server kernel: ? xfs_trans_alloc+0xec/0x1e0 [xfs]
Feb  9 08:09:46 prod-server kernel: xfs_vn_update_time+0x14d/0x1e0 [xfs]
Feb  9 08:09:46 prod-server kernel: file_update_time+0xe5/0x130
Feb  9 08:09:46 prod-server kernel: ? futex_wait+0x19e/0x210
Feb  9 08:09:46 prod-server kernel: __xfs_filemap_fault+0x15f/0x200 [xfs]
Feb  9 08:09:46 prod-server kernel: ? hrtimer_init_sleeper+0x90/0x90
Feb  9 08:09:46 prod-server kernel: do_page_mkwrite+0x53/0xa0
Feb  9 08:09:46 prod-server kernel: ? vm_normal_page+0x1a/0xc0
Feb  9 08:09:46 prod-server kernel: do_wp_page+0x29c/0x350
Feb  9 08:09:46 prod-server kernel: __handle_mm_fault+0x552/0x6d0
Feb  9 08:09:46 prod-server kernel: handle_mm_fault+0xca/0x2a0
Feb  9 08:09:46 prod-server kernel: __do_page_fault+0x1e4/0x440
Feb  9 08:09:46 prod-server kernel: do_page_fault+0x37/0x12d
Feb  9 08:09:46 prod-server kernel: ? page_fault+0x8/0x30
Feb  9 08:09:46 prod-server kernel: page_fault+0x1e/0x30
Feb  9 08:09:46 prod-server kernel: RIP: 0033:0x7f0aa111e654
Feb  9 08:09:46 prod-server kernel: Code: 41 c6 87 88 04 00 00 01 4c 89 fe e8 06 ae c3 ff 41 c6 87 88 04 00 00 00 e9 67 fc ff ff 66 0f 1f 84 00 00 00 00 00 48 8b 40 28 <48> 83 00 01 e9 b8 fe ff ff 0f 1f 00 4c 89 ff e8 c8 e2 0f 00 48 8d
Feb  9 08:09:46 prod-server kernel: RSP: 002b:00007f0a453f3640 EFLAGS: 00010202
Feb  9 08:09:46 prod-server kernel: RAX: 00007f0aa27090c8 RBX: 00007f099c0e0680 RCX: 00007f099c0e0740
Feb  9 08:09:46 prod-server kernel: RDX: 0000000000000001 RSI: 0000000000000005 RDI: 000000060f1fdc30
Feb  9 08:09:46 prod-server kernel: RBP: 00007f0a453f36e0 R08: 0000000000000000 R09: 00000000199c16be
Feb  9 08:09:46 prod-server kernel: R10: 00007f0a453f34b0 R11: 0000000000000206 R12: 00007f0a5c989280
Feb  9 08:09:46 prod-server kernel: R13: 00007f0a5c0527d0 R14: 000000060f726700 R15: 00007f0a5c8d8800
Feb  9 08:09:46 prod-server kernel: CPU: 6 PID: 0 Comm: swapper/6 Kdump: loaded Not tainted 4.18.0-553.30.1.el8_10.x86_64 #1
Feb  9 08:09:46 prod-server kernel: Hardware name: Amazon EC2 c6in.2xlarge/, BIOS 1.0 10/16/2017
Feb  9 08:09:46 prod-server kernel: Call Trace:
Feb  9 08:09:46 prod-server kernel: <IRQ>
Feb  9 08:09:46 prod-server kernel: dump_stack+0x41/0x60
Feb  9 08:09:46 prod-server kernel: warn_alloc.cold.127+0x7b/0x108
Feb  9 08:09:46 prod-server kernel: __alloc_pages_slowpath+0xcb2/0xcd0
Feb  9 08:09:46 prod-server kernel: __alloc_pages_nodemask+0x2e2/0x330
Feb  9 08:09:46 prod-server kernel: __napi_alloc_skb+0x14c/0x190
Feb  9 08:09:46 prod-server kernel: ena_alloc_skb+0x48/0xa0 [ena]
Feb  9 08:09:46 prod-server kernel: 0
Feb  9 08:09:46 prod-server kernel: ena_clean_rx_irq+0x27a/0xd40 [ena]
Feb  9 08:09:46 prod-server kernel: ? kthread_delayed_work_timer_fn+0x41/0xa0
Feb  9 08:09:46 prod-server kernel: ? try_to_wake_up+0x190/0x4b0
Feb  9 08:09:46 prod-server kernel: ena_io_poll+0x30a/0x6e0 [ena]
Feb  9 08:09:46 prod-server kernel: ? __wake_up_common+0x7a/0x190
Feb  9 08:09:46 prod-server kernel: __napi_poll+0x2d/0x130
Feb  9 08:09:46 prod-server kernel: net_rx_action+0x252/0x320
Feb  9 08:09:46 prod-server kernel: ? credit_entropy_bits+0x1c3/0x260
Feb  9 08:09:46 prod-server kernel: __do_softirq+0xdc/0x2cf
Feb  9 08:09:46 prod-server kernel: irq_exit_rcu+0xc6/0xd0
Feb  9 08:09:46 prod-server kernel: irq_exit+0xa/0x10
Feb  9 08:09:46 prod-server kernel: do_IRQ+0x7f/0xd0
Feb  9 08:09:46 prod-server kernel: common_interrupt+0xf/0xf
Feb  9 08:09:46 prod-server kernel: </IRQ>
Feb  9 08:09:46 prod-server kernel: RIP: 0010:native_safe_halt+0xe/0x20
Feb  9 08:09:46 prod-server kernel: Code: 00 a8 08 75 be e9 23 ff ff ff 31 ff e9 6a ff ff ff 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 0f 00 2d 26 41 5e 00 fb f4 <c3> cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 0f 1f 44 00
Feb  9 08:09:46 prod-server kernel: RSP: 0018:ffffa2ca0194fe28 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffd9
Feb  9 08:09:46 prod-server kernel: RAX: 0000000080004000 RBX: ffff93891f3ab864 RCX: 00000000ffffffff
Feb  9 08:09:46 prod-server kernel: RDX: ffffffffb9fc6ea0 RSI: ffffffffbb8d1ca0 RDI: 0000000000000001
Feb  9 08:09:46 prod-server kernel: RBP: ffff93891f3ab864 R08: 0000000000000001 R09: ffff93891f3ab800
Feb  9 08:09:46 prod-server kernel: R10: 0000000000000010 R11: ffff938b1f1b2484 R12: 0000000000000001
Feb  9 08:09:46 prod-server kernel: R13: ffffffffbb8d1ca0 R14: 0000000000000001 R15: 0000000000000001
Feb  9 08:09:46 prod-server kernel: ? acpi_processor_thermal_init.cold.6+0x66/0x66
Feb  9 08:09:46 prod-server kernel: acpi_idle_do_entry+0x93/0xa0
Feb  9 08:09:46 prod-server kernel: acpi_idle_enter+0x5f/0xd0
Feb  9 08:09:46 prod-server kernel: cpuidle_enter_state+0x86/0x470
Feb  9 08:09:46 prod-server kernel: cpuidle_enter+0x2c/0x40
Feb  9 08:09:46 prod-server kernel: do_idle+0x26f/0x2d0
Feb  9 08:09:46 prod-server kernel: cpu_startup_entry+0x6f/0x80
Feb  9 08:09:46 prod-server kernel: start_secondary+0x187/0x1d0
Feb  9 08:09:46 prod-server kernel: secondary_startup_64_no_verify+0xd1/0xdb
Feb  9 08:09:46 prod-server kernel: CPU: 4 PID: 107 Comm: kswapd0 Kdump: loaded Not tainted 4.18.0-553.30.1.el8_10.x86_64 #1
Feb  9 08:09:46 prod-server kernel: Hardware name: Amazon EC2 c6in.2xlarge/, BIOS 1.0 10/16/2017
Feb  9 08:09:46 prod-server kernel: Call Trace:
Feb  9 08:09:46 prod-server kernel: <IRQ>
Feb  9 08:09:46 prod-server kernel: dump_stack+0x41/0x60
Feb  9 08:09:46 prod-server kernel: warn_alloc.cold.127+0x7b/0x108
Feb  9 08:09:46 prod-server kernel: __alloc_pages_slowpath+0xcb2/0xcd0
Feb  9 08:09:46 prod-server kernel: ? tcp_v4_rcv+0xd18/0xd90
Feb  9 08:09:46 prod-server kernel: __alloc_pages_nodemask+0x2e2/0x330
Feb  9 08:09:46 prod-server kernel: new_slab+0x3f4/0x4f0
Feb  9 08:09:46 prod-server kernel: ? tcp_v4_early_demux+0xfb/0x160
Feb  9 08:09:46 prod-server kernel: ___slab_alloc+0x3a3/0x950
Feb  9 08:09:46 prod-server kernel: ? tcp_v4_early_demux+0xfb/0x160
Feb  9 08:09:46 prod-server kernel: ? ip_rcv_finish+0x171/0x430
Feb  9 08:09:46 prod-server kernel: ? napi_skb_cache_get+0x47/0x60
Feb  9 08:09:46 prod-server kernel: kmem_cache_alloc_bulk+0x153/0x300
Feb  9 08:09:46 prod-server kernel: napi_skb_cache_get+0x47/0x60
Feb  9 08:09:46 prod-server kernel: __napi_build_skb+0x14/0x70
Feb  9 08:09:46 prod-server kernel: napi_build_skb+0x11/0x80
Feb  9 08:09:46 prod-server kernel: ena_alloc_skb+0x23/0xa0 [ena]
Feb  9 08:09:46 prod-server kernel: ena_clean_rx_irq+0x5cc/0xd40 [ena]
Feb  9 08:09:46 prod-server kernel: ? migrate_swap_stop+0x190/0x200
Feb  9 08:09:46 prod-server kernel: ? napi_gro_flush+0x68/0x90
Feb  9 08:09:46 prod-server kernel: ena_io_poll+0x30a/0x6e0 [ena]
Feb  9 08:09:46 prod-server kernel: ? __wake_up_common+0x7a/0x190
Feb  9 08:09:46 prod-server kernel: __napi_poll+0x2d/0x130
Feb  9 08:09:46 prod-server kernel: net_rx_action+0x252/0x320
Feb  9 08:09:46 prod-server kernel: ? __napi_schedule_irqoff+0x58/0x80
Feb  9 08:09:46 prod-server kernel: __do_softirq+0xdc/0x2cf
Feb  9 08:09:46 prod-server kernel: irq_exit_rcu+0xc6/0xd0
Feb  9 08:09:46 prod-server kernel: irq_exit+0xa/0x10
Feb  9 08:09:46 prod-server kernel: do_IRQ+0x7f/0xd0
Feb  9 08:09:46 prod-server kernel: common_interrupt+0xf/0xf
Feb  9 08:09:46 prod-server kernel: </IRQ>
Feb  9 08:09:46 prod-server kernel: RIP: 0010:list_lru_count_one+0x2d/0x50
Feb  9 08:09:46 prod-server kernel: Code: 00 00 48 63 f6 48 c1 e6 06 48 03 37 48 85 d2 74 22 48 63 82 b8 07 00 00 48 8b 56 20 48 85 d2 74 16 85 c0 78 12 48 8b 74 c2 10 <48> 8b 46 10 c3 cc cc cc cc 48 8b 46 20 48 83 c6 08 48 8b 46 10 c3
Feb  9 08:09:46 prod-server kernel: RSP: 0018:ffffa2ca021f3bf8 EFLAGS: 00000206 ORIG_RAX: ffffffffffffffdb
Feb  9 08:09:46 prod-server kernel: RAX: 000000000000003a RBX: 0000000000000400 RCX: ffff93891f1f1450
Feb  9 08:09:46 prod-server kernel: RDX: ffff93891f156000 RSI: ffff9388096736e0 RDI: ffff93891f1f14c0
Feb  9 08:09:46 prod-server kernel: RBP: ffff93891f1f1450 R08: ffff938800401240 R09: ffff9388004013a8
Feb  9 08:09:46 prod-server kernel: R10: 0000000000000000 R11: ffffffffbb857948 R12: ffffa2ca021f3cb0
Feb  9 08:09:46 prod-server kernel: R13: 0000000000000000 R14: 00000000006000c0 R15: ffffa2ca021f3cb0
Feb  9 08:09:46 prod-server kernel: super_cache_count+0x61/0x100
Feb  9 08:09:46 prod-server kernel: do_shrink_slab+0x3a/0x330
Feb  9 08:09:46 prod-server kernel: shrink_slab+0x249/0x2f0
Feb  9 08:09:46 prod-server kernel: shrink_node+0x257/0x710
Feb  9 08:09:46 prod-server kernel: balance_pgdat+0x2d7/0x550
Feb  9 08:09:46 prod-server kernel: kswapd+0x20b/0x3d0
Feb  9 08:09:46 prod-server kernel: ? finish_wait+0x80/0x80
Feb  9 08:09:46 prod-server kernel: ? balance_pgdat+0x550/0x550
Feb  9 08:09:46 prod-server kernel: kthread+0x134/0x150
Feb  9 08:09:46 prod-server kernel: ? set_kthread_struct+0x50/0x50
Feb  9 08:09:46 prod-server kernel: ret_from_fork+0x1f/0x40
Feb  9 08:09:46 prod-server kernel: SLUB: Unable to allocate memory on node -1, gfp=0x480020(GFP_ATOMIC)
Feb  9 08:09:46 prod-server kernel:  cache: skbuff_head_cache, object size: 256, buffer size: 256, default order: 1, min order: 0
Feb  9 08:09:46 prod-server kernel:  node 0: slabs: 92, objs: 2944, free: 1021
Feb  9 08:09:46 prod-server kernel:  cache: skbuff_head_cache, object size: 256, buffer size: 256, default order: 1, min order: 0
Feb  9 08:09:46 prod-server kernel: 12732
Feb  9 08:09:46 prod-server kernel:  node 0: slabs: 90, objs: 2880, free: 1178
Feb  9 08:09:46 prod-server kernel: 12732 12732
Feb  9 08:09:46 prod-server kernel: Node 0 Normal free:125888kB min:55848kB low:69808kB high:83768kB active_anon:1959936kB inactive_anon:2399192kB active_file:3746396kB inactive_file:4027488kB unevictable:0kB writepending:508kB present:13352960kB managed:13047068kB mlocked:0kB bounce:0kB free_pcp:932kB local_pcp:80kB free_cma:0kB
Feb  9 08:09:46 prod-server kernel: lowmem_reserve[]: 0 0 0 0 0
Feb  9 08:09:46 prod-server kernel: Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 2*2048kB (UM) 2*4096kB (M) = 13312kB
Feb  9 08:09:46 prod-server kernel: Node 0 DMA32: 1626*4kB (UMEH) 2288*8kB (UMEH) 902*16kB (UME) 540*32kB (UME) 243*64kB (UME) 89*128kB (UM) 12*256kB (UM) 1*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 87048kB
Feb  9 08:09:46 prod-server kernel: Node 0 Normal: 7974*4kB (UME) 3246*8kB (UMEH) 1589*16kB (UMEH) 772*32kB (UMEH) 174*64kB (UMEH) 24*128kB (UM) 1*256kB (H) 0*512kB 1*1024kB (H) 1*2048kB (H) 0*4096kB = 125528kB
Feb  9 08:09:46 prod-server kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Feb  9 08:09:46 prod-server kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Feb  9 08:09:46 prod-server kernel: 2403763 total pagecache pages
Feb  9 08:09:46 prod-server kernel: 11004 pages in swap cache
Feb  9 08:09:46 prod-server kernel: Swap cache stats: add 15336433, delete 15325859, find 10448748/11777568
Feb  9 08:09:46 prod-server kernel: Free swap  = 16042748kB
Feb  9 08:09:46 prod-server kernel: Total swap = 18431996kB
Feb  9 08:09:46 prod-server kernel: 4124551 pages RAM
Feb  9 08:09:46 prod-server kernel: 0 pages HighMem/MovableOnly
Feb  9 08:09:46 prod-server kernel: 158551 pages reserved
Feb  9 08:09:46 prod-server kernel: 0 pages hwpoisoned

[vladem]

Thanks for opening this issue. The kernel logs indicate memory exhaustion - the system ran out of available memory. This triggered the kernel's out-of-memory handler while Mountpoint's AwsEventLoop threads were active. To determine if Mountpoint is consuming excessive memory, could you provide per-process memory metrics during the copy operations:

• Mountpoint's memory usage via the process.memory_usage metric (extended debug log may also be useful)
• Memory consumption of other processes on the system

You can use basic system tools like top to capture this, or optionally configure the CloudWatch Agent procstat plugin for more detailed tracking of the mount-s3 and other processes during large file uploads.

This will help us assess whether Mountpoint's memory usage is abnormal or if the system is generally under-provisioned for the workload. Note that --write-part-size and --maximum-throughput-gbps CLI flags may be useful to reduce memory used by MP.

[mura890]

Thank you for your confirmation. Regarding the memory at the relevant time, I'm outputting the memory usage per process every 5 minutes using the dstat command and “smem -r -k | head -n 10”. I believe the following data may be helpful. What do you think?

----system---- ----total-usage---- -dsk/total- -net/total- ---paging-- ---system-- ------memory-usage----- ----swap--- nvme nvme
     time     |usr sys idl wai stl| read  writ| recv  send|  in   out | int   csw | used  free  buf   cach| used  free|util:util
09-02 08:07:41|  7   5  82   6   0| 125M   53k|1176k  128M|   0     0 | 591k  724k|4997M  160M  124k 9994M|2336M   15G| 100:0.24
09-02 08:07:51|  5   3  88   4   0|  58M   58k|5282k   55M|1.40  9.10 | 302k  358k|4940M 5859M  124k 4357M|2337M   15G|43.1:0.59
09-02 08:08:01|  3   0  97   0   0|1638B 3686B| 814k  247k|0.30     0 |  49k   68k|4941M 5856M  124k 4359M|2337M   15G|   0:0.02
09-02 08:08:11|  3   0  97   0   0|   0    50k| 795k  148k|   0     0 |  47k   65k|4941M 5855M  124k 4361M|2337M   15G|   0:0.22
09-02 08:08:21|  3   0  97   0   0|5736B 6145B| 805k  151k|   0     0 |  47k   66k|4940M 5854M  124k 4362M|2337M   15G|0.01:0.06
09-02 08:08:31|  2   0  97   0   0|   0  5324B| 801k   18k|   0     0 |  46k   65k|4941M 5853M  124k 4362M|2337M   15G|   0:0.01
09-02 08:08:41| 11   6  82   1   0| 208k    0 | 189M  157M|51.9     0 | 414k  136k|5637M 3682M  124k 5853M|2335M   15G|   0:0.35
09-02 08:08:51| 14   6  80   0   0| 819B   60k| 209M  219M|0.20     0 | 483k  155k|5613M 1625M  124k 7933M|2335M   15G|0.01:0.21
09-02 08:09:01| 14   6  79   1   0| 410B 3686B| 210M  220M|0.10     0 | 474k  154k|5619M  153M  124k 9399M|2335M   15G|   0:0.01
09-02 08:09:11| 14   7  77   1   0| 462k 1229B| 236M  221M|50.9     0 | 488k  159k|5632M  386M  124k 9160M|2334M   15G|   0:1.44
09-02 08:09:21| 12   6  81   1   0|  18k   68k| 185M  187M|2.40     0 | 426k  147k|5670M  425M  124k 9090M|2334M   15G|0.02:0.37
09-02 08:09:31|  9   4  86   0   0|  36k   13k| 107M  131M|8.90     0 | 287k  122k|5371M  440M  124k 9380M|2334M   15G|   0:0.20
09-02 08:09:41|  9   4  85   1   0| 327k 4198B| 133M  132M|15.5     0 | 306k  125k|5620M  394M  112k 9181M|2333M   15G|   0:0.58
09-02 08:09:51|  7   3  85   5   0|  16k   83k|  78M   86M|0.50     0 | 207k   93k|5613M  161M  112k 9422M|2333M   15G|0.02:0.38
09-02 08:10:01|  1   0  86  12   0|  34k  205k| 321k 9234B|   0  51.2 |  22k   33k|5618M  157M  112k 9422M|2335M   15G|   0:0.09
09-02 08:10:11|  2   0  85  12   0| 642k 2457B|1359k   21k| 160     0 |  35k   47k|5624M  155M  112k 9417M|2329M   15G|   0:0.88
09-02 08:10:21|  7   4  79   9   0|1613k   31k| 137M   77M| 153     0 | 263k   91k|6587M  266M  112k 8343M|2323M   15G|   0:3.88
09-02 08:10:31|  7   3  89   1   0| 574k   48k|  23M  107M| 142     0 | 214k  109k|4948M 1600M  112k 8648M|2317M   15G|   0:3.84
09-02 08:10:41|  3   0  96   1   0|2458B   10k|6290k  585k|0.10     0 |  57k   65k|4949M 1593M  112k 8654M|2317M   15G|   0:0.11
09-02 08:10:51|  2   0  97   0   0|   0    38k| 781k  102k|   0     0 |  43k   60k|4949M 1593M  112k 8655M|2317M   15G|   0:0.04
09-02 08:11:01|  2   0  97   0   0| 310k   56k| 786k  226k|   0     0 |  42k   58k|4949M 1583M  112k 8664M|2317M   15G|   0:0.27
09-02 08:11:11|  2   0  97   0   0|   0  1229B| 776k  131k|   0     0 |  43k   59k|4947M 1585M  112k 8665M|2317M   15G|   0:   0
09-02 08:11:21|  2   0  97   0   0|   0  9012B| 815k  125k|   0     0 |  45k   61k|4946M 1583M  112k 8667M|2317M   15G|   0:0.02
09-02 08:11:31|  2   0  97   0   0|   0  2457B| 816k  435k|   0     0 |  44k   59k|4948M 1574M  112k 8674M|2317M   15G|   0:0.02

2026/02/09 08:00:01
  PID User     Command                         Swap      USS      PSS      RSS 
1723581 root     /usr/local/WowzaStreamingEn   524.1M     1.5G     1.5G     1.6G 
3161006 tomcat   /usr/java/default/bin/java    525.6M   853.0M   853.1M   855.9M 
248604 eset-efs-scand /opt/eset/efs/lib/scand       101.1M   590.9M   592.0M   601.3M 
544600 root     /usr/bin/mount-s3 prod-S3-bucket    70.1M   546.9M   547.0M   549.7M 
  941 root     /usr/local/WowzaStreamingEn   269.2M   267.6M   272.6M   281.1M 
1228062 alloy    /usr/bin/alloy run --server     6.4M   113.2M   113.2M   114.9M 
  685 root     /usr/lib/systemd/systemd-jo    14.3M    31.4M    78.0M   143.5M 
306573 apache   /usr/sbin/httpd -DFOREGROUN   288.0K    63.8M    66.6M    82.1M 
248588 eset-efs-updated /opt/eset/efs/lib/updated       3.4M    62.3M    62.7M    68.4M 

2026/02/09 08:05:01
  PID User     Command                         Swap      USS      PSS      RSS 
1723581 root     /usr/local/WowzaStreamingEn   524.1M     1.5G     1.5G     1.6G 
3161006 tomcat   /usr/java/default/bin/java    525.6M   853.0M   853.1M   855.9M 
248604 eset-efs-scand /opt/eset/efs/lib/scand       101.1M   590.9M   591.9M   601.3M 
544600 root     /usr/bin/mount-s3 prod-S3-bucket    70.1M   546.9M   547.0M   549.7M 
  941 root     /usr/local/WowzaStreamingEn   269.2M   267.6M   272.6M   281.1M 
1228062 alloy    /usr/bin/alloy run --server     6.4M   112.9M   112.9M   114.6M 
  685 root     /usr/lib/systemd/systemd-jo    14.3M    31.4M    78.0M   143.5M 
306573 apache   /usr/sbin/httpd -DFOREGROUN   288.0K    63.8M    66.5M    82.1M 
248588 eset-efs-updated /opt/eset/efs/lib/updated       3.4M    62.3M    62.7M    68.4M 

2026/02/09 08:10:01
  PID User     Command                         Swap      USS      PSS      RSS 
1723581 root     /usr/local/WowzaStreamingEn   524.1M     1.5G     1.5G     1.6G 
544600 root     /usr/bin/mount-s3 prod-S3-bucket    69.1M     1.2G     1.2G     1.2G 
3161006 tomcat   /usr/java/default/bin/java    530.0M   847.7M   847.8M   850.6M 
248604 eset-efs-scand /opt/eset/efs/lib/scand       109.1M   582.9M   583.9M   593.3M 
  941 root     /usr/local/WowzaStreamingEn   271.2M   265.5M   270.6M   279.0M 
1228062 alloy    /usr/bin/alloy run --server     6.4M   112.4M   112.4M   114.1M 
  685 root     /usr/lib/systemd/systemd-jo    14.4M    31.3M    78.1M   143.7M 
306573 apache   /usr/sbin/httpd -DFOREGROUN   288.0K    63.8M    66.5M    82.1M 
248588 eset-efs-updated /opt/eset/efs/lib/updated       3.4M    62.3M    62.7M    68.4M 

2026/02/09 08:15:01
  PID User     Command                         Swap      USS      PSS      RSS 
1723581 root     /usr/local/WowzaStreamingEn   505.5M     1.6G     1.6G     1.6G 
3161006 tomcat   /usr/java/default/bin/java    530.0M   847.7M   847.8M   850.6M 
248604 eset-efs-scand /opt/eset/efs/lib/scand       109.1M   582.9M   584.0M   593.3M 
544600 root     /usr/bin/mount-s3 prod-S3-bucket    69.1M   549.8M   549.9M   552.6M 
  941 root     /usr/local/WowzaStreamingEn   271.2M   265.5M   270.6M   279.0M 
1228062 alloy    /usr/bin/alloy run --server     6.4M   110.4M   110.4M   112.1M 
  685 root     /usr/lib/systemd/systemd-jo    14.4M    31.3M    78.1M   143.7M 
306573 apache   /usr/sbin/httpd -DFOREGROUN   288.0K    63.8M    66.6M    82.1M 
248588 eset-efs-updated /opt/eset/efs/lib/updated       3.4M    62.3M    62.7M    68.4M 

We confirmed that mount-s3's memory usage surged from 500MB to 1.2GB between 08:05 and 08:10.

This occurred only once and has not happened since.
We attempted to reproduce the issue using the same specifications and mount-s3 version in a different test environment following the same steps, but it did not occur. Therefore, reproducibility is low, and it may not be an issue specific to mount-s3.

Translated with DeepL.com (free version)

[jet-tong]

Thanks for the detailed memory data - this is very helpful.

The memory surge from ~550 MB to ~1.2 GB is within expected range for uploading 16-17 GB files - this is normal for multi-part upload buffering. The kernel page allocation failure was likely triggered because overall free memory dropped to ~153 MB (at 08:09:01) during the uploads (I/O at ~200 MB/s send + recv), leaving little headroom for the page allocations.

The recommendations on adjusting --write-part-size and --maximum-throughput-gbps CLI flags still apply, which may be useful to reduce memory used by MP.

Given the low reproducibility this might just be a transient event as you suggested - do let us know if these recommendations help.