jvb wont recover when theres a gc pause or gc stall ( memory leak or underspec?)

[vlausargs]

my project start with this

• i had 6 jvb (spec: cpu Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz(80 core) and 256 gb dedicated server)
• with plan to handle 500-600 participant in one room
• all of them should turn on cam (can be monitored by admin when scrolled to their grid)
• channelLastN = 28 (full screen grid)
• video res already capped to 180p
• codec VP8

on the 1st try i set it to 100 participant on each jvb but sometimes it freeze and kick people out of room
it wont recover at all and should just disband the meeting.

i also notice theres a udp packet loss and its correlation with long gc pause (more than 1500ms) when using G1GC

• i tried to tweak it by set MaxGCPauseMillis=100
• set Xmx and Xms to 16G then change it 32G, 64G, 128G and i notice it always full then high gc pause
• switch to ZGC start with Xmx 16G then gradually change to 200G also always full heap and cause GC Stall

maybe i cannot force 100 participant on 1 jvb so,
with ZGC, i also tried to lower each jvb capacity to 65, same issue, GC stall

how do i cause gc pause or stall?

• when all jvb under full capacity (last test 65 on each jvb) just scroll video grid from start to last page (tried both aggresive direct to bottom or each scroll wait until next grid render)

heres grafana (cpu, memory, network, udp error) chart for reference when gc stall happen

at this point my suspicions was to memory leak or with current server spec its not possible with 65 participant each jvb

any thoughts on this?

update: i see some intresting memory chart adding elastic apm, i will up date this issue later