Excessive memory use on startup, to the point of exhaustion; progressively worsening over time?

[SubordinalBlue]

Minecraft Version

26.1.2

Modloader

NeoForge 26.1.2.75

Mod version

26.1.2.4

Latest version?

• Yes

Repeatable?

• Yes

Link to log files

Crash report: https://mclo.gs/iANQOY7
Latest log: https://mclo.gs/h5OdfP4

Steps to reproduce the issue

Unknown.

It's "repeatable" in the sense that one of our two Craftoria 2 servers was exhausting the heap on every attempt to start up.

(more details below)

Description of the issue you are having

The story so far:

Our Craftoria 2 server for North America (C2:NA), began exhausting the heap on startup a few days ago; the other server, EU, is fine and has roughly as active a player base as the NA server

The logs were not helpful in pinpointing the root cause then.

A fresh world would start fine, without apparent heap usage in excess of normal.

At that time, changing the GC algo allowed the server to squeeze in a full startup before exhausting the memory we currently have on that host.

Today it began to exhaust the heap on startup again.

Updating the server to our latest pack version at least got us a proper watchdog timeout and crash-report.

Still heap exhaustion, but stack trace led me to try removing Accelerated Decay, and that did remove the problem. Memory usage is as expected now.

Given all that, I suspect a problem rooted in the world state, that can get progressively worse, linked to Accelerated Decay.

Will share more details if I find them; not sure what to look for in the world data.

Information

• I have provided as much information as possible

[SubordinalBlue]

Forgot to add some specific numbers: this is with the host currently capping this server at 20G.
For comparision, this server and world are currently, with Accelerated Decay removed, running along with one player at 5GB used.

[MichaelHillcox]

Please add spark and run a 120 second profiler spark profiler start --timeout 120 and provide me that report

[SubordinalBlue]

Will do… is there a way to get spark to start profiling very early in startup?Server wasn’t even able to complete startup and take any commands.I realize this sounds odd, to suspect AD; I also need to put it back on with that particular world and see if goes back to not completing startup before crashing on heap exhaustion, for a little more certain diagnosis. _< edit to remove fluff from replying via email >_

[SubordinalBlue]

Update:

1. Putting Accelerated Decay back in, with that particular world, brings back the same behavior: server exhausts the ~20G heap before it will respond to commands; though it otherwise seems to have just about finished it's startup.

2. Using a spark config of:

{
  "backgroundProfiler": false,
  "compressBackgroundProfilerOutput": true
}

and then waiting for the watchdog timeout to kill the server (tough even then the hosting panel thinks it still needs a hard kill, so open file handles I assume), nets me several .jfr.tmp files in the 'config/spark/tmp/' folder.

I've used jfr . assemble ./compiled_profile.jfr on my local machine to produce a proper .jfr:

jfr tmps and assembled, from Craftoria 2-NA world.zip

TBH, getting into learning territory here for me, about spark and jfr, so that's good.

Cheers.

[MichaelHillcox]

The file attached appears to have a 0kb dump which isn't super helpful :P

What is your watchdog / max ticks set to in your config?

Alternatively, if you'd like, you can send over your server to me and I can debug for you.

My discord is 'Righthuggingnow'

[SubordinalBlue]

Well that's a 'doh!' for me, not catching the resulting .jfr is an empty file.

Hmm... this world that's seemingly not happy with AD, or vice versa, is about 12G in size.

I may start a download of the whole thing from the hosted server it's on tomorrow, so using a java agent or some other harness might allow getting some useful metrics on what's going on.