Seeking guidance regarding jittery behaviour observed after thousands of rounds

[sarkaria]

Royale and RL

I have been using Royale for several weeks now to implement a Reinforcement Learning (RL) agent. Training an RL agent requires running thousands of rounds and adjusting learning parameters from experience as it is sampled during battle.

Even for small state-spaces, many thousands of rounds are necessary. So while it is tempting to select max TPS, I realize the behaviour is compromised likely due to skipped turns.

So instead I run at about 200 TPS. And everything seems to be fine. I happily observe learning :-)
Until that is, after somewhere between 4000 and 6000 rounds, the simulation becomes jittery. The UI is seen to freeze, then some time later, momentarily unfreeze. And repeat. The tanks appear to process a short sequence of commands and freeze again. The rate of logging generated by my bot decreases suggests it is not a UI issue, though I could be wrong.

Any guidance that may allow me run simulations indefinitely would be appreciated :-).
Below I've provided some details of my configuration

Field	Value
Bot language	Python
Royale version	0.34.1
Battle config	1 on 1. Opponent bot is RamFire
HW	Macbook M3 with 16GB
References	#32

[flemming-n-larsen]

@sarkaria
It’s difficult to pinpoint exactly what’s going wrong, but I’ll share some pointers that might help. 🙂

Process isolation: The GUI runs in its own private process. It also starts the server and booter, which each run in separate processes. Your bot likewise runs in its own process. Because of this, it’s unlikely that the GUI or server are directly affected by your bot—unless there’s an unexpected bug.

Resource usage: The reinforcement process may consume significant system resources (CPU and memory). This can indirectly impact the performance of the server and GUI since they share the same operating system. I recommend opening Activity Monitor and checking the Java processes. You should be able to identify which ones correspond to your bot, the server, and the GUI, and then see how much CPU and memory each is using.

Bot console logs: If you have a console window open with logging enabled, this can sometimes destabilize the GUI. Try running without the bot console to see if stability improves. (This issue may be resolved in future updates.)

Standalone training: Consider running or training your bot using the server in standalone mode. Documentation is available on how to set this up.

Training rounds and state persistence: It may help to limit the number of training rounds and implement a way to save the bot’s state or “memory.” That way, you can resume training in subsequent sessions without starting from scratch.

[sarkaria]

Thank you!
I'll give those suggestions a try.

[flemming-n-larsen]

@sarkaria
I made some optimizations in the GUI recently. I hope this might help improving the jittery you have been experiencing?

You can tryout the newest version 0.35.1 here

[sarkaria]

Thank you Flemming! I look forward to try this out. (Sorry the New Year has started busy and I’ve not had a chance to resume my migration to Royale yet).

…

On Jan 27, 2026, at 2:33 PM, Flemming N. Larsen ***@***.***> wrote: @sarkaria <https://github.com/sarkaria> I made some optimizations in the GUI recently. I hope this might help improving the jittery you have been experiencing? You can tryout the newest version 0.35.1 here <https://github.com/robocode-dev/tank-royale/releases/tag/v0.35.1> — Reply to this email directly, view it on GitHub <#180 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHRB7XE2UOCJEWPY7KKNYF34I7RSDAVCNFSM6AAAAACPALI56KVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTKNRSGIZTQOI>. You are receiving this because you were mentioned.

[sarkaria]

Hello again @flemming-n-larsen,

Thank you for the version update. I gave it a try and ran into the same problem. However, this time I am able to provide some additional information! Maybe this will be of use:

1. In one experiment, I set up a battle between two RamFire bots running at 500 TPS. "Jittery" behaviour was observed after many many hours and thousands of rounds. While that in itself was not helpful, an OOM error message in the server log was. (See the attached server-log.rtf). After which no further GUI updates were observed.
2. Suspecting a memory problem, I ran a subsequent experiment in which I tracked the memory usage via the Mac's Activity app. The memory usage was seen to gradually increase.
3. jcmd <PID> GC.heap_info reported a total reserved value of 4194304K (4GB). I interpreted this to be the default heap size allocated to the server process.
4. When queried over the course of the battle, the used value reported by this command indicated an upward trend in memory consumption.
5. "Jittery" behaviour was observed as used neared total reserved.

I could be wrong, but it seems like memory is leaking. As usage nears the limit, latency increases in affected thread(s), which I guess causes jitters. In one (and only one case), I finally observed the thread finally failing with an java.lang.OutOfMemoryError.

What do you think?

server-log.rtf

[flemming-n-larsen]

@sarkaria Thank you taking time to investigate this further and also provide the server-log which is very helpful! ❤️

From the log we see:

...
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-3216473"\
Exception in thread "WebSocketSelector-3" java.lang.OutOfMemoryError: Java heap space\
...

The Java Virtual Machine (JVM) definitely runs out of memory, and it seems to be related to web sockets. So it might be a memory leak in the framework I use for web sockets. I will see if it is possible to work-around somehow.

I will have a look into it. But I cannot promise to find it and fix it quicky, as this can be a difficult issue to solve - especially figuring out where the issue occurs. But I will definitely try my best. 😊

[sarkaria]

Hello @flemming-n-larsen,
No problem!

• As a workaround I can increase the heap size when starting the server process. At least this way I know I can train my RL agent for a bit longer.

• FYI: My next step is to build a neural net into the RL agent that manages learning in the bot. I already have that code ready. I just need to integrate it.

[flemming-n-larsen]

@sarkaria
Just a short update, and good news. 😊
I have located the issue (memory leak), and is working on fixing it. I have good progress, and just need to test many battles and no side effects.
So with the next version, the issue should be fixed. 😊

[sarkaria]

Thats great news. Well done!

[flemming-n-larsen]

@sarkaria
I just released version 0.35.3, fixes multiple and hopefully all memory leaks with the server.
I ran 10.000 rounds (2x RamFire with max TPS), and I did not detect any leaks/changes with my memory profiler.

[sarkaria]

Thank you!