Startup: Limit metaspace max size

[spinscale]

With the introduction of the metaspace replacing the perm gen space in java 8, the way I understand it, that now native memory outside of the heap is used for this area. Also it is configured to be unbounded by default, so if a classloader is leaky (main question is, can this happen, potentially by plugins?), we might not only go OOM with the VM, but with the whole operating system/container.

We might want to check if we can configure -XX:MaxMetaspaceSize on startup. Maybe there are other tools like a leak detector (I've seen once on github, not sure how useful, seemed to be for servlets) or this is not a problem at all the way we do things.

Putting it up for discussion thus.

[danielmitterdorfer]

+1. Unbounded growth for finite resources is a recipe for disaster.

[robgil]

+1 for less sleepless nights

[danielmitterdorfer]

Despite my initial enthusiasm I don't recall a single instance in the last years where metaspace usage was problematic. Hence, I'd opt for going with the JVM's defaults for now.

[rjernst]

Let's close then. If compelling evidence to add this comes up, we can open a new issue.

[original-brownbear]

I'd like to reopen this.

We've seen some unexpected memory use patterns on cloud lately which hit the 137 OOM kill eventually. In a number of instances the thing that finally tipped things over the edge was the C1 or C2 compiler thread.
When profiling ES and running some high load indexing I'm seeing about 100M used for meta space and the number keeps increasing the more things I do (searches etc) and thus the more code gets compiled.

For low memory nodes of say 2G physical memory that are then run at 1G heap size and 500M direct buffer limit per out recommendations, I would assume that there is a good chance that a resize of the meta space could lead to trouble.
Maybe we should set something like 256M for both the size as well as the max size of the meta space to avoid resizing and it growing boundlessly after all (or at least do so for smaller heap size setups where it's a significant proportion of the overall memory used)?

[colings86]

This was discussed in a FixIt meeting where there were comments that limiting the metaspace can be problematic as the effect to getting it wrong is that the JVM dies. The amount of metaspace we need is a moving target as we add classes, plugins get installed, Painless scripts, etc. which makes it hard to ensure we pick the right value. Additionally there is the elastic metaspace JEP coming in JDK 16 so the metaspace would grow and shrink as needed which should help here.

The code cache however is something we should look into limiting. With the code cache, the worst consequence is cached compiled code is evicted and needs to be recompiled, a performance hit rather than killing the JVM. This really only arises on small instances though as on the large instances there’s enough native memory outside the heap to accommodate metaspace and the code cache, etc.

We agreed that we should investigate limiting the code cache size but not the metaspace size for now

[mark-vieira]

@rjernst since this is ergonomics stuff, would this be a core/infra issue?