Add the `HeapDumpGzipLevel` JVM option to compress heap dumps created by `HeapDumpOnOutOfMemoryError`

[ldematte]

Description

Today we specify -XX:+HeapDumpOnOutOfMemoryError to instruct the JVM to produce a heap dump before exiting in case of OOME. The heap dump is non-compressed; in order to save space on disk, we can specify this parameter to compress it, e.g. -XX:HeapDumpGzipLevel=9
Note: this will happend .gz to the dump filename, we need to make sure this is not an issue.

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[mosche]

@ldematte I suppose it's worth investigating how much this delays the generation of the heap dump to not negatively impact time to recovery of the container.
Unless storage on the host is critical, it might be advisable to compress the heap dump on a non critical path just before moving it to S3 / cloud storage.

[geekpete]

If there's any way we can easily perform multi-threaded compression (Eg like pigz vs gzip) with configurable core count % to use with a sensible default (eg for a host with a large number of cores or allocated processors, use more than one,etc), this would speed up the delay for a huge heap dump sending a single cpu core to 100% utilisation?

[rjernst]

@geekpete This is only in the case that Elasticsearch is already dying due to OOME, the utilization doesn't matter.

@ldematte I think there are tradeoffs here. Compression means more has to happen in the jvm before it can die, and therefore before we can start build our own zip and upload the dump. IMO speed to getting to that point is more important, and whether the dump is compressed by the jvm or not, we still have to have sufficient disk space for it to be written by the jvm.

[ecki]

BTW: I would suggest a moderate level since higher levels just give a minor compression improvement but make it slower. I think gzip default is 6.