Unable to use `jhsdb jmap` to generate heap dump

[kzalewski11]

Describe the bug

Generating a heap dump with jhsdb jmap from either a live running JVM or from a core dump (generated using gdb) results in this error:

java.lang.ArrayIndexOutOfBoundsException: Array index out of range: 10
        at jdk.hotspot.agent/sun.jvm.hotspot.utilities.GenericArray.getAddressAt(GenericArray.java:97)
        at jdk.hotspot.agent/sun.jvm.hotspot.utilities.MethodArray.at(MethodArray.java:61)
        at jdk.hotspot.agent/sun.jvm.hotspot.oops.ConstMethod.getMethod(ConstMethod.java:131)
        at jdk.hotspot.agent/sun.jvm.hotspot.oops.ConstMethod.isNative(ConstMethod.java:439)
        at jdk.hotspot.agent/sun.jvm.hotspot.oops.ConstMethod.getLineNumberFromBCI(ConstMethod.java:305)
        at jdk.hotspot.agent/sun.jvm.hotspot.oops.Method.getLineNumberFromBCI(Method.java:279)
        at jdk.hotspot.agent/sun.jvm.hotspot.utilities.HeapHprofBinWriter.dumpStackFrame(HeapHprofBinWriter.java:867)
        at jdk.hotspot.agent/sun.jvm.hotspot.utilities.HeapHprofBinWriter.dumpStackTraces(HeapHprofBinWriter.java:846)
        at jdk.hotspot.agent/sun.jvm.hotspot.utilities.HeapHprofBinWriter.write(HeapHprofBinWriter.java:460)
        at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.writeHeapHprofBin(JMap.java:216)
        at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.run(JMap.java:103)
        at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:278)
        at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.start(Tool.java:241)
        at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.execute(Tool.java:134)
        at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.main(JMap.java:202)
        at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.runJMAP(SALauncher.java:340)
        at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.main(SALauncher.java:500)

This occurs both on our cloud instances and locally, and I suspect an issue with the jhsdb jmap tool. I have a colleague who has successfully run jmap commands successfully on the live JVM. Unfortunately this isn't a viable option for the OOM situation since the jmap option to run against a core dump file has been moved inside jhsdb.

To Reproduce

Directly from running JVM:
jhsdb jmap --pid <pid>
From core file:
jhsdb jmap --binaryheap --dumpfile <core_file> --exe <java_exe> --core <core_file>

Expected behavior

1. Successful creation of heap dump from live JVM
2. Successful generation of heap dump from core dump file

Platform information

OS: Ubuntu 24.04.2 LTS
Version Corretto-21.0.1.12.1

Additional context

We are currently starting our application with the JVM option -XX:+HeapDumpOnOutOfMemoryError. We've encountered a number of cases where the heap dump generation is taking much longer than expected, and has resulted in considerable downtime, sometimes around an hour or more for a ~6GB heap. As an attempt to work around this, I've been testing the option -XX:OnOutOfMemoryError to use gdb to take a core dump and subsequently parse out a heap dump from that, so we don't need to traverse the heap before restarting. This traversal is one of our suspects for why the heap dump is taking so long to generate.

[eastig]

Hi @kzalewski11,
Thank you for reporting the issue.

Based on the provided stack trace, the exception happened whilst processing stack frames. We tried to get details of a method a processed stack frame belonged to. We got a class which should hold the method. However the method was not found among methods of the class. It looks to me the issue might be application specific.

Do you have the issue only when you run your application?

[oehme]

This error only happened with that application. But running the "traditional" jmap tool against the live process works and -XX:+HeapDumpOnOutOfMemoryError works as well. jhsdb jmap --pid against the live process fails with the error described above. So the problem seems to be jhsdb specific. Do jhsdb jmap and "traditional" jmap use different code paths/mechanisms?

[eastig]

Do jhsdb jmap and "traditional" jmap use different code paths/mechanisms?

Looking into code I see they have different code paths. jmap just sends requests to JVM where all work is done. So not much is done in Java code. jhsdb gets access to JVM internals and works with them in Java code.

Your case might trigger a bug: internals seen in Java code are in the inconsistent state. This might happen if there is a data race: the Java code processes data which JVM is updating at the same time.

The only way to find the inconsistency is to use a debugger. When jhsdb fails, we need to check what is something specific about the class and the method it holds.

[eastig]

@kzalewski11 What architecture do you have the issue on? ARM64 or x86?

[shipilev]

Stepping back: From additional context it looks that you are trying to side-step the slow heap dump performance.

Corretto-21.0.1.12.1, or shortly 21.0.1, is quite old. Current release is 21.0.7.

With 21.0.1, you are likely having two problems:

• Encountering massive heap dump performance regression in JDK 21 GA (see JDK-8317692), the fix for which was backported to 21.0.2. The comments around that fix mention there was a 10x performance hit without the fix.
• Missing further improvements in heap dump performance (see JDK-8319650, JDK-8320924), both were backported in 21.0.4. The comments around those fixes say there is additional 3..5x improvement on the table.

So I think it is likely the normal heap dumping would start behaving reasonably well if you upgrade at least to 21.0.4, hopefully to 21.0.7. If we blindly multiply the best impacts from both problems above, you might see up to 50x faster heap dump, so 1 hour would become a few minutes. If successful, this would obviate the need to go jhsdb route altogether.

[kzalewski11]

@kzalewski11 What architecture do you have the issue on? ARM64 or x86?

x86.

Stepping back: From additional context it looks that you are trying to side-step the slow heap dump performance.

Corretto-21.0.1.12.1, or shortly 21.0.1, is quite old. Current release is 21.0.7.

With 21.0.1, you are likely having two problems:

* Encountering massive heap dump performance regression in JDK 21 GA (see [JDK-8317692](https://bugs.openjdk.org/browse/JDK-8317692)), the fix for which was backported to `21.0.2`. The comments around that fix mention there was a 10x performance hit without the fix.

* Missing further improvements in heap dump performance (see [JDK-8319650](https://bugs.openjdk.org/browse/JDK-8319650), [JDK-8320924](https://bugs.openjdk.org/browse/JDK-8320924)), both were backported in `21.0.4`. The comments around those fixes say there is additional 3..5x improvement on the table.

So I think it is likely the normal heap dumping would start behaving reasonably well if you upgrade at least to 21.0.4, hopefully to 21.0.7. If we blindly multiply the best impacts from both problems above, you might see up to 50x faster heap dump, so 1 hour would become a few minutes. If successful, this would obviate the need to go jhsdb route altogether.

This is good to know. I'll explore this option as I expect a small lift to just upgrade the patch version. Thanks!

[kzalewski11]

We've updated to the latest JDK21 and are monitoring the times taken to generate heap dumps for a while to see if our overarching issue is resolved.

We were curious if the jhsdb jmap ArrayIndexOutOfBoundsException was still present using the new version, so I gave it a quick try this morning and am still seeing the same result. So far I haven't seen excessive heap dump times so time will tell if this is worth continuing to investigate for our situation, but we figured it would be good to add this note in case there is any investigation taking place as to how this exception can happen.

[shipilev]

Some more context: jhsdb is a diagnostic interface that is primarily targeted to JVM developers, and it is often misbehaving in surprising ways. Using it is really a last-ditch option in the field, when everything else fails. In fact, the realization that users reach for "dump the core, extract heap dump from core" path due to poor normal heap dump performance was the reason why we spent time on JDK-8319650 and JDK-8320924.

So if normal heap dumps are reasonably fast for you, use normal paths and forget about jhsdb :)