Unhandled exception Type=Segmentation error

[Jenson3210]

Hi,

We're using 24.0.0.11-full-java17-openj9-ubi to run. However, I think also 24.0.0.12-full-java17-openj9-ubi would not work.

We're seeing crashes with error log:

Unhandled exception
Type=Segmentation error vmState=0x00040000
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
...
----------- Stack Backtrace -----------
decodeStackFrameDataFromStackMapTable+0x1a (0x00007F7808D6024A [libj9vrb29.so+0x1124a])
generateJ9RtvExceptionDetails+0x9a5 (0x00007F7808D5F0F5 [libj9vrb29.so+0x100f5])
j9bcv_createVerifyErrorString+0x332 (0x00007F780A80C932 [libj9vm29.so+0x17c932])
classInitStateMachine+0xab5 (0x00007F780A6B3225 [libj9vm29.so+0x23225])
resolveStaticMethodRefInto+0x2c8 (0x00007F780A707BD8 [libj9vm29.so+0x77bd8])
resolveStaticMethodRef+0x22 (0x00007F780A707DB2 [libj9vm29.so+0x77db2])

We think it might be related to this bug.
According to the details there, it should be resolved by semeru 17.0.14.0
However, it seems that there is no version available other than the one used in the dockerfile
that contains semeru 17.0.14.

Our startup logs contain

Launching defaultServer (Open Liberty 24.0.0.11/wlp-1.0.95.cl241120241021-1102) on Eclipse OpenJ9 VM, version 17.0.13+11 (en_US)

showing it is 17.0.13 only.

Now, i could not find a way to raise this question/issue on ibm ubi image that is used in the dockerfile, so raised it here instead hoping for some insights.

[leochr]

@Jenson3210 The error message and the method match the referenced bug/APAR, but the stacktrace is slightly different. So I don't know for sure that the fix for the APAR would resolve your error. Semeru team would likely know.

Repository for Semeru is at https://github.com/ibmruntimes/Semeru-Runtimes/issues

As per the release roadmap for Semeru, 17.0.14.0 is planned for February 2025. Semeru containers are typically available 1-2 weeks after the binary (non-container) release. Liberty images can be expected to be updated within 1-2 weeks of Semeru container release.

[Jenson3210]

@leochr, Thanks a lot for the quick answer. Raised our question over there also.

At this moment testing an e2e pipeline every 30 minutes with 23.0.0.4-full-java17-openj9-ubi, in which we did not get this error so far.
So either the liberty version difference or the semeru runtime internally which might be the root cause here.

[leochr]

I checked 23.0.0.4-full-java17-openj9-ubi and it included JDK 17.0.6+10. I am not sure whether changes in Semeru or Liberty runtime (or a combination of the two) is the root cause.

[JorisNens]

@leochr Our pipeline failed already several times now but we are not able to retreive the dump file.
The dump file logged is not there (or visisble) at that location. In the OpenLiberty documentation I found following command:
server dump defaultServer --include=heap,system,thread

I tried it on a healthy and I get a dump file. But on a crashed container the command does not finish (even after one hour), I only get a empty directory.
I also tried without the include tags. And I also tried the javadump version. But the commands never finish.
Can you advice on this?

[leochr]

@JorisNens Heap and system dumps are heavyweights. The thread dump is lightweight and commonly useful. Can you try with --include=thread to see if that works?

[JorisNens]

@leochr I tried with --include=thread only but the same issue. The command does not stop. In the OpenShift metrics view I see a small increase in memory but I do not see an increase in cpu (<10mCore usage).

[leochr]

Kevin (@kgibm), adding you, in case you have any suggestions or insights to debug/resolve the issue with gathering dump. Thank you.

[kgibm]

Most commonly with containers, crash core dumps go to the worker node: https://eclipse.dev/openj9/docs/xdump/#piped-system-dumps

Check cat /proc/sys/kernel/core_pattern and review the link above to see where.

As far as the hanging server dump, I've seen this once before in containers but the customer couldn't reproduce. What we would need is, to start, javacores of both the Liberty and server dump processes using kill -3 $PID after the hang.

[leochr]

Thank you Kevin!

@JorisNens, please try what Kevin suggested and provide the artifacts so we can investigate further. Thank you.

[jorisnenscolruyt]

Hi sorry for the late response.
The kill does seems to generate a file: javacore.20250106.154207.1.0003.txt
I tried to convert with jpackcore but the fileformat is unknown. I also tried to convert to hprof, but was not able to.

I see some company details in the file, so I first have to check internally if I can share the txt file.

[kgibm]

tried to convert with jpackcore but the fileformat is unknown. I also tried to convert to hprof, but was not able to.

A javacore is a thread dump which is just a small text file that shows basic information about a Java process such as stack traces of all threads: https://eclipse.dev/openj9/docs/dump_javadump/

You can either review it in a text editor or in a free tool such as the IBM Thread and Monitor Dump Analyzer.

jpackcore is for operating system coredumps produced by a crash or some other mechanism that produces a J9 System Dump such as server dump --include=system. By default, kill -3 produces javacores, not system dumps. If you want to produce system dumps on kill instead of other techniques such as server dump, then I suggest starting the JVM with this argument:

-Xdump:system:events=user2,request=exclusive+prepwalk

Then execute kill -USR2 $PID to request a system dump.

I see some company details in the file, so I first have to check internally if I can share the txt file.

Yes, please scrub any confidential information. For the issue of server dump being hung, the main things we're interested in are the stacks of the threads and any lock contention, so we'd just want the 0SECTION THREADS section with the thread stacks and the 0SECTION LOCKS section for lock information.

[jorisnenscolruyt]

javacore.20250106.154207.1.0003_clean2.txt
In attachment the javacore file

I changed the setting to collect the system dump with the Kill command. I will send a new javacore file when the crash happens again.

[kgibm]

@jorisnenscolruyt Was this taken during the server dump hang? If so, then I don't see that the command reached Liberty, so that suggests the server dump itself is hung. If so, then please reproduce with a javacore of the server dump process itself using kill -3 $PID after it's hung.

I changed the setting to collect the system dump with the Kill command. I will send a new javacore file when the crash happens again.

A javacore and system dump will be produced automatically on a crash, so you do not need to execute the kill command in that condition. My understanding is that you were just trying to test system dumps (since that's what the Java team needs to evaluate the crash) using Liberty's server dump but server dump is hanging. The kill -USR2 command is an alternative to test the production of system dumps instead of using server dump (although we would also like to investigate why server dump is hanging).

[jorisnenscolruyt]

I was able to trigger a new javacore dump:
javacore.20250107.200210.1.0003 copy.txt
This was with performing a kill -3 command.
Then I did a server dump, but it hung. I performed the kill -3 command again and I received this file:

javacore.20250107.200605.1.0004 copy.txt

When the server crashed a dump is automaticly created but the dump creation fails:

JVMDUMP039I Processing dump event "gpf", detail "" at 2025/01/07 19:23:22 - please wait.
JVMDUMP032I JVM requested System dump using '/opt/ol/wlp/output/defaultServer/core.20250107.192322.1.0001.dmp' in response to an event
JVMPORT030W /proc/sys/kernel/core_pattern setting "|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h" specifies that the core dump is to be piped to an external program. Attempting to rename either core or core.205. Review the manual for the external program to find where the core dump is written and ensure the program does not truncate it.
JVMPORT049I The core file created by child process with pid = 205 was not found. Review the documentation for the /proc/sys/kernel/core_pattern program "|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h" to find where the core file is written and ensure that program does not truncate it.
JVMDUMP012E Error in System dump: /opt/ol/wlp/output/defaultServer/core.20250107.192322.1.0001.dmp

That is why I try to generate a dump with the kill command or server dump command.

The following error is printed in the server logs:

Unhandled exception
Type=Segmentation error vmState=0x00040000
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
Handler1=00007FBC90ECFD60 Handler2=00007FBC90E28750 InaccessibleAddress=0000000000000000
RDI=00007FBC351F0090 RSI=0000000000000000 RAX=0000000000000000 RBX=00007FBC8C0F4320
RCX=00007FBC351F0120 RDX=000000000000FFFF R8=00007FBC8C0F4320 R9=0000000000000050
R10=00000000FFFFFFFF R11=0000000000000001 R12=00007FBC351F0120 R13=00007FBC351F0070
R14=00007FBC351F0090 R15=0000000000000001
RIP=00007FBC9024524A GS=0000 FS=0000 RSP=00007FBC351F0000
EFlags=0000000000010246 CS=0033 RBP=00007FBC351F0090 ERR=0000000000000004
TRAPNO=000000000000000E OLDMASK=0000000000000000 CR2=0000000000000000
xmm0=9271f630b53fb101 (f: 3040850176.000000, d: -7.950421e-220)
xmm1=9271f63000000000 (f: 0.000000, d: -7.950417e-220)
xmm2=00000000b7654321 (f: 3076866816.000000, d: 1.520174e-314)
xmm3=0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm4=00000000ff000000 (f: 4278190080.000000, d: 2.113707e-314)
xmm5=0000003000000020 (f: 32.000000, d: 1.018558e-312)
xmm6=00000000ff000000 (f: 4278190080.000000, d: 2.113707e-314)
xmm7=0000000001a16730 (f: 27354928.000000, d: 1.351513e-316)
xmm8=00000000c0112200 (f: 3222348288.000000, d: 1.592052e-314)
xmm9=24245b9facd5b18c (f: 2899685888.000000, d: 1.400442e-134)

[kgibm]

I was able to trigger a new javacore dump:
javacore.20250107.200210.1.0003 copy.txt
This was with performing a kill -3 command.
Then I did a server dump, but it hung. I performed the kill -3 command again and I received this file:
javacore.20250107.200605.1.0004 copy.txt

These are from the same PID 1 which is Liberty. The server dump process is a separate Java process. You'll have to look in ps (or under /proc if you don't have ps installed) to find the PID of server dump.

JVMDUMP039I Processing dump event "gpf", detail "" at 2025/01/07 19:23:22 - please wait.
JVMDUMP032I JVM requested System dump using '/opt/ol/wlp/output/defaultServer/core.20250107.192322.1.0001.dmp' in response to an event

This means the system dump went to systemd-coredump on the worker node. You should find it on the OpenShift worker node, most commonly at /var/lib/systemd/coredump/. You'll need a cluster-admin role to access the worker node.