JTReg Test Fail: sun/nio/cs/FindDecoderBugs.java

[M-Davies]

Failure link

JDK8/J9

• test category, openjdk
• OS/architecture, s390x_xl (https://ci.adoptopenjdk.net/job/Test_openjdk8_j9_sanity.openjdk_s390x_linux_xl/72/)

22:34:13  openjdk version "1.8.0_252"
22:34:13  OpenJDK Runtime Environment (build 1.8.0_252-202003251711-b07)
22:34:13  Eclipse OpenJ9 VM (build master-1218a31db, JRE 1.8.0 Linux s390x-64-Bit 20200325_111 (JIT enabled, AOT enabled)
22:34:13  OpenJ9   - 1218a31db
22:34:13  OMR      - 4175c2cba
22:34:13  JCL      - 22b5378e955 based on jdk8u252-b07)

Optional info

• intermittent failure (yes)
• New test

Failure output (captured from console output)

22:51:36  [2020-03-25 22:51:33,041] Agent[1]: stderr: Unhandled exception
22:51:36  [2020-03-25 22:51:33,041] Agent[1]: stderr: Type=Segmentation error vmState=0x000561ad
22:51:36  [2020-03-25 22:51:33,041] Agent[1]: stderr: J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=f65c8068 Signal_Code=00000001
22:51:36  [2020-03-25 22:51:33,041] Agent[1]: stderr: Handler1=000003FF9F8395A0 Handler2=000003FF9F620DA8 InaccessibleAddress=0000000000000000
22:51:36  [2020-03-25 22:51:33,041] Agent[1]: stderr: gpr0=0000000000000001 gpr1=0000000000000000 gpr2=0000000000000000 gpr3=000003FED4DF46F0
22:51:36  [2020-03-25 22:51:33,041] Agent[1]: stderr: gpr4=0000000000000004 gpr5=000003FED4E161C0 gpr6=000003FED4DF4640 gpr7=000003FF9CCF29D0
22:51:36  [2020-03-25 22:51:33,041] Agent[1]: stderr: gpr8=000003FF6594C300 gpr9=000003FED4E16180 gpr10=0000000000000000 gpr11=000003FF9CCF0890
22:51:36  [2020-03-25 22:51:33,041] Agent[1]: stderr: gpr12=000003FFA069B000 gpr13=000003FF9E79E9E0 gpr14=000003FF9E23932A gpr15=000003FF9CCEFEF0
22:51:36  [2020-03-25 22:51:33,041] Agent[1]: stderr: psw=000003FF9E239332 mask=0705000180000000 fpc=00880000 bea=000003FF9E232544
22:51:36  [2020-03-25 22:51:33,041] Agent[1]: stderr: fpr0 409a000000000000 (f: 0.000000, d: 1.664000e+03)
22:51:36  [2020-03-25 22:51:33,041] Agent[1]: stderr: fpr1 000003ff9ccf2ac0 (f: 2630822656.000000, d: 2.172101e-311)
22:51:36  [2020-03-25 22:51:33,041] Agent[1]: stderr: fpr2 0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:51:36  [2020-03-25 22:51:33,041] Agent[1]: stderr: fpr3 000003ff0000014e (f: 334.000000, d: 2.170802e-311)
22:51:36  [2020-03-25 22:51:33,041] Agent[1]: stderr: fpr4 409a000000000000 (f: 0.000000, d: 1.664000e+03)
22:51:36  [2020-03-25 22:51:33,041] Agent[1]: stderr: fpr5 3fcc723e7dcde3fb (f: 2110645248.000000, d: 2.222365e-01)
22:51:36  [2020-03-25 22:51:33,041] Agent[1]: stderr: fpr6 000003ff00000139 (f: 313.000000, d: 2.170802e-311)
22:51:36  [2020-03-25 22:51:33,041] Agent[1]: stderr: fpr7 3fe5558df34826af (f: 4081592064.000000, d: 6.666937e-01)
22:51:36  [2020-03-25 22:51:33,041] Agent[1]: stderr: fpr8 0000000000100000 (f: 1048576.000000, d: 5.180654e-318)
22:51:36  [2020-03-25 22:51:33,041] Agent[1]: stderr: fpr9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:51:36  [2020-03-25 22:51:33,041] Agent[1]: stderr: fpr10 000003ff9cbfd000 (f: 2629816320.000000, d: 2.172101e-311)
22:51:36  [2020-03-25 22:51:33,041] Agent[1]: stderr: fpr11 000003fff667f485 (f: 4134008064.000000, d: 2.172844e-311)
22:51:36  [2020-03-25 22:51:33,042] Agent[1]: stderr: fpr12 000003ffd677c31c (f: 3598172928.000000, d: 2.172579e-311)
22:51:36  [2020-03-25 22:51:33,042] Agent[1]: stderr: fpr13 000003ffdb3fdb18 (f: 3678395136.000000, d: 2.172619e-311)
22:51:36  [2020-03-25 22:51:33,042] Agent[1]: stderr: fpr14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
22:51:36  [2020-03-25 22:51:33,042] Agent[1]: stderr: fpr15 000003ff104fda78 (f: 273668736.000000, d: 2.170937e-311)
22:51:36  [2020-03-25 22:51:33,042] Agent[1]: stderr: Module=/home/jenkins/workspace/Test_openjdk8_j9_sanity.openjdk_s390x_linux_xl/openjdkbinary/j2sdk-image/jre/lib/s390x/default/libj9jit29.so
22:51:36  [2020-03-25 22:51:33,042] Agent[1]: stderr: Module_base_address=000003FF9DC80000
22:51:36  [2020-03-25 22:51:33,042] Agent[1]: stderr: 
22:51:36  [2020-03-25 22:51:33,042] Agent[1]: stderr: Method_being_compiled=sun/nio/cs/ext/GB18030$Decoder.decodeBufferLoop(Ljava/nio/ByteBuffer;Ljava/nio/CharBuffer;)Ljava/nio/charset/CoderResult;
22:51:36  [2020-03-25 22:51:33,042] Agent[1]: stderr: Target=2_90_20200325_111 (Linux 4.4.0-170-generic)
22:51:36  [2020-03-25 22:51:33,042] Agent[1]: stderr: CPU=s390x (4 logical CPUs) (0x1f723a000 RAM)
22:51:36  [2020-03-25 22:51:33,042] Agent[1]: stderr: ----------- Stack Backtrace -----------
22:51:36  [2020-03-25 22:51:33,042] Agent[1]: stderr: (0x000003FF9E239332 [libj9jit29.so+0x5b9332])
22:51:36  [2020-03-25 22:51:33,042] Agent[1]: stderr: (0x000003FF9E23A022 [libj9jit29.so+0x5ba022])
22:51:36  [2020-03-25 22:51:33,042] Agent[1]: stderr: (0x000003FF9E238986 [libj9jit29.so+0x5b8986])
22:51:36  [2020-03-25 22:51:33,042] Agent[1]: stderr: (0x000003FF9E23A022 [libj9jit29.so+0x5ba022])
22:51:36  [2020-03-25 22:51:33,042] Agent[1]: stderr: (0x000003FF9E238986 [libj9jit29.so+0x5b8986])
22:51:36  [2020-03-25 22:51:33,042] Agent[1]: stderr: (0x000003FF9E23A022 [libj9jit29.so+0x5ba022])
22:51:36  [2020-03-25 22:51:33,042] Agent[1]: stderr: (0x000003FF9E23F576 [libj9jit29.so+0x5bf576])
22:51:36  [2020-03-25 22:51:33,042] Agent[1]: stderr: (0x000003FF9E47E982 [libj9jit29.so+0x7fe982])
22:51:36  [2020-03-25 22:51:33,042] Agent[1]: stderr: (0x000003FF9E4FD41E [libj9jit29.so+0x87d41e])
22:51:36  [2020-03-25 22:51:33,042] Agent[1]: stderr: (0x000003FF9E4FD604 [libj9jit29.so+0x87d604])
22:51:36  [2020-03-25 22:51:33,042] Agent[1]: stderr: (0x000003FF9E4FCBA6 [libj9jit29.so+0x87cba6])
22:51:36  [2020-03-25 22:51:33,042] Agent[1]: stderr: (0x000003FF9E4FD23A [libj9jit29.so+0x87d23a])
22:51:36  [2020-03-25 22:51:33,042] Agent[1]: stderr: (0x000003FF9E3C3E34 [libj9jit29.so+0x743e34])
22:51:36  [2020-03-25 22:51:33,042] Agent[1]: stderr: (0x000003FF9E3C65D8 [libj9jit29.so+0x7465d8])
22:51:36  [2020-03-25 22:51:33,042] Agent[1]: stderr: (0x000003FF9E3C5894 [libj9jit29.so+0x745894])
22:51:36  [2020-03-25 22:51:33,043] Agent[1]: stderr: (0x000003FF9E3C6D80 [libj9jit29.so+0x746d80])
22:51:36  [2020-03-25 22:51:33,043] Agent[1]: stderr: (0x000003FF9E1769EC [libj9jit29.so+0x4f69ec])
22:51:36  [2020-03-25 22:51:33,043] Agent[1]: stderr: (0x000003FF9DDF07FA [libj9jit29.so+0x1707fa])
22:51:36  [2020-03-25 22:51:33,043] Agent[1]: stderr: (0x000003FF9DDF172A [libj9jit29.so+0x17172a])
22:51:36  [2020-03-25 22:51:33,043] Agent[1]: stderr: (0x000003FF9F622420 [libj9prt29.so+0x22420])
22:51:36  [2020-03-25 22:51:33,043] Agent[1]: stderr: (0x000003FF9DDF3430 [libj9jit29.so+0x173430])
22:51:36  [2020-03-25 22:51:33,043] Agent[1]: stderr: (0x000003FF9DDF3A76 [libj9jit29.so+0x173a76])
22:51:36  [2020-03-25 22:51:33,043] Agent[1]: stderr: (0x000003FF9DDEF448 [libj9jit29.so+0x16f448])
22:51:36  [2020-03-25 22:51:33,043] Agent[1]: stderr: (0x000003FF9DDEF710 [libj9jit29.so+0x16f710])
22:51:36  [2020-03-25 22:51:33,043] Agent[1]: stderr: (0x000003FF9DDEF7F8 [libj9jit29.so+0x16f7f8])
22:51:36  [2020-03-25 22:51:33,043] Agent[1]: stderr: (0x000003FF9F622420 [libj9prt29.so+0x22420])
22:51:36  [2020-03-25 22:51:33,043] Agent[1]: stderr: (0x000003FF9DDEFD1A [libj9jit29.so+0x16fd1a])
22:51:36  [2020-03-25 22:51:33,043] Agent[1]: stderr: (0x000003FF9F7058F0 [libj9thr29.so+0x58f0])
22:51:36  [2020-03-25 22:51:33,043] Agent[1]: stderr: (0x000003FFA0687934 [libpthread.so.0+0x7934])
22:51:36  [2020-03-25 22:51:33,043] Agent[1]: stderr: (0x000003FFA046DCE2 [libc.so.6+0xedce2])

Dump Files https://ibm.box.com/shared/static/bq89d6wwq34eg7j85gji425fc4r1ixxd.gz

[DanHeidinga]

Tagging as jit

22:51:36  [2020-03-25 22:51:33,041] Agent[1]: stderr: Type=Segmentation error vmState=0x000561ad
22:51:36  [2020-03-25 22:51:33,042] Agent[1]: stderr: Method_being_compiled=sun/nio/cs/ext/GB18030$Decoder.decodeBufferLoop(Ljava/nio/ByteBuffer;Ljava/nio/CharBuffer;)Ljava/nio/charset/CoderResult;
22:51:36  [2020-03-25 22:51:33,042] Agent[1]: stderr: Target=2_90_20200325_111 (Linux 4.4.0-170-generic)

fyi @fjeremic

[fjeremic]

Launching a grinder to find reproducibility rate:
https://ci.adoptopenjdk.net/job/Grinder/2585/

[fjeremic]

Grinder is showing 1/10 failure rate. Unfortunately debug symbols are not present in the JVM which ran this build (#8874 is tracking the fix), so I had to work around to get the symbols. I downloaded the same nightly build [1] and loaded the core file from the failure job.

gdb fails to load all the shared libraries because the path on the grinder machine is different. We can fix this by supplying the correct path via set solib-search-path to add the directory where all the missing shared libraries are supposed to be in the build we downloaded from [1] on our local system. gdb is then able to load the symbols from the shared library. Still however the bt command does not work for whatever reason. We do have an escape hatch though as the symbol addresses are printed in the original standard output of the failure, so we can create a quick gdb script:

define printBackTrace
info symbol 0x000003FF9E239332
info symbol 0x000003FF9E23A022
info symbol 0x000003FF9E238986
info symbol 0x000003FF9E23A022
info symbol 0x000003FF9E238986
info symbol 0x000003FF9E23A022
info symbol 0x000003FF9E23F576
info symbol 0x000003FF9E47E982
info symbol 0x000003FF9E4FD41E
info symbol 0x000003FF9E4FD604
info symbol 0x000003FF9E4FCBA6
info symbol 0x000003FF9E4FD23A
info symbol 0x000003FF9E3C3E34
info symbol 0x000003FF9E3C65D8
info symbol 0x000003FF9E3C5894
info symbol 0x000003FF9E3C6D80
info symbol 0x000003FF9E1769EC
info symbol 0x000003FF9DDF07FA
info symbol 0x000003FF9DDF172A
info symbol 0x000003FF9F622420
info symbol 0x000003FF9DDF3430
info symbol 0x000003FF9DDF3A76
info symbol 0x000003FF9DDEF448
info symbol 0x000003FF9DDEF710
info symbol 0x000003FF9DDEF7F8
info symbol 0x000003FF9F622420
info symbol 0x000003FF9DDEFD1A
info symbol 0x000003FF9F7058F0
info symbol 0x000003FFA0687934
info symbol 0x000003FFA046DCE2
end

Running this script yields:

>>> printBackTrace
TR_ForwardDFSetAnalysis<TR_BitVector*>::analyzeNodeIfPredecessorsAnalyzed(TR_RegionStructure*, TR_BitVector&) + 3570 in section .text of /sandbox1/fjeremic/temp/jdk8u252-b07/jre/lib/s390x/default/libj9jit29.so
TR_ForwardDFSetAnalysis<TR_BitVector*>::analyzeRegionStructure(TR_RegionStructure*, bool) + 858 in section .text of /sandbox1/fjeremic/temp/jdk8u252-b07/jre/lib/s390x/default/libj9jit29.so
TR_ForwardDFSetAnalysis<TR_BitVector*>::analyzeNodeIfPredecessorsAnalyzed(TR_RegionStructure*, TR_BitVector&) + 1094 in section .text of /sandbox1/fjeremic/temp/jdk8u252-b07/jre/lib/s390x/default/libj9jit29.so
TR_ForwardDFSetAnalysis<TR_BitVector*>::analyzeRegionStructure(TR_RegionStructure*, bool) + 858 in section .text of /sandbox1/fjeremic/temp/jdk8u252-b07/jre/lib/s390x/default/libj9jit29.so
TR_ForwardDFSetAnalysis<TR_BitVector*>::analyzeNodeIfPredecessorsAnalyzed(TR_RegionStructure*, TR_BitVector&) + 1094 in section .text of /sandbox1/fjeremic/temp/jdk8u252-b07/jre/lib/s390x/default/libj9jit29.so
TR_ForwardDFSetAnalysis<TR_BitVector*>::analyzeRegionStructure(TR_RegionStructure*, bool) + 858 in section .text of /sandbox1/fjeremic/temp/jdk8u252-b07/jre/lib/s390x/default/libj9jit29.so
TR_BasicDFSetAnalysis<TR_BitVector*>::performAnalysis(TR_Structure*, bool) + 158 in section .text of /sandbox1/fjeremic/temp/jdk8u252-b07/jre/lib/s390x/default/libj9jit29.so
TR_ReachingDefinitions::perform() + 178 in section .text of /sandbox1/fjeremic/temp/jdk8u252-b07/jre/lib/s390x/default/libj9jit29.so
TR_UseDefInfo::_runReachingDefinitions(TR_ReachingDefinitions&, TR_UseDefInfo::AuxiliaryData&) + 94 in section .text of /sandbox1/fjeremic/temp/jdk8u252-b07/jre/lib/s390x/default/libj9jit29.so
TR_UseDefInfo::performAnalysis(TR_UseDefInfo::AuxiliaryData&) + 164 in section .text of /sandbox1/fjeremic/temp/jdk8u252-b07/jre/lib/s390x/default/libj9jit29.so
TR_UseDefInfo::prepareUseDefInfo(bool, bool, bool, bool) + 3142 in section .text of /sandbox1/fjeremic/temp/jdk8u252-b07/jre/lib/s390x/default/libj9jit29.so
TR_UseDefInfo::TR_UseDefInfo(TR::Compilation*, TR::CFG*, TR::Optimizer*, bool, bool, bool, bool, bool, bool, bool) + 810 in section .text of /sandbox1/fjeremic/temp/jdk8u252-b07/jre/lib/s390x/default/libj9jit29.so
OMR::Optimizer::createUseDefInfo(TR::Compilation*, bool, bool, bool, bool, bool, bool) + 220 in section .text of /sandbox1/fjeremic/temp/jdk8u252-b07/jre/lib/s390x/default/libj9jit29.so
OMR::Optimizer::performOptimization(OptimizationStrategy const*, int, int, int) + 6936 in section .text of /sandbox1/fjeremic/temp/jdk8u252-b07/jre/lib/s390x/default/libj9jit29.so
OMR::Optimizer::performOptimization(OptimizationStrategy const*, int, int, int) + 3540 in section .text of /sandbox1/fjeremic/temp/jdk8u252-b07/jre/lib/s390x/default/libj9jit29.so
OMR::Optimizer::optimize() + 496 in section .text of /sandbox1/fjeremic/temp/jdk8u252-b07/jre/lib/s390x/default/libj9jit29.so
OMR::Compilation::compile() + 3236 in section .text of /sandbox1/fjeremic/temp/jdk8u252-b07/jre/lib/s390x/default/libj9jit29.so
TR::CompilationInfoPerThreadBase::compile(J9VMThread*, TR::Compilation*, TR_ResolvedMethod*, TR_J9VMBase&, TR_OptimizationPlan*, TR::SegmentAllocator const&) + 1282 in section .text of /sandbox1/fjeremic/temp/jdk8u252-b07/jre/lib/s390x/default/libj9jit29.so
TR::CompilationInfoPerThreadBase::wrappedCompile(J9PortLibrary*, void*) + 1026 in section .text of /sandbox1/fjeremic/temp/jdk8u252-b07/jre/lib/s390x/default/libj9jit29.so
omrsig_protect + 872 in section .text of /sandbox1/fjeremic/temp/jdk8u252-b07/jre/lib/s390x/default/libj9prt29.so
TR::CompilationInfoPerThreadBase::compile(J9VMThread*, TR_MethodToBeCompiled*, J9::J9SegmentProvider&) + 1088 in section .text of /sandbox1/fjeremic/temp/jdk8u252-b07/jre/lib/s390x/default/libj9jit29.so
TR::CompilationInfoPerThread::processEntry(TR_MethodToBeCompiled&, J9::J9SegmentProvider&) + 590 in section .text of /sandbox1/fjeremic/temp/jdk8u252-b07/jre/lib/s390x/default/libj9jit29.soTR::CompilationInfoPerThread::processEntries() + 1088 in section .text of /sandbox1/fjeremic/temp/jdk8u252-b07/jre/lib/s390x/default/libj9jit29.so
TR::CompilationInfoPerThread::run() + 56 in section .text of /sandbox1/fjeremic/temp/jdk8u252-b07/jre/lib/s390x/default/libj9jit29.so
protectedCompilationThreadProc(J9PortLibrary*, TR::CompilationInfoPerThread*) + 160 in section .text of /sandbox1/fjeremic/temp/jdk8u252-b07/jre/lib/s390x/default/libj9jit29.so
omrsig_protect + 872 in section .text of /sandbox1/fjeremic/temp/jdk8u252-b07/jre/lib/s390x/default/libj9prt29.so
compilationThreadProc(void*) + 578 in section .text of /sandbox1/fjeremic/temp/jdk8u252-b07/jre/lib/s390x/default/libj9jit29.so
thread_wrapper + 280 in section .text of /sandbox1/fjeremic/temp/jdk8u252-b07/jre/lib/s390x/default/libj9thr29.so
pthread_create@@GLIBC_2.2 + 3044 in section .text of /lib/s390x-linux-gnu/libpthread.so.0
putsgent + 258 in section .text of /lib/s390x-linux-gnu/libc.so.6

So the issue happens in the optimizer while creating usedef info. @PushkarBettadpur could you please help the optimizer team investigate this one? Let's help the experts collect trace logs with necessary tracing information so we can determine what went wrong. As a first step please reach out to @andrewcraik and/or @cathyzhyi to figure out what tracing options we may need, then let's help collect the logging.

[1] https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u-2020-03-26-04-11/OpenJDK8U-jdk_s390x_linux_openj9_linuxXL_2020-03-26-04-11.tar.gz

[andrewcraik]

The dataflow engine has been very stable - I would suggest running traceFull paranoidOptCheck and see if we can make sure the trees are valid before the opt.

[PushkarBettadpur]

Took a stab at this recently and unfortunately what I'm finding is that once I start adding any tracing options, the failure isn't reproducible anymore. Perhaps traceFull might be too heavy in this case?

[fjeremic]

The following is a grinder from a nightly build a few days ago which shows the failure is still present:
https://ci.adoptopenjdk.net/job/Grinder/2917

This problem is effectively impossible to reproduce with logging as the benchmark runs only for several seconds and is very sensitive to any tracing. Even with a specific inlining plan I was not able to get a reproduction of the issue massaging the command line.

However #9120 to the rescue! Using my latest set of prototype changes to improvements to the jitdump I am able to gather the necessary logs with traceFull,paranoidOptCheck. Unfortunately paranoid opt check does not seem to catch any issues here. I'm attaching the jitdump with tracing usedef generation as well as BVA:

jitdump.20200427.140209.43974.0004.zip

The crash happens here:

   │0x3ff90d3eb5e <_ZN23TR_ForwardDFSetAnalysisIP12TR_BitVectorE33analyzeNodeIfPredecessorsAnalyzedEP18TR_RegionStructureRS0_+3526> brasl  %r14,0x3ff90ceb2c0 <_ZN12TR_BitVector5printEPN2TR│   
   │0x3ff90d3eb64 <_ZN23TR_ForwardDFSetAnalysisIP12TR_BitVectorE33analyzeNodeIfPredecessorsAnalyzedEP18TR_RegionStructureRS0_+3532> lg     %r2,184(%r15)                                    │   
   │0x3ff90d3eb6a <_ZN23TR_ForwardDFSetAnalysisIP12TR_BitVectorE33analyzeNodeIfPredecessorsAnalyzedEP18TR_RegionStructureRS0_+3538> lg     %r1,200(%r15)                                    │   
   │0x3ff90d3eb70 <_ZN23TR_ForwardDFSetAnalysisIP12TR_BitVectorE33analyzeNodeIfPredecessorsAnalyzedEP18TR_RegionStructureRS0_+3544> lgf    %r4,96(%r1)                                      │   
   │0x3ff90d3eb76 <_ZN23TR_ForwardDFSetAnalysisIP12TR_BitVectorE33analyzeNodeIfPredecessorsAnalyzedEP18TR_RegionStructureRS0_+3550> lg     %r3,72(%r2)                                      │   
   │0x3ff90d3eb7c <_ZN23TR_ForwardDFSetAnalysisIP12TR_BitVectorE33analyzeNodeIfPredecessorsAnalyzedEP18TR_RegionStructureRS0_+3556> brasl  %r14,0x3ff90d37d70 <_ZN21TR_BasicDFSetAnalysisIP1│   
   │0x3ff90d3eb82 <_ZN23TR_ForwardDFSetAnalysisIP12TR_BitVectorE33analyzeNodeIfPredecessorsAnalyzedEP18TR_RegionStructureRS0_+3562> l      %r4,24(%r11)                                     │   
   │0x3ff90d3eb86 <_ZN23TR_ForwardDFSetAnalysisIP12TR_BitVectorE33analyzeNodeIfPredecessorsAnalyzedEP18TR_RegionStructureRS0_+3566> lgr    %r10,%r2                                         │  
  >│0x3ff90d3eb8a <_ZN23TR_ForwardDFSetAnalysisIP12TR_BitVectorE33analyzeNodeIfPredecessorsAnalyzedEP18TR_RegionStructureRS0_+3570> l      %r2,20(%r2)                                      │   
   │0x3ff90d3eb8e <_ZN23TR_ForwardDFSetAnalysisIP12TR_BitVectorE33analyzeNodeIfPredecessorsAnalyzedEP18TR_RegionStructureRS0_+3574> c      %r4,24(%r10)                                     │   
   │0x3ff90d3eb92 <_ZN23TR_ForwardDFSetAnalysisIP12TR_BitVectorE33analyzeNodeIfPredecessorsAnalyzedEP18TR_RegionStructureRS0_+3578> je     0x3ff90d3f0ae <_ZN23TR_ForwardDFSetAnalysisIP12TR│   
   │0x3ff90d3eb96 <_ZN23TR_ForwardDFSetAnalysisIP12TR_BitVectorE33analyzeNodeIfPredecessorsAnalyzedEP18TR_RegionStructureRS0_+3582> l      %r3,16(%r11)                                     │   
   │0x3ff90d3eb9a <_ZN23TR_ForwardDFSetAnalysisIP12TR_BitVectorE33analyzeNodeIfPredecessorsAnalyzedEP18TR_RegionStructureRS0_+3586> c      %r3,16(%r10)                                     │   
   │0x3ff90d3eb9e <_ZN23TR_ForwardDFSetAnalysisIP12TR_BitVectorE33analyzeNodeIfPredecessorsAnalyzedEP18TR_RegionStructureRS0_+3590> jh     0x3ff90d3f2ee <_ZN23TR_ForwardDFSetAnalysisIP12TR│   
   │0x3ff90d3eba2 <_ZN23TR_ForwardDFSetAnalysisIP12TR_BitVectorE33analyzeNodeIfPredecessorsAnalyzedEP18TR_RegionStructureRS0_+3594> ltr    %r4,%r4                                          │

Which seems to correspond to the following source code location:
https://github.com/eclipse/omr/blob/cfd669060771dd827724daa1622b625ab5aa85aa/compiler/optimizer/BitVectorAnalysis.cpp#L1319

The call (brasl) immediately preceeding the crash is a call to getContainer which gets the TR_BitVector object which we use for the comparison. The equality operator of the TR_BitVector class can be found here:
https://github.com/eclipse/omr/blob/master/compiler/infra/BitVector.hpp#L594-L611

Which gets inlined into the crashing function. We can see this because the first thing we do in the equality operator is compare the value of _lastChunkWithNonZero, which happens to be at offset 24 in this JVM:
https://github.com/eclipse/omr/blob/cfd669060771dd827724daa1622b625ab5aa85aa/compiler/infra/BitVector.hpp#L847-L855

That is, (8 + 8 + 4 + 4). And this aligns with the instruction sequence which loads 24 bytes off from the object returned by getContainer. It appears the the bitvector returned by getContainer was NULL for some reason.

@andrewcraik I think we'll need an expert in this area to take a look at the log. Transferring the issue over to you. I can collect additional tracing if needed via the jitdump mechanism. @vijaysun-omr FYI for jitdump work paying dividends already.