Skip to content

Conversation

Deigue
Copy link
Contributor

@Deigue Deigue commented Aug 7, 2025

Initial set of stopgap changes to enable DDR to work with the Open XL beta.

@Deigue Deigue force-pushed the openxl-ddr branch 2 times, most recently from c0a4493 to 0afd5d0 Compare August 11, 2025 15:30
@Deigue
Copy link
Contributor Author

Deigue commented Aug 11, 2025

Comment on lines 543 to 545
//newAttr->_stringdata = strndup(firstQuote + 1, secondQuote - firstQuote - 1);
//temporary placeholder to get around compilation, replace with alt implementation later.
newAttr->_stringdata = strdup(firstQuote + 1);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This cannot stay this way: we want just the characters between the quotes.

@Deigue
Copy link
Contributor Author

Deigue commented Aug 20, 2025

@keithc-ca @joransiu
So it looks like the path I want to run leading up to the DwarfParser logic should execute as follows: ->
Starting from ddrgen

if ((DDR_RC_OK == rc) && !options.debugFiles.empty()) {
rc = scanner.startScan(&portLibrary, &ir, &options.debugFiles, options.excludesFile);

DwarfScanner::startScan(OMRPortLibrary *portLibrary, Symbol_IR *ir, vector<string> *debugFiles, const char *excludesFilePath)
{
DEBUGPRINTF("Initializing libDwarf:");
DDR_RC rc = loadExcludesFile(portLibrary, excludesFilePath);
if (DDR_RC_OK == rc) {
/* Read list of debug files to scan from the input file. */
for (vector<string>::iterator it = debugFiles->begin(); it != debugFiles->end(); ++it) {
Symbol_IR newIR(ir);
rc = scanFile(portLibrary, &newIR, it->c_str());
if (DDR_RC_OK != rc) {
ERRMSG("Failure scanning %s\n", it->c_str());

DwarfScanner::scanFile(OMRPortLibrary *portLibrary, Symbol_IR *ir, const char *filepath)
{
OMRPORT_ACCESS_FROM_OMRPORT(portLibrary);
DDR_RC rc = DDR_RC_OK;
int res = DW_DLV_ERROR;
Dwarf_Error error = NULL;
intptr_t fd = omrfile_open(filepath, EsOpenRead, 0);
if (fd < 0) {
ERRMSG("Failure attempting to open %s\nExiting...\n", filepath);
rc = DDR_RC_ERROR;
}
if (DDR_RC_OK == rc) {
Dwarf_Handler errhand = 0;
Dwarf_Ptr errarg = NULL;
intptr_t native_fd = omrfile_convert_omrfile_fd_to_native_fd(fd);
DwarfScanner::scanFileName = filepath;
res = ddr_dw_init((int)native_fd, errhand, errarg, &_debug, &error);

ddr_dw_init(
int fd,
Dwarf_Handler errhand,
Dwarf_Ptr errarg,
Dwarf_Debug *dbg,
Dwarf_Error *error)
{
Dwarf_Unsigned access = DW_DLC_READ;
return dwarf_init(fd, access, errhand, errarg, dbg, error);

stringstream command;
command << toolpath << " " << DwarfScanner::getScanFileName() << " 2>&1";
fp = popen(command.str().c_str(), "r");

I've been trying to debug around these, I see from the make files that there is a executable ddrgen that is invoked/executed at some point, that should trigger the line of events above and begin process debug info files.
I had added a bunch of printfs to look around what is happening, but from my stdout logs I am only hitting the
about to startScan: checkpoint.

I do see a few indicators, like:

#if OMR_LIBDWARF_VERSION >= DW_LIBDWARF_MAKE_VERSION(0, 9)

which if not matched, it will bypass entering the intended logic above.
I also noticed that where Keith had suggested to add:

elseif(OMR_OS_ZOS AND (OMR_TOOLCONFIG STREQUAL "openxl"))
	set(default_debug_ext ".dwo")

Reviewing the references, it is only coming into play here?

set(dbg_file "${output_name}${OMR_DEBUG_INFO_OUTPUT_EXTENSION}")

(But the above is unapplicable code, as it is in the else() clause of OMR_OS_ZOS currently, this might be fine, assuming this is a function that is meant to separate debug symbols (which is already being done), I'm just not confident that specifying .dwo is being applied wherever it is actually needed wherever the code needs to know those are the type of files dwarfparser needs)

Going to try to enable the debug statements so that the already existing DEBUGPRINTF will come active, maybe that should help.

But any suggestions or ideas, in case I am straying the wrong path? Doesn't look like it is quite reaching the llvm-dwarfdump
logic at all quite yet ...

@Deigue
Copy link
Contributor Author

Deigue commented Aug 20, 2025

Hmm, it looks like it is not even getting past into the scanner code, because of this in ddrgen.cpp
!options.debugFiles.empty(): false

if ((DDR_RC_OK == rc) && !options.debugFiles.empty()) {
rc = scanner.startScan(&portLibrary, &ir, &options.debugFiles, options.excludesFile);

from where is this (ddrgen.cpp) invoked? Because it looks like the options parsed from the arguments is such that the debugFiles is completely empty, so it thinks there is no work to be done and skips over everything downstream.

I tried to find the command where it gets kicked off, but I am getting confused (or only seeing .mk or old build code that may not be applicable). The GNUmakefile seems to show how the target_files is passed, and perhaps the ddrgen is executed from here.
But basically something akin to ddr_artifacts.mk bash command that runs ddrgen is what I was trying to find to go from there, where the filelist is being included from the call being made.

I printed out the arguments being passed in to this executable being called, I was expecting the dwo file paths, but actually I got these:

8237 argv[0]: /jit/team/gauravc/repos/openj9-openjdk-jdk21-zos/build/zos-s390x-server-release/vm/runtime/omr_ddrgen
 8238 argv[1]: --show-empty
 8239 argv[2]: --macrolist
 8240 argv[3]: /jit/team/gauravc/repos/openj9-openjdk-jdk21-zos/build/zos-s390x-server-release/vm/ddr_info/sets/j9ddr.macros
 8241 argv[4]: --excludes
 8242 argv[5]: /jit/team/gauravc/repos/openj9-openjdk-jdk21-zos/openj9/runtime/ddr/excludes
 8243 argv[6]: --blob
 8244 argv[7]: /jit/team/gauravc/repos/openj9-openjdk-jdk21-zos/build/zos-s390x-server-release/vm/runtime/j9ddr.dat
 8245 argv[8]: --superset
 8246 argv[9]: /jit/team/gauravc/repos/openj9-openjdk-jdk21-zos/build/zos-s390x-server-release/vm/superset.dat
 8247 argv[10]: --overrides
 8248 argv[11]: /jit/team/gauravc/repos/openj9-openjdk-jdk21-zos/openj9/runtime/ddr/overrides

Reading the configure method, it expects a -f or --filelist for debug file paths like the dwo files, which doesnt seem to be passed at all.

@keithc-ca
Copy link
Contributor

not even getting past into the scanner code, because
!options.debugFiles.empty(): false

I don't see anything here that would select .dwo files instead of .debuginfo files. My understanding is that Open XL produces the former which would explain why none of the latter are found.

@Deigue Deigue force-pushed the openxl-ddr branch 3 times, most recently from 4f8ab1a to 7f76f1d Compare September 4, 2025 18:20
@Deigue
Copy link
Contributor Author

Deigue commented Sep 18, 2025

I am seeing some errors quite early in the build when trying to switch to the temporary directory libdwarf

-- Starting with CMake version 3.5.1
-- Found Git: /rsusr/miniconda/envs/stlaba0/bin/git (found version "2.14.4-12")
-- LIBELF_LIBRARY = /jit/team/gauravc/include/libdwarf/libelfdwarf64.x
-- Could NOT find LibElf (missing:  ELF_H_INCLUDE_DIR LIBELF_H_INCLUDE_DIR)
-- LIBDWARF_LIBRARY =
-- Found LibDwarf: /jit/team/gauravc/include/libdwarf/libdwarf.a
-- Found Perl: /rsusr/miniconda/envs/stlaba0/bin/perl (found version "5.24.4")
You have called ADD_LIBRARY for library j9dyn without any source files. This typically indicates a problem with your CMakeLists.txt file
-- Creating shared library for j9jit
-- Adding PRIVATE compiler options to j9jit for j9jit.
-- Configuring incomplete, errors occurred!

I tried changing the LIBELF_LIBRARY too, to correspond to the same .x file, but it doesnt seem to help the situation.
When I reverted to a omr master (baseline) build, it goes past the configuration steps and builds without problems, and it still shows LibElf missing, but the error message is a bit different - doesn't mention (missing: ELF_H_INCLUDE_DIR LIBELF_H_INCLUDE_DIR)

The changes I made to find LibDwarf seem to be doing the correct job, in that it is using the newer .x file I point it to, and the hints param helps it find the correct libdwarf.a and header files to pick up. But perhaps its affecting something related to finding LibElf and its related files, what would I need to change for LibElf if anything, so I can keep the way LibDwarf is loading from my temp directory with the newer library?

The CMakeError.log has some other error/noise, but I am not sure it is helpful:

Detecting CXX compiler ABI info failed to compile with the following output:
Change Dir: /jit/team/gauravc/repos/openj9-openjdk-jdk21-zos/build/zos-s390x-server-release/vm/CMakeFiles/CMakeTmp

Run Build Command:"/rsusr/zopen/usr/local/bin/make" "cmTC_df178/fast"
make[4]: Entering directory '/jit/team/gauravc/repos/openj9-openjdk-jdk21-zos/build/zos-s390x-server-release/vm/CMak
eFiles/CMakeTmp'
/rsusr/zopen/usr/local/bin/make -f CMakeFiles/cmTC_df178.dir/build.make CMakeFiles/cmTC_df178.dir/build
make[5]: Entering directory '/jit/team/gauravc/repos/openj9-openjdk-jdk21-zos/build/zos-s390x-server-release/vm/CMak
eFiles/CMakeTmp'
Building CXX object CMakeFiles/cmTC_df178.dir/CMakeCXXCompilerABI.cpp.o
/jit/team/gauravc/downloads/tony-0711/usr/lpp/IBM/cnw/v2r2/openxl/bin/ibm-clang++64      -o CMakeFiles/cmTC_df178.di
r/CMakeCXXCompilerABI.cpp.o -c /rsusr/openj9_resources/share/cmake-3.5/Modules/CMakeCXXCompilerABI.cpp
Linking CXX executable cmTC_df178
/rsusr/openj9_resources/bin/cmake -E cmake_link_script CMakeFiles/cmTC_df178.dir/link.txt --verbose=1
/jit/team/gauravc/downloads/tony-0711/usr/lpp/IBM/cnw/v2r2/openxl/bin/ibm-clang++64       -V CMakeFiles/cmTC_df178.d
ir/CMakeCXXCompilerABI.cpp.o  -o cmTC_df178
ibm-clang: error: unsupported option '-V CMakeFiles/cmTC_df178.dir/CMakeCXXCompilerABI.cpp.o'
ibm-clang: error: no input files
make[5]: *** [CMakeFiles/cmTC_df178.dir/build.make:98: cmTC_df178] Error 1
make[5]: Leaving directory '/jit/team/gauravc/repos/openj9-openjdk-jdk21-zos/build/zos-s390x-server-release/vm/CMake
Files/CMakeTmp'
make[4]: *** [Makefile:126: cmTC_df178/fast] Error 2
make[4]: Leaving directory '/jit/team/gauravc/repos/openj9-openjdk-jdk21-zos/build/zos-s390x-server-release/vm/CMake
Files/CMakeTmp'



Determining if the dladdr exist failed with the following output:
Change Dir: /jit/team/gauravc/repos/openj9-openjdk-jdk21-zos/build/zos-s390x-server-release/vm/CMakeFiles/CMakeTmp

Run Build Command:"/rsusr/zopen/usr/local/bin/make" "cmTC_971fc/fast"
make[4]: Entering directory '/jit/team/gauravc/repos/openj9-openjdk-jdk21-zos/build/zos-s390x-server-release/vm/CMak
eFiles/CMakeTmp'
/rsusr/zopen/usr/local/bin/make -f CMakeFiles/cmTC_971fc.dir/build.make CMakeFiles/cmTC_971fc.dir/build
make[5]: Entering directory '/jit/team/gauravc/repos/openj9-openjdk-jdk21-zos/build/zos-s390x-server-release/vm/CMak
eFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_971fc.dir/CheckSymbolExists.c.o

@Deigue
Copy link
Contributor Author

Deigue commented Sep 18, 2025

I am also able to confirm that it is successfully compiling with below command: (standalone)

cd /jit/team/gauravc/repos/openj9-openjdk-jdk21-zos/build/zos-s390x-server-release/vm/runtime/omr/ddr/lib/ddr-scanner && /jit/team/gauravc/downloads/tony-0711/usr/lpp/IBM/cnw/v2r2/openxl/bin/ibm-clang++64  -D__STDC_LIMIT_MACROS -D_AE_BIMODAL -D_ALL_SOURCE -D_OPEN_THREADS=3 -D_XOPEN_SOURCE=600 -DHAVE_LIBDWARF_DWARF_H -DHAVE_LIBDWARF_LIBDWARF_H -DIBM_ATOE -DIPv6_FUNCTION_SUPPORT -DJ9ZOS390 -DJ9ZOS39064 -DLONGLONG -DOPENJ9_BUILD -DSUPPORTS_THREAD_LOCAL -DZOS -fignore-exceptions -fstrict-aliasing -mzos-target=ZOSV2R5 -m64 -O3 -fstack-protector -g -gdwarf-5 -gsplit-dwarf -march=arch10 -std=gnu++14 -fasm -fno-rtti   -fexec-charset=ISO8859-1 -I/jit/team/gauravc/include -isystem /jit/team/gauravc/repos/openj9-openjdk-jdk21-zos/omr/cmake/modules/platform/os/../../../../util/a2e/headers -I/jit/team/gauravc/repos/openj9-openjdk-jdk21-zos/build/zos-s390x-server-release/vm/runtime -I/jit/team/gauravc/repos/openj9-openjdk-jdk21-zos/omr/ddr/include -I/jit/team/gauravc/repos/openj9-openjdk-jdk21-zos/omr/include_core -I/jit/team/gauravc/repos/openj9-openjdk-jdk21-zos/build/zos-s390x-server-release/vm/runtime/omr -I/usr/include -I/usr/lpp/cbclib/include -I/include_core  -o CMakeFiles/omr_ddr_scanner.dir/dwarf/DwarfScanner.cpp.o -c /jit/team/gauravc/repos/openj9-openjdk-jdk21-zos/omr/ddr/lib/ddr-scanner/dwarf/DwarfScanner.cpp

Just having problems having to use this include and make it to that part of the build with using the right libelfdwarf64.x

@Deigue Deigue force-pushed the openxl-ddr branch 6 times, most recently from d16fb74 to fea9ac5 Compare September 25, 2025 15:31
@Deigue Deigue force-pushed the openxl-ddr branch 2 times, most recently from 48b339a to 8ca4e95 Compare September 30, 2025 18:10
Deigue added 2 commits October 9, 2025 13:12
Merge/rebase prior changes:
- rebase eclipse/master 2025-09-17
- keep DwarfParser.cpp (debug changes, unused currently)
- cleanup old dwo related changes and merge commits

Signed-off-by: Gaurav Chaudhari <[email protected]>
Signed-off-by: Gaurav Chaudhari <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants