[Do not merge][15_1_X] Enable ROOT debug #9761

makortel · 2025-03-27T18:06:28Z

For debugging in cms-sw/cmssw#47697 (comment)

Cherry-picked from #9733

makortel · 2025-03-27T18:06:35Z

cmsbuild · 2025-03-27T18:06:50Z

A new Pull Request was created by @makortel for branch IB/CMSSW_15_1_X/master.

@iarspider, @smuzaffar can you please review it and eventually sign? Thanks.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

cmsbuild · 2025-03-27T18:06:51Z

cms-bot internal usage

cmsbuild · 2025-03-28T07:22:55Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-387b8e/45251/summary.html
COMMIT: 670c837
CMSSW: CMSSW_15_1_X_2025-03-27-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9761/45251/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found 1 errors in the following unit tests:

---> test testDiMuonBiasesPlotting had ERRORS

Comparison Summary

Summary:

You potentially added 12 lines to the logs
Reco comparison results: 14 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3909207
DQMHistoTests: Total failures: 99
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3909088
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 215 log files, 184 edm output root files, 50 DQM output files
TriggerResults: no differences found

makortel · 2025-04-07T17:31:24Z

@cmsbuild, please test

Previous build was removed

makortel · 2025-04-07T17:51:23Z

@smuzaffar I'm wondering if something along please build would be useful for cases like here where we're really interested only in the build itself and the tests could be skipped?

cmsbuild · 2025-04-07T21:37:19Z

-1

Failed Tests: Build
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-387b8e/45423/summary.html
COMMIT: 670c837
CMSSW: CMSSW_15_1_X_2025-04-07-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9761/45423/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@cms-sw Update cmssw-wm-tools.spec #9776

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-387b8e/45423/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-387b8e/45423/git-merge-result

Build

I found compilation error when building:

/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/bin/c++ -O3 -pthread -pipe -Werror=main -Werror=pointer-arith -Werror=overlength-strings -Wno-vla -Werror=overflow -std=c++20 -ftree-vectorize -Werror=array-bounds -Werror=format-contains-nul -Werror=type-limits -fvisibility-inlines-hidden -fno-math-errno --param vect-max-version-for-alias-checks=50 -Xassembler --compress-debug-sections -Wno-error=array-bounds -Warray-bounds -fuse-ld=bfd -march=x86-64-v3 -felide-constructors -fmessage-length=0 -Wall -Wno-non-template-friend -Wno-long-long -Wreturn-type -Wextra -Wpessimizing-move -Wclass-memaccess -Wno-cast-function-type -Wno-unused-but-set-parameter -Wno-ignored-qualifiers -Wno-unused-parameter -Wunused -Wparentheses -Werror=return-type -Werror=missing-braces -Werror=unused-value -Werror=unused-label -Werror=address -Werror=format -Werror=sign-compare -Werror=write-strings -Werror=delete-non-virtual-dtor -Werror=strict-aliasing -Werror=narrowing -Werror=unused-but-set-variable -Werror=reorder -Werror=unused-variable -Werror=conversion-null -Werror=return-local-addr -Wnon-virtual-dtor -Werror=switch -fdiagnostics-show-option -Wno-unused-local-typedefs -Wno-attributes -Wno-psabi -Wno-error=unused-variable -DALPAKA_DEFAULT_HOST_MEMORY_ALIGNMENT=128 -DALPAKA_DISABLE_VENDOR_RNG -DBOOST_DISABLE_ASSERTS -flto=auto -fipa-icf -flto-odr-type-merging -fno-fat-lto-objects -Wodr -shared -Wl,-E    -Wl,-z,defs     tmp/el8_amd64_gcc12/src/CalibCalorimetry/EcalCorrelatedNoiseAnalysisModules/src/CalibCalorimetryEcalCorrelatedNoiseAnalysisModules/EcnaAnalyzer.cc.o tmp/el8_amd64_gcc12/src/CalibCalorimetry/EcalCorrelatedNoiseAnalysisModules/src/CalibCalorimetryEcalCorrelatedNoiseAnalysisModules/SealModule.cc.o -o tmp/el8_amd64_gcc12/src/CalibCalorimetry/EcalCorrelatedNoiseAnalysisModules/src/CalibCalorimetryEcalCorrelatedNoiseAnalysisModules/libCalibCalorimetryEcalCorrelatedNoiseAnalysisModules.so -Wl,-E -Wl,--hash-style=gnu -Wl,--as-needed -Wl,-z,noexecstack -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_1_X_2025-04-07-1100/biglib/el8_amd64_gcc12 -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_1_X_2025-04-07-1100/lib/el8_amd64_gcc12 -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_1_X_2025-04-07-1100/external/el8_amd64_gcc12/lib -L/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02884/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_15_1_X_2025-04-07-1100/biglib/el8_amd64_gcc12 -L/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02884/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_15_1_X_2025-04-07-1100/lib/el8_amd64_gcc12 -L/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02884/el8_amd64_gcc12/cms/cmssw/CMSSW_15_1_X_2025-04-06-0000/lib/el8_amd64_gcc12 -L/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02884/el8_amd64_gcc12/cms/cmssw/CMSSW_15_1_X_2025-04-06-0000/biglib/el8_amd64_gcc12 -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_1_X_2025-04-07-1100/static/el8_amd64_gcc12 -L/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02884/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_15_1_X_2025-04-07-1100/static/el8_amd64_gcc12 -L/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02884/el8_amd64_gcc12/cms/cmssw/CMSSW_15_1_X_2025-04-06-0000/static/el8_amd64_gcc12 -lDataFormatsEcalDigi -lDataFormatsEcalDetId -lFWCoreFramework -lDataFormatsDetId -lFWCoreCommon -lFWCoreServiceRegistry -lDataFormatsCommon -lFWCoreParameterSet -lFWCoreAbstractServices -lFWCoreMessageLogger -lDataFormatsProvenance -lCalibCalorimetryEcalCorrelatedNoiseAnalysisAlgos -lFWCorePluginManager -lFWCoreReflection -lFWCoreVersion -lFWCoreConcurrency -lHeterogeneousCoreAlpakaInterface -lFWCoreUtilities -lGui -lGpad -lGraf -lPhysics -lHist -lMatrix -lTree -lNet -lThread -lMathCore -lRint -lRIO -lboost_program_options -lCore -lboost_thread -lboost_date_time -lpng -lpcre -lbz2 -ljpeg -lturbojpeg -luuid -ltbb -llzma -lz -lfmt -lcms-md5 -lcrypt -ldl -lrt -lstdc++fs -ltinyxml2
Leaving library rule at CalibCalorimetry/EcalCorrelatedNoiseAnalysisModules
@@@@ Running EDM write config for CalibCalorimetryEcalCorrelatedNoiseAnalysisModules
edmWriteConfigs: /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.11-864308583194219b57d821e91f95ee35/root-6.32.11/interpreter/llvm-project/clang/lib/Serialization/ASTReader.cpp:6482: void clang::ASTReader::ReadPragmaDiagnosticMappings(clang::DiagnosticsEngine&): Assertion `IDAndOffset.first.isValid() && "invalid FileID for transition"' failed.
/bin/sh: line 1: 2579202 Aborted                 (core dumped) edmWriteConfigs -p /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_1_X_2025-04-07-1100/tmp/el8_amd64_gcc12/src/CalibCalorimetry/EcalCorrelatedNoiseAnalysisModules/src/CalibCalorimetryEcalCorrelatedNoiseAnalysisModules/libCalibCalorimetryEcalCorrelatedNoiseAnalysisModules.so
gmake: *** [lib/el8_amd64_gcc12/CalibCalorimetryEcalCorrelatedNoiseAnalysisModules.edmplugin] Error 1
>> Leaving Package CalibCalorimetry/EcalCorrelatedNoiseAnalysisModules
>> Package CalibCalorimetry/EcalCorrelatedNoiseAnalysisModules built
>> Entering Package CalibCalorimetry/EcalPedestalOffsets
>> Leaving Package CalibCalorimetry/EcalPedestalOffsets
>> Package CalibCalorimetry/EcalPedestalOffsets built

makortel · 2025-04-07T22:07:47Z

@pcanal The ROOT debug build failed with an assertion failure in

root-6.32.11/interpreter/llvm-project/clang/lib/Serialization/ASTReader.cpp:6482: void clang::ASTReader::ReadPragmaDiagnosticMappings(clang::DiagnosticsEngine&): Assertion `IDAndOffset.first.isValid() && "invalid FileID for transition"' failed.

makortel · 2025-04-07T22:07:58Z

@cmsbuild, please test

Let's see if this reproduces

cmsbuild · 2025-04-08T02:13:49Z

-1

Failed Tests: Build
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-387b8e/45428/summary.html
COMMIT: 670c837
CMSSW: CMSSW_15_1_X_2025-04-07-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9761/45428/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@cms-sw Update cmssw-wm-tools.spec #9776

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-387b8e/45428/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-387b8e/45428/git-merge-result

Build

I found compilation error when building:

/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/bin/c++ -O3 -pthread -pipe -Werror=main -Werror=pointer-arith -Werror=overlength-strings -Wno-vla -Werror=overflow -std=c++20 -ftree-vectorize -Werror=array-bounds -Werror=format-contains-nul -Werror=type-limits -fvisibility-inlines-hidden -fno-math-errno --param vect-max-version-for-alias-checks=50 -Xassembler --compress-debug-sections -Wno-error=array-bounds -Warray-bounds -fuse-ld=bfd -march=x86-64-v3 -felide-constructors -fmessage-length=0 -Wall -Wno-non-template-friend -Wno-long-long -Wreturn-type -Wextra -Wpessimizing-move -Wclass-memaccess -Wno-cast-function-type -Wno-unused-but-set-parameter -Wno-ignored-qualifiers -Wno-unused-parameter -Wunused -Wparentheses -Werror=return-type -Werror=missing-braces -Werror=unused-value -Werror=unused-label -Werror=address -Werror=format -Werror=sign-compare -Werror=write-strings -Werror=delete-non-virtual-dtor -Werror=strict-aliasing -Werror=narrowing -Werror=unused-but-set-variable -Werror=reorder -Werror=unused-variable -Werror=conversion-null -Werror=return-local-addr -Wnon-virtual-dtor -Werror=switch -fdiagnostics-show-option -Wno-unused-local-typedefs -Wno-attributes -Wno-psabi -Wno-error=unused-variable -DALPAKA_DEFAULT_HOST_MEMORY_ALIGNMENT=128 -DALPAKA_DISABLE_VENDOR_RNG -DBOOST_DISABLE_ASSERTS -flto=auto -fipa-icf -flto-odr-type-merging -fno-fat-lto-objects -Wodr -shared -Wl,-E    -Wl,-z,defs     tmp/el8_amd64_gcc12/src/CalibCalorimetry/EcalCorrelatedNoiseAnalysisModules/src/CalibCalorimetryEcalCorrelatedNoiseAnalysisModules/EcnaAnalyzer.cc.o tmp/el8_amd64_gcc12/src/CalibCalorimetry/EcalCorrelatedNoiseAnalysisModules/src/CalibCalorimetryEcalCorrelatedNoiseAnalysisModules/SealModule.cc.o -o tmp/el8_amd64_gcc12/src/CalibCalorimetry/EcalCorrelatedNoiseAnalysisModules/src/CalibCalorimetryEcalCorrelatedNoiseAnalysisModules/libCalibCalorimetryEcalCorrelatedNoiseAnalysisModules.so -Wl,-E -Wl,--hash-style=gnu -Wl,--as-needed -Wl,-z,noexecstack -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_1_X_2025-04-07-1100/biglib/el8_amd64_gcc12 -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_1_X_2025-04-07-1100/lib/el8_amd64_gcc12 -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_1_X_2025-04-07-1100/external/el8_amd64_gcc12/lib -L/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02884/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_15_1_X_2025-04-07-1100/biglib/el8_amd64_gcc12 -L/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02884/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_15_1_X_2025-04-07-1100/lib/el8_amd64_gcc12 -L/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02884/el8_amd64_gcc12/cms/cmssw/CMSSW_15_1_X_2025-04-06-0000/lib/el8_amd64_gcc12 -L/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02884/el8_amd64_gcc12/cms/cmssw/CMSSW_15_1_X_2025-04-06-0000/biglib/el8_amd64_gcc12 -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_1_X_2025-04-07-1100/static/el8_amd64_gcc12 -L/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02884/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_15_1_X_2025-04-07-1100/static/el8_amd64_gcc12 -L/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02884/el8_amd64_gcc12/cms/cmssw/CMSSW_15_1_X_2025-04-06-0000/static/el8_amd64_gcc12 -lDataFormatsEcalDigi -lDataFormatsEcalDetId -lFWCoreFramework -lDataFormatsDetId -lFWCoreCommon -lFWCoreServiceRegistry -lDataFormatsCommon -lFWCoreParameterSet -lFWCoreAbstractServices -lFWCoreMessageLogger -lDataFormatsProvenance -lCalibCalorimetryEcalCorrelatedNoiseAnalysisAlgos -lFWCorePluginManager -lFWCoreReflection -lFWCoreVersion -lFWCoreConcurrency -lHeterogeneousCoreAlpakaInterface -lFWCoreUtilities -lGui -lGpad -lGraf -lPhysics -lHist -lMatrix -lTree -lNet -lThread -lMathCore -lRint -lRIO -lboost_program_options -lCore -lboost_thread -lboost_date_time -lpng -lpcre -lbz2 -ljpeg -lturbojpeg -luuid -ltbb -llzma -lz -lfmt -lcms-md5 -lcrypt -ldl -lrt -lstdc++fs -ltinyxml2
Leaving library rule at CalibCalorimetry/EcalCorrelatedNoiseAnalysisModules
@@@@ Running EDM write config for CalibCalorimetryEcalCorrelatedNoiseAnalysisModules
edmWriteConfigs: /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.11-864308583194219b57d821e91f95ee35/root-6.32.11/interpreter/llvm-project/clang/lib/Serialization/ASTReader.cpp:6482: void clang::ASTReader::ReadPragmaDiagnosticMappings(clang::DiagnosticsEngine&): Assertion `IDAndOffset.first.isValid() && "invalid FileID for transition"' failed.
/bin/sh: line 1: 1645205 Aborted                 (core dumped) edmWriteConfigs -p /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_1_X_2025-04-07-1100/tmp/el8_amd64_gcc12/src/CalibCalorimetry/EcalCorrelatedNoiseAnalysisModules/src/CalibCalorimetryEcalCorrelatedNoiseAnalysisModules/libCalibCalorimetryEcalCorrelatedNoiseAnalysisModules.so
gmake: *** [lib/el8_amd64_gcc12/CalibCalorimetryEcalCorrelatedNoiseAnalysisModules.edmplugin] Error 1
>> Leaving Package CalibCalorimetry/EcalCorrelatedNoiseAnalysisModules
>> Package CalibCalorimetry/EcalCorrelatedNoiseAnalysisModules built
>> Entering Package CalibCalorimetry/EcalPedestalOffsets
>> Leaving Package CalibCalorimetry/EcalPedestalOffsets
>> Package CalibCalorimetry/EcalPedestalOffsets built

smuzaffar · 2025-04-08T07:38:13Z

@smuzaffar I'm wondering if something along please build would be useful for cases like here where we're really interested only in the build itself and the tests could be skipped?

@makortel , I agree, it will a be a nice thing to add. We can add please build(|only) or enable build-only extra comment so that please test only do the build part ( externals and cmssw)

makortel · 2025-04-08T13:20:52Z

Thanks @smuzaffar!

makortel · 2025-04-08T13:22:05Z

@pcanal The edmWriteConfigs failure seems reproducible (and we probably should investigate it separately).

The /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9761/45428/install.sh might be sufficient to continue debugging cms-sw/cmssw#47697

pcanal · 2025-04-08T18:44:55Z

That build seems to have a broken root.exe due to cuda.pcm

[pcanal@cmsdev42 CMSSW_15_1_X_2025-04-07-1100]$ root.exe -b -q
root.exe: /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.11-864308583194219b57d821e91f95ee35/root-6.32.11/interpreter/llvm-project/clang/lib/Serialization/ASTReader.cpp:6482: void clang::ASTReader::ReadPragmaDiagnosticMappings(clang::DiagnosticsEngine&): Assertion `IDAndOffset.first.isValid() && "invalid FileID for transition"' failed.
Aborted (core dumped)

Details

(gdb) 
#6  0x00007fffe9cdf8cd in clang::ASTReader::ReadAST (this=0xd5a920, FileName=..., Type=clang::serialization::MK_PrebuiltModule, ImportLoc=..., ClientLoadCapabilities=0, Imported=0x0)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.11-864308583194219b57d821e91f95ee35/root-6.32.11/interpreter/llvm-project/clang/lib/Serialization/ASTReader.cpp:4505
4505    in /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.11-864308583194219b57d821e91f95ee35/root-6.32.11/interpreter/llvm-project/clang/lib/Serialization/ASTReader.cpp
(gdb) p FileName
$2 = {static npos = 18446744073709551615, Data = 0x1dc41a0 "/cvmfs/cms-ci.cern.ch/week0/PR_eb19f643/el8_amd64_gcc12/lcg/root/6.32.11-864308583194219b57d821e91f95ee35/lib/cuda.pcm", Length = 118}
(gdb)

makortel · 2025-04-08T19:04:04Z

Ok, so now we have a connection to #9765

makortel · 2025-05-06T15:46:07Z

@cmsbuild, please test

Let's try again

cmsbuild · 2025-05-06T19:56:26Z

-1

Failed Tests: Build
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-387b8e/45883/summary.html
COMMIT: 670c837
CMSSW: CMSSW_15_1_X_2025-05-06-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9761/45883/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-387b8e/45883/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-387b8e/45883/git-merge-result

Build

I found compilation error when building:

/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/bin/c++ -O3 -pthread -pipe -Werror=main -Werror=pointer-arith -Werror=overlength-strings -Wno-vla -Werror=overflow -std=c++20 -ftree-vectorize -Werror=array-bounds -Werror=format-contains-nul -Werror=type-limits -fvisibility-inlines-hidden -fno-math-errno --param vect-max-version-for-alias-checks=50 -Xassembler --compress-debug-sections -Wno-error=array-bounds -Warray-bounds -fuse-ld=bfd -march=x86-64-v3 -felide-constructors -fmessage-length=0 -Wall -Wno-non-template-friend -Wno-long-long -Wreturn-type -Wextra -Wpessimizing-move -Wclass-memaccess -Wno-cast-function-type -Wno-unused-but-set-parameter -Wno-ignored-qualifiers -Wno-unused-parameter -Wunused -Wparentheses -Werror=return-type -Werror=missing-braces -Werror=unused-value -Werror=unused-label -Werror=address -Werror=format -Werror=sign-compare -Werror=write-strings -Werror=delete-non-virtual-dtor -Werror=strict-aliasing -Werror=narrowing -Werror=unused-but-set-variable -Werror=reorder -Werror=unused-variable -Werror=conversion-null -Werror=return-local-addr -Wnon-virtual-dtor -Werror=switch -fdiagnostics-show-option -Wno-unused-local-typedefs -Wno-attributes -Wno-psabi -Wno-error=unused-variable -DALPAKA_DEFAULT_HOST_MEMORY_ALIGNMENT=128 -DALPAKA_DISABLE_VENDOR_RNG -DBOOST_DISABLE_ASSERTS -flto=auto -fipa-icf -flto-odr-type-merging -fno-fat-lto-objects -Wodr -shared -Wl,-E    -Wl,-z,defs     tmp/el8_amd64_gcc12/src/CalibCalorimetry/EcalCorrelatedNoiseAnalysisModules/src/CalibCalorimetryEcalCorrelatedNoiseAnalysisModules/EcnaAnalyzer.cc.o tmp/el8_amd64_gcc12/src/CalibCalorimetry/EcalCorrelatedNoiseAnalysisModules/src/CalibCalorimetryEcalCorrelatedNoiseAnalysisModules/SealModule.cc.o -o tmp/el8_amd64_gcc12/src/CalibCalorimetry/EcalCorrelatedNoiseAnalysisModules/src/CalibCalorimetryEcalCorrelatedNoiseAnalysisModules/libCalibCalorimetryEcalCorrelatedNoiseAnalysisModules.so -Wl,-E -Wl,--hash-style=gnu -Wl,--as-needed -Wl,-z,noexecstack -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_1_X_2025-05-06-1100/biglib/el8_amd64_gcc12 -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_1_X_2025-05-06-1100/lib/el8_amd64_gcc12 -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_1_X_2025-05-06-1100/external/el8_amd64_gcc12/lib -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_1_X_2025-05-06-1100/static/el8_amd64_gcc12 -lDataFormatsEcalDigi -lDataFormatsEcalDetId -lFWCoreFramework -lDataFormatsDetId -lFWCoreCommon -lFWCoreServiceRegistry -lDataFormatsCommon -lFWCoreParameterSet -lFWCoreAbstractServices -lFWCoreMessageLogger -lDataFormatsProvenance -lFWCoreConcurrency -lFWCorePluginManager -lFWCoreReflection -lHeterogeneousCoreAlpakaInterface -lCalibCalorimetryEcalCorrelatedNoiseAnalysisAlgos -lFWCoreUtilities -lFWCoreVersion -lGui -lGpad -lGraf -lPhysics -lHist -lMatrix -lTree -lNet -lThread -lMathCore -lRint -lRIO -lboost_program_options -lCore -lboost_thread -lboost_date_time -lpng -lpcre -lbz2 -ljpeg -lturbojpeg -luuid -ltbb -llzma -lz -lfmt -lcms-md5 -lcrypt -ldl -lrt -lstdc++fs -ltinyxml2
Leaving library rule at CalibCalorimetry/EcalCorrelatedNoiseAnalysisModules
@@@@ Running EDM write config for CalibCalorimetryEcalCorrelatedNoiseAnalysisModules
edmWriteConfigs: /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.13-77e6ec8b4f0eebe72829752f82c6909e/root-6.32.13/interpreter/llvm-project/clang/lib/Serialization/ASTReader.cpp:6482: void clang::ASTReader::ReadPragmaDiagnosticMappings(clang::DiagnosticsEngine&): Assertion `IDAndOffset.first.isValid() && "invalid FileID for transition"' failed.
/bin/sh: line 1: 171739 Aborted                 (core dumped) edmWriteConfigs -p /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_1_X_2025-05-06-1100/tmp/el8_amd64_gcc12/src/CalibCalorimetry/EcalCorrelatedNoiseAnalysisModules/src/CalibCalorimetryEcalCorrelatedNoiseAnalysisModules/libCalibCalorimetryEcalCorrelatedNoiseAnalysisModules.so
gmake: *** [lib/el8_amd64_gcc12/CalibCalorimetryEcalCorrelatedNoiseAnalysisModules.edmplugin] Error 1
>> Leaving Package CalibCalorimetry/EcalCorrelatedNoiseAnalysisModules
>> Package CalibCalorimetry/EcalCorrelatedNoiseAnalysisModules built
>> Entering Package CalibCalorimetry/EcalPedestalOffsets
>> Leaving Package CalibCalorimetry/EcalPedestalOffsets
>> Package CalibCalorimetry/EcalPedestalOffsets built

makortel · 2025-05-06T20:05:19Z

Ok, so the ROOT debug build seems to continue to fail

makortel · 2025-05-20T16:06:45Z

@cmsbuild, please test

Let's see if #9867 would have an impact

cmsbuild · 2025-05-20T20:38:30Z

-1

Failed Tests: Build
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-387b8e/46265/summary.html
COMMIT: 670c837
CMSSW: CMSSW_15_1_X_2025-05-20-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9761/46265/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@asavincms [L1T] Calo trigger Zero Suppression at Layer1 included in Online producer cmssw#48121

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-387b8e/46265/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-387b8e/46265/git-merge-result

Build

I found compilation error when building:

/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/bin/c++ -O3 -pthread -pipe -Werror=main -Werror=pointer-arith -Werror=overlength-strings -Wno-vla -Werror=overflow -std=c++20 -ftree-vectorize -Werror=array-bounds -Werror=format-contains-nul -Werror=type-limits -fvisibility-inlines-hidden -fno-math-errno --param vect-max-version-for-alias-checks=50 -Xassembler --compress-debug-sections -Wno-error=array-bounds -Warray-bounds -fuse-ld=bfd -march=x86-64-v3 -felide-constructors -fmessage-length=0 -Wall -Wno-non-template-friend -Wno-long-long -Wreturn-type -Wextra -Wpessimizing-move -Wclass-memaccess -Wno-cast-function-type -Wno-unused-but-set-parameter -Wno-ignored-qualifiers -Wno-unused-parameter -Wunused -Wparentheses -Werror=return-type -Werror=missing-braces -Werror=unused-value -Werror=unused-label -Werror=address -Werror=format -Werror=sign-compare -Werror=write-strings -Werror=delete-non-virtual-dtor -Werror=strict-aliasing -Werror=narrowing -Werror=unused-but-set-variable -Werror=reorder -Werror=unused-variable -Werror=conversion-null -Werror=return-local-addr -Wnon-virtual-dtor -Werror=switch -fdiagnostics-show-option -Wno-unused-local-typedefs -Wno-attributes -Wno-psabi -Wno-error=unused-variable -DALPAKA_DEFAULT_HOST_MEMORY_ALIGNMENT=128 -DALPAKA_DISABLE_VENDOR_RNG -DBOOST_DISABLE_ASSERTS -flto=auto -fipa-icf -flto-odr-type-merging -fno-fat-lto-objects -Wodr -shared -Wl,-E    -Wl,-z,defs     tmp/el8_amd64_gcc12/src/CalibCalorimetry/EcalCorrelatedNoiseAnalysisModules/src/CalibCalorimetryEcalCorrelatedNoiseAnalysisModules/EcnaAnalyzer.cc.o tmp/el8_amd64_gcc12/src/CalibCalorimetry/EcalCorrelatedNoiseAnalysisModules/src/CalibCalorimetryEcalCorrelatedNoiseAnalysisModules/SealModule.cc.o -o tmp/el8_amd64_gcc12/src/CalibCalorimetry/EcalCorrelatedNoiseAnalysisModules/src/CalibCalorimetryEcalCorrelatedNoiseAnalysisModules/libCalibCalorimetryEcalCorrelatedNoiseAnalysisModules.so -Wl,-E -Wl,--hash-style=gnu -Wl,--as-needed -Wl,-z,noexecstack -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_1_X_2025-05-20-1100/biglib/el8_amd64_gcc12 -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_1_X_2025-05-20-1100/lib/el8_amd64_gcc12 -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_1_X_2025-05-20-1100/external/el8_amd64_gcc12/lib -L/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_1_X_2025-05-20-1100/static/el8_amd64_gcc12 -lDataFormatsEcalDigi -lDataFormatsEcalDetId -lFWCoreFramework -lDataFormatsDetId -lFWCoreCommon -lFWCoreServiceRegistry -lDataFormatsCommon -lFWCoreParameterSet -lFWCoreAbstractServices -lFWCoreMessageLogger -lDataFormatsProvenance -lFWCoreConcurrency -lFWCorePluginManager -lFWCoreReflection -lHeterogeneousCoreAlpakaInterface -lCalibCalorimetryEcalCorrelatedNoiseAnalysisAlgos -lFWCoreUtilities -lFWCoreVersion -lGui -lGpad -lGraf -lPhysics -lHist -lMatrix -lTree -lNet -lThread -lMathCore -lRint -lRIO -lboost_program_options -lCore -lboost_thread -lboost_date_time -lpng -lpcre -lbz2 -ljpeg -lturbojpeg -luuid -ltbb -llzma -lz -lfmt -lcms-md5 -lcrypt -ldl -lrt -lstdc++fs -ltinyxml2
Leaving library rule at CalibCalorimetry/EcalCorrelatedNoiseAnalysisModules
@@@@ Running EDM write config for CalibCalorimetryEcalCorrelatedNoiseAnalysisModules
edmWriteConfigs: /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.13-d19fc720eeca6b9cf6add8184b011d25/root-6.32.13/interpreter/llvm-project/clang/lib/Serialization/ASTReader.cpp:6482: void clang::ASTReader::ReadPragmaDiagnosticMappings(clang::DiagnosticsEngine&): Assertion `IDAndOffset.first.isValid() && "invalid FileID for transition"' failed.
/bin/sh: line 1: 3205471 Aborted                 (core dumped) edmWriteConfigs -p /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_15_1_X_2025-05-20-1100/tmp/el8_amd64_gcc12/src/CalibCalorimetry/EcalCorrelatedNoiseAnalysisModules/src/CalibCalorimetryEcalCorrelatedNoiseAnalysisModules/libCalibCalorimetryEcalCorrelatedNoiseAnalysisModules.so
gmake: *** [lib/el8_amd64_gcc12/CalibCalorimetryEcalCorrelatedNoiseAnalysisModules.edmplugin] Error 1
>> Leaving Package CalibCalorimetry/EcalCorrelatedNoiseAnalysisModules
>> Package CalibCalorimetry/EcalCorrelatedNoiseAnalysisModules built
>> Entering Package CalibCalorimetry/EcalCorrectionModules
>> Leaving Package CalibCalorimetry/EcalCorrectionModules
>> Package CalibCalorimetry/EcalCorrectionModules built

makortel · 2025-05-20T22:01:28Z

So the problem here seems to be independent of #9867

smuzaffar · 2025-05-21T14:51:01Z

@vgvassilev , can you please check why root built with -DCMAKE_BUILD_TYPE=Debug -DLLVM_BUILD_TYPE=Debug fails with error root-6.32.13/interpreter/llvm-project/clang/lib/Serialization/ASTReader.cpp:6482: void clang::ASTReader::ReadPragmaDiagnosticMappings(clang::DiagnosticsEngine&): Assertion IDAndOffset.first.isValid() && "invalid FileID for transition"' failed . you can reproduce the issue with following commands. Note that this failure is with ROOT 6.32 . For root master branch debug root works fine

#login to lxplus
ssh lxplus
#start cmssw el8 pptainer
/cvmfs/cms.cern.ch/common/cmssw-el8
# setup cmssw env which has root built in debug mode
cd /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9761/46265/CMSSW_15_1_X_2025-05-20-1100
eval `scram run -sh`
#run test
python3 -c 'import ROOT'

vgvassilev · 2025-05-25T09:37:17Z

@vgvassilev , can you please check why root built with -DCMAKE_BUILD_TYPE=Debug -DLLVM_BUILD_TYPE=Debug fails with error root-6.32.13/interpreter/llvm-project/clang/lib/Serialization/ASTReader.cpp:6482: void clang::ASTReader::ReadPragmaDiagnosticMappings(clang::DiagnosticsEngine&): Assertion IDAndOffset.first.isValid() && "invalid FileID for transition"' failed . you can reproduce the issue with following commands. Note that this failure is with ROOT 6.32 . For root master branch debug root works fine
#login to lxplus
ssh lxplus
#start cmssw el8 pptainer
/cvmfs/cms.cern.ch/common/cmssw-el8
# setup cmssw env which has root built in debug mode
cd /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9761/46265/CMSSW_15_1_X_2025-05-20-1100
eval `scram run -sh`
#run test
python3 -c 'import ROOT'

This seems to stumble upon trying to load "/cvmfs/cms-ci.cern.ch/week0/PR_eb19f643/el8_amd64_gcc12/lcg/root/6.32.13-d19fc720eeca6b9cf6add8184b011d25/lib/cuda.pcm"

#0  0x00007ffff6aa552f in raise () from /lib64/libc.so.6
#1  0x00007ffff6a78e65 in abort () from /lib64/libc.so.6
#2  0x00007ffff6a78d39 in __assert_fail_base.cold.0 () from /lib64/libc.so.6
#3  0x00007ffff6a9de86 in __assert_fail () from /lib64/libc.so.6
#4  0x00007fffe982362b in clang::ASTReader::ReadPragmaDiagnosticMappings (this=0xea0820, Diag=...)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.13-d19fc720eeca6b9cf6add8184b011d25/root-6.32.13/interpreter/llvm-project/clang/lib/Serialization/ASTReader.cpp:6482
#5  0x00007fffe9816f07 in clang::ASTReader::InitializeContext (this=0xea0820)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.13-d19fc720eeca6b9cf6add8184b011d25/root-6.32.13/interpreter/llvm-project/clang/lib/Serialization/ASTReader.cpp:5090
#6  0x00007fffe98148cd in clang::ASTReader::ReadAST (this=0xea0820, FileName=..., Type=clang::serialization::MK_PrebuiltModule, 
    ImportLoc=..., ClientLoadCapabilities=0, Imported=0x0)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.13-d19fc720eeca6b9cf6add8184b011d25/root-6.32.13/interpreter/llvm-project/clang/lib/Serialization/ASTReader.cpp:4505
#7  0x00007fffe93a3665 in clang::CompilerInstance::findOrCompileModuleAndReadAST (this=0x4076c0, ModuleName=..., ImportLoc=..., 
    ModuleNameLoc=..., IsInclusionDirective=false)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.13-d19fc720eeca6b9cf6add8184b011d25/root-6.32.13/interpreter/llvm-project/clang/lib/Frontend/CompilerInstance.cpp:1846
#8  0x00007fffe93a41f0 in clang::CompilerInstance::loadModule (this=0x4076c0, ImportLoc=..., Path=..., 
    Visibility=clang::Module::AllVisible, IsInclusionDirective=false)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.13-d19fc720eeca6b9cf6add8184b011d25/root-6.32.13/interpreter/llvm-project/clang/lib/Frontend/CompilerInstance.cpp:1989
#9  0x00007fffea3d81e3 in clang::Sema::ActOnModuleImport (this=0x6914a0, StartLoc=..., ExportLoc=..., ImportLoc=..., Path=..., 
    IsPartition=false)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.13-d19fc720eeca6b9cf6add8184b011d25/root-6.32.13/interpreter/llvm-project/clang/lib/Sema/SemaModule.cpp:525
#10 0x00007fffe848ee8c in cling::Interpreter::loadModule (this=0x49ead0, M=0x6dcf70, complain=true)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.13-d19fc720eeca6b9cf6add8184b011d25/root-6.32.13/interpreter/cling/lib/Interpreter/Interpreter.cpp:926
#11 0x00007fffe848ec35 in cling::Interpreter::loadModule (this=0x49ead0, moduleName="cuda", complain=true)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.13-d19fc720eeca6b9cf6add8184b011d25/root-6.32.13/interpreter/cling/lib/Interpreter/Interpreter.cpp:890
#12 0x00007fffe81e1d26 in LoadModule (ModuleName="cuda", interp=...)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.13-d19fc720eeca6b9cf6add8184b011d25/root-6.32.13/core/metacling/src/TCling.cxx:1055
#13 0x00007fffe81e3045 in RegisterCxxModules (clingInterp=...)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.13-d19fc720eeca6b9cf6add8184b011d25/root-6.32.13/core/metacling/src/TCling.cxx:1275
#14 0x00007fffe81e546d in TCling::TCling (this=0x4fa4f0, name=0x7fffeee9a603 "C++", title=0x7fffeee9a5ed "cling C++ Interpreter", 
    argv=0x7ffffffedb20, interpLibHandle=0x59fc60)
--Type <RET> for more, q to quit, c to continue without paging--
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.13-d19fc720eeca6b9cf6add8184b011d25/root-6.32.13/core/metacling/src/TCling.cxx:1567
#15 0x00007fffe81df66e in CreateInterpreter (interpLibHandle=0x59fc60, argv=0x7ffffffedb20)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.13-d19fc720eeca6b9cf6add8184b011d25/root-6.32.13/core/metacling/src/TCling.cxx:611
#16 0x00007ffff5fdf07c in TROOT::InitInterpreter (this=0x7ffff628bdc0 <ROOT::Internal::GetROOT1()::alloc>)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.13-d19fc720eeca6b9cf6add8184b011d25/root-6.32.13/core/base/src/TROOT.cxx:2126
#17 0x00007ffff5fd87d3 in ROOT::Internal::GetROOT2 ()
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.13-d19fc720eeca6b9cf6add8184b011d25/root-6.32.13/core/base/src/TROOT.cxx:391
#18 0x00007ffff5fd8b35 in ROOT::GetROOT ()
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.13-d19fc720eeca6b9cf6add8184b011d25/root-6.32.13/core/base/src/TROOT.cxx:473
#19 0x00007ffff63c9b70 in (anonymous namespace)::ApplicationStarter::ApplicationStarter (
    this=0x7ffff63ffa50 <(anonymous namespace)::_applicationStarter>)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.13-d19fc720eeca6b9cf6add8184b011d25/root-6.32.13/bindings/pyroot/cppyy/cppyy-backend/clingwrapper/src/clingwrapper.cxx:218
#20 0x00007ffff63dbe1b in __static_initialization_and_destruction_0 (__initialize_p=1, __priority=65535)
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.13-d19fc720eeca6b9cf6add8184b011d25/root-6.32.13/bindings/pyroot/cppyy/cppyy-backend/clingwrapper/src/clingwrapper.cxx:323
#21 0x00007ffff63dc2fb in _GLOBAL__sub_I_clingwrapper.cxx(void) ()
    at /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/lcg/root/6.32.13-d19fc720eeca6b9cf6add8184b011d25/root-6.32.13/bindings/pyroot/cppyy/cppyy-backend/clingwrapper/src/clingwrapper.cxx:2848
#22 0x00007ffff7dd611a in call_init (l=<optimized out>, argc=argc@entry=3, argv=argv@entry=0x7fffffff2348, env=env@entry=0x545390)
    at dl-init.c:72
#23 0x00007ffff7dd621a in call_init (env=0x545390, argv=0x7fffffff2348, argc=3, l=<optimized out>) at dl-init.c:118
#24 _dl_init (main_map=0x60f6a0, argc=3, argv=0x7fffffff2348, env=0x545390) at dl-init.c:119
#25 0x00007ffff6bd012c in _dl_catch_exception () from /lib64/libc.so.6
#26 0x00007ffff7ddd92e in dl_open_worker (a=0x7ffffffeec40) at dl-open.c:788

smuzaffar · 2025-05-26T15:25:09Z

@vgvassilev , is it a known issue with root 6.32.00 debug build? Why we only see this for ROOT debug build and not for non-debug builds?

vgvassilev · 2025-05-26T15:27:57Z

@vgvassilev , it is a known issue with root 6.32.00 debug build? Why we only see this for ROOT debug build and not for non-debug builds?

I think we don't see it in newer ROOT versions because llvm is newer.

smuzaffar · 2025-05-26T15:43:14Z

@vgvassilev , it is a known issue with root 6.32.00 debug build? Why we only see this for ROOT debug build and not for non-debug builds?

I think we don't see it in newer ROOT versions because llvm is newer.

This is correct @vgvassilev but we see inconsistent behavior within root 6.32 itself. python3 -c 'import ROOT' fails when ROOT 6.32 and root/llvm is build in debug mode but does not fail for ROOT 6.32 in non-debug mode? e.g. the following works (where we have root 6.32 build in non-debug mode and root also contains cuda.pcm and import ROOT works)

> ssh lxplus
> cd /tmp/$(whoami)
> /cvmfs/cms.cern.ch/common/cmssw-el8
> scram p  CMSSW_15_1_X_2025-05-26-1100
> cd CMSSW_15_1_X_2025-05-26-1100
> eval `scram run -sh`
> strace -o out.log -f python3 -c 'import ROOT'
> echo $?
0
> grep cuda.pcm out.log
919297 stat("/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02891/el8_amd64_gcc12/lcg/root/6.32.13-b42babf347314b02960f7ec363565e45/lib/cuda.pcm", {st_mode=S_IFREG|0644, st_size=4500660, ...}) = 0
919297 openat(AT_FDCWD, "/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02891/el8_amd64_gcc12/lcg/root/6.32.13-b42babf347314b02960f7ec363565e45/lib/cuda.pcm", O_RDONLY|O_CLOEXEC) = 3

vgvassilev · 2025-05-26T17:45:37Z

@vgvassilev , it is a known issue with root 6.32.00 debug build? Why we only see this for ROOT debug build and not for non-debug builds?

I think we don't see it in newer ROOT versions because llvm is newer.

This is correct @vgvassilev but we see inconsistent behavior within root 6.32 itself. python3 -c 'import ROOT' fails when ROOT 6.32 and root/llvm is build in debug mode but does not fail for ROOT 6.32 in non-debug mode? e.g. the following works (where we have root 6.32 build in non-debug mode and root also contains cuda.pcm and import ROOT works)
> ssh lxplus
> cd /tmp/$(whoami)
> /cvmfs/cms.cern.ch/common/cmssw-el8
> scram p  CMSSW_15_1_X_2025-05-26-1100
> cd CMSSW_15_1_X_2025-05-26-1100
> eval `scram run -sh`
> strace -o out.log -f python3 -c 'import ROOT'
> echo $?
0
> grep cuda.pcm out.log
919297 stat("/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02891/el8_amd64_gcc12/lcg/root/6.32.13-b42babf347314b02960f7ec363565e45/lib/cuda.pcm", {st_mode=S_IFREG|0644, st_size=4500660, ...}) = 0
919297 openat(AT_FDCWD, "/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02891/el8_amd64_gcc12/lcg/root/6.32.13-b42babf347314b02960f7ec363565e45/lib/cuda.pcm", O_RDONLY|O_CLOEXEC) = 3

That makes sense, this is an assert that triggers only in debug mode. Probably it is harmless to the workflows...

cmsbuild · 2025-05-27T18:15:15Z

Pull request #9761 was updated.

makortel · 2025-05-27T18:16:04Z

@cmsbuild, please test

Let's see if not creating the cuda.pcm for the debug build would succeed

cmsbuild added externals-pending pending-signatures tests-started orp-pending labels Mar 27, 2025

makortel mentioned this pull request Mar 27, 2025

Segmentation violation when running hltGetConfiguration with option -output full cms-sw/cmssw#47697

Open

cmsbuild added tests-rejected and removed tests-started labels Mar 28, 2025

cmsbuild added tests-started and removed tests-rejected labels Apr 7, 2025

cmsbuild added tests-rejected and removed tests-started labels Apr 7, 2025

cmsbuild added tests-started tests-rejected and removed tests-rejected tests-started labels Apr 7, 2025

cmsbuild added tests-started tests-rejected and removed tests-rejected tests-started labels May 6, 2025

This was referenced May 13, 2025

[Do not merge][ROOT6]Build root and llvm in Debug mode #9858

Closed

[ROOT Master, ARM] Multiple RelVals failed - Error parsing payload code for class L2MuonTrajectorySeed cms-sw/cmssw#47774

Open

cmsbuild added tests-started tests-rejected and removed tests-rejected tests-started labels May 20, 2025

[Do not merge][15_1_X] Enable ROOT debug

5f7fee0

makortel force-pushed the rootDebug branch from 670c837 to 5f7fee0 Compare May 27, 2025 18:14

cmsbuild added tests-pending and removed tests-rejected labels May 27, 2025

cmsbuild added tests-started and removed tests-pending labels May 27, 2025

[Do not merge][15_1_X] Enable ROOT debug #9761

Are you sure you want to change the base?

[Do not merge][15_1_X] Enable ROOT debug #9761

Conversation

makortel commented Mar 27, 2025

Uh oh!

makortel commented Mar 27, 2025

Uh oh!

cmsbuild commented Mar 27, 2025

Uh oh!

cmsbuild commented Mar 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented Mar 28, 2025

Unit Tests

Comparison Summary

Uh oh!

makortel commented Apr 7, 2025

Uh oh!

makortel commented Apr 7, 2025

Uh oh!

cmsbuild commented Apr 7, 2025

Build

Uh oh!

makortel commented Apr 7, 2025

Uh oh!

makortel commented Apr 7, 2025

Uh oh!

cmsbuild commented Apr 8, 2025

Build

Uh oh!

smuzaffar commented Apr 8, 2025

Uh oh!

makortel commented Apr 8, 2025

Uh oh!

makortel commented Apr 8, 2025

Uh oh!

pcanal commented Apr 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

makortel commented Apr 8, 2025

Uh oh!

makortel commented May 6, 2025

Uh oh!

cmsbuild commented May 6, 2025

Build

Uh oh!

makortel commented May 6, 2025

Uh oh!

makortel commented May 20, 2025

Uh oh!

cmsbuild commented May 20, 2025

Build

Uh oh!

makortel commented May 20, 2025

Uh oh!

smuzaffar commented May 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vgvassilev commented May 25, 2025

Uh oh!

smuzaffar commented May 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vgvassilev commented May 26, 2025

Uh oh!

smuzaffar commented May 26, 2025

Uh oh!

vgvassilev commented May 26, 2025

Uh oh!

cmsbuild commented May 27, 2025

Uh oh!

makortel commented May 27, 2025

Uh oh!

Uh oh!

cmsbuild commented Mar 27, 2025 •

edited

Loading

pcanal commented Apr 8, 2025 •

edited

Loading

smuzaffar commented May 21, 2025 •

edited

Loading

smuzaffar commented May 26, 2025 •

edited

Loading