[ASAN] Stack buffer overflow problem in ZGammaplusJetsPostProcessor

[Dr15Jones]

The ASAN report of CMSSW_16_1_ASAN_X_2026-02-13-2300 has

=================================================================
==3896974==ERROR: AddressSanitizer: dynamic-stack-buffer-overflow on address 0x7fff27036258 at pc 0x14dd7883640f bp 0x7fff270361c0 sp 0x7fff270361b8
WRITE of size 4 at 0x7fff27036258 thread T0
    #0 0x14dd7883640e in ZGammaplusJetsPostProcessor::histos_(dqm::implementation::IBooker&, dqm::implementation::IGetter&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02928/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_1_ASAN_X_2026-02-13-2300/lib/el8_amd64_gcc13/pluginDQMOfflineTriggerPlugins.so+0xe3640e) (BuildId: d9ebd8c70c8de94ba8d41d3e3bd2685b8934b624)
    #1 0x14dd788377a6 in ZGammaplusJetsPostProcessor::dqmEndJob(dqm::implementation::IBooker&, dqm::implementation::IGetter&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02928/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_1_ASAN_X_2026-02-13-2300/lib/el8_amd64_gcc13/pluginDQMOfflineTriggerPlugins.so+0xe377a6) (BuildId: d9ebd8c70c8de94ba8d41d3e3bd2685b8934b624)
    #2 0x14dd77ef970a in non-virtual thunk to DQMEDHarvester::endProcessBlockProduce(edm::ProcessBlock&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02928/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_1_ASAN_X_2026-02-13-2300/lib/el8_amd64_gcc13/pluginDQMOfflineTriggerPlugins.so+0x4f970a) (BuildId: d9ebd8c70c8de94ba8d41d3e3bd2685b8934b624)
    #3 0x14dde61f5463 in edm::one::EDProducerBase::doEndProcessBlock(edm::ProcessBlockPrincipal const&, edm::ModuleCallingContext const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02928/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_1_ASAN_X_2026-02-13-2300/lib/el8_amd64_gcc13/libFWCoreFramework.so+0x9f5463) (BuildId: cd49c44efbc3d088b561b8a03c4a67f45290adbe)
    #4 0x14dde6187b66 in edm::WorkerT<edm::one::EDProducerBase>::implDoEndProcessBlock(edm::ProcessBlockPrincipal const&, edm::ModuleCallingContext const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02928/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_1_ASAN_X_2026-02-13-2300/lib/el8_amd64_gcc13/libFWCoreFramework.so+0x987b66) (BuildId: cd49c44efbc3d088b561b8a03c4a67f45290adbe)
    #5 0x14dde5c4ad90 in decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::ProcessBlockPrincipal, (edm::TransitionActionType)3> >(edm::OccurrenceTraits<edm::ProcessBlockPrincipal, (edm::TransitionActionType)3>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::ProcessBlockPrincipal, (edm::TransitionActionType)3>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::ProcessBlockPrincipal, (edm::TransitionActionType)3> >(edm::OccurrenceTraits<edm::ProcessBlockPrincipal, (edm::TransitionActionType)3>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::ProcessBlockPrincipal, (edm::TransitionActionType)3>::Context const*)::{lambda()#1}) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02928/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_1_ASAN_X_2026-02-13-2300/lib/el8_amd64_gcc13/libFWCoreFramework.so+0x44ad90) (BuildId: cd49c44efbc3d088b561b8a03c4a67f45290adbe)
    #6 0x14dde5c4b5b8 in std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::ProcessBlockPrincipal, (edm::TransitionActionType)3> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::ProcessBlockPrincipal, (edm::TransitionActionType)3>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::ProcessBlockPrincipal, (edm::TransitionActionType)3>::Context const*) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02928/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_1_ASAN_X_2026-02-13-2300/lib/el8_amd64_gcc13/libFWCoreFramework.so+0x44b5b8) (BuildId: cd49c44efbc3d088b561b8a03c4a67f45290adbe)
    #7 0x14dde5c4c270 in edm::SerialTaskQueue::QueuedTask<edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::ProcessBlockPrincipal, (edm::TransitionActionType)3> >::execute()::{lambda()#1}&>(tbb::detail::d2::task_group&, edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::ProcessBlockPrincipal, (edm::TransitionActionType)3> >::execute()::{lambda()#1}&)::{lambda()#1}>::execute() (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02928/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_1_ASAN_X_2026-02-13-2300/lib/el8_amd64_gcc13/libFWCoreFramework.so+0x44c270) (BuildId: cd49c44efbc3d088b561b8a03c4a67f45290adbe)
    #8 0x14dde6d51da3 in tbb::detail::d2::function_task<edm::SerialTaskQueue::spawn(edm::SerialTaskQueue::TaskBase&)::{lambda()#1}>::execute(tbb::detail::d1::execution_data&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02928/el8_amd64_gcc13/cms/cmssw/CMSSW_16_1_ASAN_X_2026-02-11-2300/lib/el8_amd64_gcc13/libFWCoreConcurrency.so+0x10da3) (BuildId: d995c0c53152e49ff571d3f3aa5037234b305008)
    #9 0x14dde562b2f2 in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter>(tbb::detail::d1::task*, tbb::detail::r1::external_waiter&) src/tbb/task_dispatcher.h:344
    #10 0x14dde562b2f2 in tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<tbb::detail::r1::external_waiter>(tbb::detail::d1::task*, tbb::detail::r1::external_waiter&) src/tbb/task_dispatcher.h:487
    #11 0x14dde562b2f2 in tbb::detail::r1::task_dispatcher::execute_and_wait(tbb::detail::d1::task*, tbb::detail::d1::wait_context&, tbb::detail::d1::task_group_context&) src/tbb/task_dispatcher.cpp:169
    #12 0x14dde5bab5e0 in edm::FinalWaitingTask::wait() (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02928/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_1_ASAN_X_2026-02-13-2300/lib/el8_amd64_gcc13/libFWCoreFramework.so+0x3ab5e0) (BuildId: cd49c44efbc3d088b561b8a03c4a67f45290adbe)
    #13 0x14dde5b480e6 in edm::EventProcessor::endProcessBlock(bool, bool) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02928/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_1_ASAN_X_2026-02-13-2300/lib/el8_amd64_gcc13/libFWCoreFramework.so+0x3480e6) (BuildId: cd49c44efbc3d088b561b8a03c4a67f45290adbe)
    #14 0x14dde5b8a231 in edm::EventProcessor::runToCompletion() (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02928/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_1_ASAN_X_2026-02-13-2300/lib/el8_amd64_gcc13/libFWCoreFramework.so+0x38a231) (BuildId: cd49c44efbc3d088b561b8a03c4a67f45290adbe)
    #15 0x40c675 in tbb::detail::d1::task_arena_function<main::{lambda()#1}::operator()() const::{lambda()#1}, void>::operator()() const (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02928/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_1_ASAN_X_2026-02-13-2300/bin/el8_amd64_gcc13/cmsRun+0x40c675) (BuildId: 694c6e09d3cc1c4294aa07d047c5db71a3b7eb37)
    #16 0x14dde561a8c1 in tbb::detail::r1::task_arena_impl::execute(tbb::detail::d1::task_arena_base&, tbb::detail::d1::delegate_base&) src/tbb/arena.cpp:860
    #17 0x4156de in main::{lambda()#1}::operator()() const (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02928/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_1_ASAN_X_2026-02-13-2300/bin/el8_amd64_gcc13/cmsRun+0x4156de) (BuildId: 694c6e09d3cc1c4294aa07d047c5db71a3b7eb37)
    #18 0x408b7c in main (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02928/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_1_ASAN_X_2026-02-13-2300/bin/el8_amd64_gcc13/cmsRun+0x408b7c) (BuildId: 694c6e09d3cc1c4294aa07d047c5db71a3b7eb37)
    #19 0x14dde4264864 in __libc_start_main (/lib64/libc.so.6+0x3a864) (BuildId: 1faac7cdefc71ce73027e33a84650684eecd1635)
    #20 0x408edd in _start (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02928/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_1_ASAN_X_2026-02-13-2300/bin/el8_amd64_gcc13/cmsRun+0x408edd) (BuildId: 694c6e09d3cc1c4294aa07d047c5db71a3b7eb37)

Address 0x7fff27036258 is located in stack of thread T0
SUMMARY: AddressSanitizer: dynamic-stack-buffer-overflow (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02928/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_1_ASAN_X_2026-02-13-2300/lib/el8_amd64_gcc13/pluginDQMOfflineTriggerPlugins.so+0xe3640e) (BuildId: d9ebd8c70c8de94ba8d41d3e3bd2685b8934b624) in ZGammaplusJetsPostProcessor::histos_(dqm::implementation::IBooker&, dqm::implementation::IGetter&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
Shadow bytes around the buggy address:
  0x7fff27035f80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x7fff27036000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x7fff27036080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x7fff27036100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x7fff27036180: 00 00 00 00 00 00 00 00 00 00 00 00 ca ca ca ca
=>0x7fff27036200: 00 00 00 00 00 00 00 00 00 00 00[cb]cb cb cb cb
  0x7fff27036280: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x7fff27036300: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x7fff27036380: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x7fff27036400: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x7fff27036480: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @Dr15Jones.

@Dr15Jones, @ftenchini, @makortel, @mandrenguyen, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[Dr15Jones]

This is most probably from the array on the stack defined here

cmssw/DQMOffline/Trigger/plugins/ZGammaplusJetsPostProcessor.cc

Lines 83 to 87 in 20a7dd3

	float bins[DB_nbins + 1];
	for (int i = 0; i <= (h_DirectBalance->GetXaxis()->GetNbins()) + 1; i++) {
	double xlow = h_DirectBalance->GetXaxis()->GetBinLowEdge(i);
	bins[i] = xlow;
	}

Where the for loop goes off the end of the storage size. The buffer should be declared as

float bins[DB_nbins + 2]; 

[Dr15Jones]

assign DQMOffline/Trigger

[cmsbuild]

New categories assigned: dqm

@ctarricone,@gabrielmscampos,@nothingface0,@rseidita you have been requested to review this Pull request/Issue and eventually sign? Thanks

[dan131riley]

This is seen in WF 2025.0020001. It's also seen in UBSAN with the stack trace consistent with what @Dr15Jones found:

src/DQMOffline/Trigger/plugins/ZGammaplusJetsPostProcessor.cc:86:11: runtime error: index 22 out of bounds for type 'float [*]'
    #0 0x1498f851457b in ZGammaplusJetsPostProcessor::histos_(dqm::implementation::IBooker&, dqm::implementation::IGetter&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) src/DQMOffline/Trigger/plugins/ZGammaplusJetsPostProcessor.cc:86
    #1 0x1498f85169b8 in ZGammaplusJetsPostProcessor::dqmEndJob(dqm::implementation::IBooker&, dqm::implementation::IGetter&) src/DQMOffline/Trigger/plugins/ZGammaplusJetsPostProcessor.cc:53
    #2 0x1498f76ebd0a in non-virtual thunk to DQMEDHarvester::endProcessBlockProduce(edm::ProcessBlock&) (/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02928/el8_amd64_gcc13/cms/cmssw-patch/CMSSW_16_1_UBSAN_X_2026-02-13-2300/lib/el8_amd64_gcc13/pluginDQMOfflineTriggerPlugins.so+0x28ebd0a) (BuildId: 971be546785d6437396e70a48565d215a701d1a1)

[esiam]

Hi @dan131riley! This PR was opened by me. I never got this error and tests were passed. How should I handle this?

[esiam]

Hi @dan131riley , @Dr15Jones.

Tried some local tests in a small file and printing the edges of the bins I got this:

0 -58.8571
1 12
2 15
3 20
4 25
5 30
6 35
7 40
8 45
9 50
10 60
11 70
12 85
13 105
14 130
15 175
16 230
17 300
18 400
19 500
20 700
21 1000
22 1500

while I would like to get the bins like they are defined here:
std::vector bins_ = {12, 15, 20, 25, 30, 35, 40, 45, 50, 60, 70,
85, 105, 130, 175, 230, 300, 400, 500, 700, 1000, 1500}; // Z or photon pT Binning
link

so changing it like:

  int DB_nbins = h_DirectBalance->GetXaxis()->GetNbins();

  float bins[DB_nbins + 1];

  for (int i = 0; i <= (h_DirectBalance->GetXaxis()->GetNbins()) ; i++) {
    double xlow = h_DirectBalance->GetXaxis()->GetBinLowEdge(i+1);
    bins[i] = xlow;
  }

MonitorElement *meanDBvsRefPt = ibooker.book1D(outName, title, DB_nbins, bins);

I get:

0 12
1 15
2 20
3 25
4 30
5 35
6 40
7 45
8 50
9 60
10 70
11 85
12 105
13 130
14 175
15 230
16 300
17 400
18 500
19 700
20 1000
21 1500

Which is what I was going for.
Not sure if this: h_DirectBalance->GetXaxis()->GetBinLowEdge(i+1); is error proof. For me does not give an error. But ..

Also works this way:

float bins[DB_nbins + 1];

 for (int i = 1; i <= (h_DirectBalance->GetXaxis()->GetNbins()); i++) {
   double xlow = h_DirectBalance->GetXaxis()->GetBinLowEdge(i);

   bins[i-1] = xlow;
 }
 bins[21] = 1500;
 MonitorElement *meanDBvsRefPt = ibooker.book1D(outName, title, DB_nbins, bins);

But I also then should fix here:

for (int i = 1; i <= hp->GetNbinsX() + 1; i++) {
    pr->SetBinContent(i + 1, hp->GetBinContent(i));
    if (hp->GetBinError(i) != 0.)
      pr->SetBinError(i + 1, hp->GetBinError(i));
  }

It should be:

for (int i = 1; i <= hp->GetNbinsX(); i++) {
    pr->SetBinContent(i , hp->GetBinContent(i));
    if (hp->GetBinError(i) != 0.)
      pr->SetBinError(i, hp->GetBinError(i));
  }

To be correct.

If you also agree that these are the issues, or if it can be tested in the workflow that you mention above, let me know if I should do any actions to fix this.
Apologies for missing this.

[Dr15Jones]

That does appear to be the issue to me.
Just running the job in a regular release will not give the messages. To see the messages you need to build in either the ASAN or UBSAN release as they compile the code in a special way which instruments the code to look for problems.

[esiam]

Hi! @Dr15Jones thank you for your comment, I did not know about those releases! I build CMSSW_16_1_ASAN_ and checked again and the error is removed after the changes.
I have not faced this again, so now, I guess, I should open a new PR fixing this issue?
Also I want to ask because there is an open backport on CMSSW_16_0, can I force commit this fix there? (or should it be a new commit in order to be consistent regarding the history in the 2 versions?)