DQM bin-by-bin test failing since we moved to ROOT 6.3.6

[smuzaffar]

https://github.com/cms-sw/cmssw/blob/master/DQMServices/FileIO/scripts/compareHistograms.py which we run to compare DQM root files is failing we error like [a] (see #49419) It looks like root change root-project/root#18735 (backport for root 6.36 : root-project/root#18846 ) might be causing this error.

Do we need to update compareHistograms.py for ROOT 6.36 ?
We should also have a simople unit test in IBs to run ompareHistograms.py e.g. a simple command

compareHistograms.py -b a/DQM_V0001_R000000001__Global__CMSSW_X_Y_Z__RECO.root -p b/DQM_V0001_R000000001__Global__CMSSW_X_Y_Z__RECO.root -o result -n N -t M -r CMSSW_VERSION

should be enough to catch such errors in IBs

[a]

PR file successfully opened
Traceback (most recent call last):
  File "/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/49419/49598/CMSSW_16_0_X_2025-11-21-1100/bin/el8_amd64_gcc13/compareHistograms.py", line 345, in <module>
    create_dif(args.base_file, args.pr_file, args.pr_number, args.test_number, cmssw_version, args.num_processes, args.output_dir)
  File "/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/49419/49598/CMSSW_16_0_X_2025-11-21-1100/bin/el8_amd64_gcc13/compareHistograms.py", line 80, in create_dif
    compare(shared_paths, pr_flat_dict, base_flat_dict, paths_to_save_in_pr, paths_to_save_in_base)
  File "/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/49419/49598/CMSSW_16_0_X_2025-11-21-1100/bin/el8_amd64_gcc13/compareHistograms.py", line 182, in compare
    pr_array = np.array(pr_item)
  File "/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02916/el8_amd64_gcc13/lcg/root/6.36.05-302290958d85790e78b48654c2f216ba/lib/ROOT/_pythonization/_uhi.py", line 367, in _getitem
    uhi_index = _compute_common_index(self, index)
  File "/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02916/el8_amd64_gcc13/lcg/root/6.36.05-302290958d85790e78b48654c2f216ba/lib/ROOT/_pythonization/_uhi.py", line 223, in _compute_common_index
    return [_compute_uhi_index(self, idx, axis, include_flow_bins) for axis, idx in enumerate(index)]
  File "/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02916/el8_amd64_gcc13/lcg/root/6.36.05-302290958d85790e78b48654c2f216ba/lib/ROOT/_pythonization/_uhi.py", line 223, in <listcomp>
    return [_compute_uhi_index(self, idx, axis, include_flow_bins) for axis, idx in enumerate(index)]
  File "/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02916/el8_amd64_gcc13/lcg/root/6.36.05-302290958d85790e78b48654c2f216ba/lib/ROOT/_pythonization/_uhi.py", line 192, in _compute_uhi_index
    return _process_index_for_axis(self, index, axis)
  File "/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02916/el8_amd64_gcc13/lcg/root/6.36.05-302290958d85790e78b48654c2f216ba/lib/ROOT/_pythonization/_uhi.py", line 180, in _process_index_for_axis
    raise IndexError(f"Histogram index {index - 1} out of range for axis {axis}. Valid range: (0,{nbins})")
IndexError: Histogram index 30 out of range for axis 0. Valid range: (0,30)

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @smuzaffar.

@Dr15Jones, @ftenchini, @makortel, @mandrenguyen, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[smuzaffar]

assign dqm

[cmsbuild]

New categories assigned: dqm

@ctarricone,@gabrielmscampos,@nothingface0,@rseidita you have been requested to review this Pull request/Issue and eventually sign? Thanks

[mmusich]

urgent

• without DQM bin by bin comparisons, it's sometimes difficult to draw conclusions on PR tests.

[makortel]

Should an error in compareHistograms.py lead to PR test failure? If not, how about at least a warning in the PR test summary message?

[smuzaffar]

I agree that a failure in compareHistograms.py should at least mark DQM bin by bin comparison entry as failed.
We should have a unit test for running compareHistograms.py so that we catch such issue in ROOT6 IBs

[smuzaffar]

@cms-sw/dqm-l2 , there is a suggestion from root team that "if you don't need to access/compare the contents of the underflow/overflow bins, you can use pr_item.values() to get a NumPy array of your bin values and proceed with your comparison (instead of np.array(pr_item), that shouldn't raise any indexing errors". So do we need to access/compare the contents of the underflow/overflow bins ? I have checked and following change worked ( or did not fail) [a].

--- a/DQMServices/FileIO/scripts/compareHistograms.py
+++ b/DQMServices/FileIO/scripts/compareHistograms.py
@@ -143,8 +143,8 @@ def compareMP(shared_paths, pr_flat_dict, base_flat_dict, iProc, return_dict):
 
       elif pr_item.InheritsFrom('TH1') and base_item.InheritsFrom('TH1'):
          # Compare bin by bin
-         pr_array = np.array(pr_item)
-         base_array = np.array(base_item)
+         pr_array = pr_item.values()
+         base_array = base_item.values()
 
          if pr_array.shape != base_array.shape or not np.allclose(pr_array, base_array, equal_nan=True):
             are_different = True
@@ -179,8 +179,8 @@ def compare(shared_paths, pr_flat_dict, base_flat_dict, paths_to_save_in_pr, pat
 
       elif pr_item.InheritsFrom('TH1') and base_item.InheritsFrom('TH1'):
          # Compare bin by bin
-         pr_array = np.array(pr_item)
-         base_array = np.array(base_item)
+         pr_array = pr_item.values()
+         base_array = base_item.values()
 
          if pr_array.shape != base_array.shape or not np.allclose(pr_array, base_array, equal_nan=True):
             are_different = True

[a]

compareHistograms.py -b ~/public/DQM_V0001_R000000001__Global__CMSSW_X_Y_Z__RECO.root -p ~/public/DQM_V0001_R000000001__Global__CMSSW_X_Y_Z__RECO.root -o `pwd`/res -n cmssw_49419 -t 71938 -r CMSSW_FOO
Baseline file successfully opened
PR file successfully opened
No differences were observed - output will not be written
No differences were observed - output will not be written
Base output file. PR output file. Changed elements, removed elements, added elements:
DQM_V0001_R000000001__RelVal_wfpublic_base__CMSSW_FOO-PRcmssw_49419-71938__DQMIO.root
DQM_V0001_R000000001__RelVal_wfpublic_pr__CMSSW_FOO-PRcmssw_49419-71938__DQMIO.root
0 0 0