Change max-memory-used check to use real difference in MB not percentage diff. Increase the threshold to 100MB. Flag PR is check fails.

[gartung]

No description provided.

[cmsbuild]

A new Pull Request was created by @gartung for branch master.

@akritkbehera, @cmsbuild, @iarspider, @smuzaffar can you please review it and eventually sign? Thanks.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

[cmsbuild]

cms-bot internal usage

[gartung]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-20c417/49585/summary.html
COMMIT: 1ecd417
CMSSW: CMSSW_16_0_X_2025-11-20-1100/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cms-bot/2622/49585/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially removed 4 lines from the logs
• Reco comparison results: 6 differences found in the comparisons
• Reco comparison had 2 failed jobs
• DQMHistoTests: Total files compared: 51
• DQMHistoTests: Total histograms compared: 3906528
• DQMHistoTests: Total failures: 71
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3906437
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 50 files compared)
• Checked 218 log files, 188 edm output root files, 51 DQM output files
• TriggerResults: no differences found

[smuzaffar]

@gartung , currently there is no indication on the summary page that shows if there are any memory comparisons which have memory threshold warning or error. Also system treats memory increase and decrease in the same way ( red color either there is +10MB or -10MB). Was this done on purpose?

[gartung]

Yes. It's the absolute value of the difference. Maybe it should only be for positive values.

[cmsbuild]

Pull request #2622 was updated.

[cmsbuild]

Pull request #2622 was updated.

[cmsbuild]

Pull request #2622 was updated.

[cmsbuild]

Pull request #2622 was updated.

[cmsbuild]

Pull request #2622 was updated.

[gartung]

@smuzaffar Do I need to modify report-pull-request-results.py to flag the PR as failed if the maxmem comparison fails?

[cmsbuild]

Pull request #2622 was updated.

[cmsbuild]

Pull request #2622 was updated.

[makortel]

Also system treats memory increase and decrease in the same way ( red color either there is +10MB or -10MB). Was this done on purpose?

Maybe it should only be for positive values.

I think flagging negative values (i.e. PR decreases the memory usage) would be useful as well. We could start with the same thresholds (10 MB and 80 MB), but maybe use different colors? (two shades of green? blue and green? we will probably end up with a very color blind unfriendly palette though)

[makortel]

If any job's memory usage goes over the "error" threshold of 80 MB, I'd like to have a message in the PR test summary message (i.e. cms-sw/cmsdist#10199 (comment)) telling that at high level, e.g. number of workflow steps (jobs) whose memory usage increased (or whose memory usage decreased).

@smuzaffar Would you be ok for now to tag @cms-sw/core-l2 in that message? For the beginning I'd like to inspect the behavior. Eventually we could drop the tagging.