Skip to content

Update to relval_gpu.pyfor Alpaka and Pixel Triplets Alpaka Workflows #45694

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Sep 6, 2024
Merged

Update to relval_gpu.pyfor Alpaka and Pixel Triplets Alpaka Workflows #45694

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
8bb7330
Move relval_gpu to Alpaka + add D110 wfs
Aug 13, 2024
f6388e4
Keeping the workflows needed by the bot
Aug 14, 2024
aa1ebac
Add triplets validation, profiling and fullreco
Aug 14, 2024
992e347
Fix for Phase2 GPUvsCPU DQM (no raw DQM needed)
Aug 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions Configuration/PyReleaseValidation/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,11 +33,15 @@ The offsets currently in use are:
* 0.402: Alpaka, pixel only quadruplets, portable
* 0.403: Alpaka, pixel only quadruplets, portable vs. CPU validation
* 0.404: Alpaka, pixel only quadruplets, portable profiling
* 0.406: Alpaka, pixel only triplets, portable
* 0.407: Alpaka, pixel only triplets, portable vs. CPU validation
* 0.407: Alpaka, pixel only triplets, portable profiling
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo.

* 0.412: Alpaka, ECAL only, portable
* 0.422: Alpaka, HCAL only, portable
* 0.423: Alpaka, HCAL only, portable vs CPU validation
* 0.424: Alpaka, HCAL only, portable profiling
* 0.492: Alpaka, full reco with pixel quadruplets
* 0.496: Alpaka, full reco with pixel triplets
* 0.5: Pixel tracking only + 0.1
* 0.501: Patatrack, pixel only quadruplets, on CPU
* 0.502: Patatrack, pixel only quadruplets, with automatic offload to GPU if available
Expand Down
106 changes: 52 additions & 54 deletions Configuration/PyReleaseValidation/python/relval_gpu.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,70 +13,68 @@

# mc WFs to run in IB:

# mc 2023 Alpaka pixel-only quadruplets: TTbar: any backend, any backend vs cpu validation, profiling
# mc 2023 #FIXME to be removed as soon as cms-bot is updated
# no PU Alpaka pixel-only TTbar: quadruplets any backend and profiling; ECAL-only any backend; HCAL-only any backend and profiling
# mc 2024
# no PU
# Alpaka pixel-only quadruplets: TTbar: any backend, any backend vs cpu validation, profiling
# Alpaka pixel-only triplets: TTbar: any backend, any backend vs cpu validation, profiling
# Alpaka ECAL-only: TTbar: any backend
# Alpaka HCAL-only: TTbar: any backend, any backend vs cpu validation, profiling
# Alpaka with full reco and pixel-only quadruplets: TTbar: any backend
# Alpaka with full reco and pixel-only: TTbar: any backend quadruplets, any backend triplets
# Alpaka pixel-only quadruplets: ZMM: any backend, any backend vs cpu validation, profiling
# Alpaka pixel-only triplets: ZMM: any backend, any backend vs cpu validation, profiling
# Alpaka pixel-only quadruplets: Single Nu E10: any backend
# Patatrack pixel-only quadruplets: ZMM - on GPU (optional), GPU-vs-CPU validation, profiling
# Patatrack pixel-only triplets: ZMM - on GPU (optional), GPU-vs-CPU validation, profiling
# Patatrack pixel-only quadruplets: TTbar - on GPU (optional), GPU-vs-CPU validation, profiling
# Patatrack pixel-only triplets: TTbar - on GPU (optional), GPU-vs-CPU validation, profiling
# Patatrack ECAL-only: TTbar - on GPU (optional), GPU-vs-CPU validation, profiling
# Patatrack HCAL-only: TTbar - on GPU (optional), GPU-vs-CPU validation, profiling
# Patatrack pixel-only quadruplets, ECAL, HCAL: TTbar - on GPU (optional), GPU-vs-CPU validation
# Patatrack pixel-only triplets, ECAL, HCAL: TTbar - on GPU (optional), GPU-vs-CPU validation
# full reco with Patatrack pixel-only quadruplets: TTbar - on GPU (optional), GPU-vs-CPU validation
# full reco with Patatrack pixel-only triplets: TTbar - on GPU (optional), GPU-vs-CPU validation
# Patatrack pixel-only quadruplets: Single Nu E10 on GPU (optional)
# with PU
# Alpaka pixel-only quadruplets: TTbar with PU: any backend, any backend vs cpu validation, profiling
# Alpaka pixel-only triplets: TTbar with PU: any backend, any backend vs cpu validation, profiling
# Alpaka ECAL-only: TTbar with PU: any backend
# Alpaka HCAL-only: TTbar with PU: any backend, any backend vs cpu validation, profiling
# Alpaka with full reco and pixel-only quadruplets: TTbar with PU: any backend
# Alpaka with full reco and pixel-only: TTbar with PU: any backend quadruplets, any backend triplets
# Alpaka pixel-only quadruplets: ZMM with PU: any backend, any backend vs cpu validation, profiling
# Alpaka pixel-only triplets: ZMM with PU: any backend, any backend vs cpu validation, profiling
# Alpaka pixel-only quadruplets: Single Nu E10 with PU: any backend
# mc 2026 Patatrack pixel-only quadruplets: Single Nu E10: on GPU (optional)
# mc 2026
# no PU
# Alpaka pixel-only: TTbar: quadruplets any backend, any backend vs cpu validation, profiling, triplets
# Alpaka pixel-only: Single Nu E10: any backend
# with PU
# Alpaka pixel-only: TTbar with PU: quadruplets any backend, any backend vs cpu validation, profiling

numWFIB = [
# 2023, Alpaka-based
12434.402, 12434.403, 12434.404,
#12434.406, 12434.407, 12434.408,
12434.412,#12434.413, 12434.414,
12434.422, 12434.423, 12434.424,
#12434.482, 12434.483, 12434.484
#12434.486, 12434.487, 12434.488
12434.492,#12434.493
12450.402, 12450.403, 12450.404,
#12450.406, 12450.407, 12450.408,
12461.402,

# 2023, CUDA-based
12450.502, 12450.503, 12450.504,
12450.506, 12450.507, 12450.508,
12434.502, 12434.503, 12434.504,
12434.506, 12434.507, 12434.508,
12434.512, 12434.513, 12434.514,
12434.522, 12434.523, 12434.524,
12434.582, 12434.583,#12434.584,
12434.586, 12434.587,#12434.588,
12434.592, 12434.593,
12434.596, 12434.597,
12461.502,

# 2023 with PU, Alpaka-based
12634.402, 12634.403, 12634.404,
#12634.406, 12634.407, 12634.408
12634.412,#12634.413, 12634.414
12634.422, 12634.423, 12634.424,
#12634.482, 12634.483, 12634.484
#12634.486, 12634.487, 12634.488
12634.492,#12634.493
12650.402, 12650.403, 12650.404,
#12650.406, 12650.407, 12650.408
12661.402,

# 2026, CUDA-based
24861.502
# 2023, Alpaka-based noPU
12434.402,12434.403,12434.412,12434.422,12434.423,

# 2024, Alpaka-based noPU
12834.402, 12834.403, 12834.404,
12834.406, 12834.407, 12834.408,
12834.412,#12834.413, 12834.414,
12834.422, 12834.423, 12834.424,
#12834.482, 12834.483, 12834.484
#12834.486, 12834.487, 12834.488
12834.492, 12834.493,
12850.402, 12850.403, 12850.404,
12450.406, 12450.407, 12450.408,
12861.402,

# 2024 with PU, Alpaka-based
13034.402, 13034.403, 13034.404,
13034.406, 13034.407, 13034.408,
13034.412, #13034.413, 13034.414
13034.422, 13034.423, 13034.424,
#13034.482, 13034.483, 13034.484
#13034.486, 13034.487, 13034.488
13034.492, 13034.493,
13050.402, 13050.403, 13050.404,
13050.406, 13050.407, 13050.408,
13061.402,

# 2026, Alpaka-based noPU
29634.402, 29634.403, 29634.404, 29634.406,
29661.402,

# 2026, Alpaka-based PU
29834.402, 29834.403, 29834.404
]

for numWF in numWFIB:
Expand Down
90 changes: 80 additions & 10 deletions Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1125,7 +1125,7 @@ def setup_(self, step, stepName, stepDict, k, properties):
'--customise': 'RecoTracker/Configuration/customizePixelTracksForTriplets.customizePixelTracksForTriplets,RecoTracker/Configuration/customizePixelOnlyForProfiling.customizePixelOnlyForProfilingGPUOnly'
},
harvest = None,
suffix = 'Patatrack_PixelOnlyTripletsGPU_Profiling',
suffix = 'Patatrack_PixelOnlyTripletsGPU_Profiling',
offset = 0.508,
)

Expand All @@ -1136,7 +1136,7 @@ def setup_(self, step, stepName, stepDict, k, properties):
upgradeWFs['PatatrackECALOnlyAlpaka'] = PatatrackWorkflow(
digi = {
# customize the ECAL Local Reco part of the HLT menu for Alpaka
'--procModifiers': 'alpaka', # alpaka modifier activates customiseHLTForAlpaka
'--procModifiers': 'alpaka',
'--customise' : 'HeterogeneousCore/AlpakaServices/customiseAlpakaServiceMemoryFilling.customiseAlpakaServiceMemoryFilling',
},
reco = {
Expand Down Expand Up @@ -1312,7 +1312,7 @@ def setup_(self, step, stepName, stepDict, k, properties):
# - HCAL-only reconstruction using Alpaka with DQM and Validation
upgradeWFs['PatatrackHCALOnlyAlpakaValidation'] = PatatrackWorkflow(
digi = {
'--procModifiers': 'alpaka', # alpaka modifier activates customiseHLTForAlpaka
'--procModifiers': 'alpaka',
'--customise' : 'HeterogeneousCore/AlpakaServices/customiseAlpakaServiceMemoryFilling.customiseAlpakaServiceMemoryFilling',
},
reco = {
Expand All @@ -1332,7 +1332,7 @@ def setup_(self, step, stepName, stepDict, k, properties):
# - HCAL-only reconstruction using GPU and Alpaka with DQM and Validation for PF Alpaka vs CPU comparisons
upgradeWFs['PatatrackHCALOnlyGPUandAlpakaValidation'] = PatatrackWorkflow(
digi = {
'--procModifiers': 'alpaka', # alpaka modifier activates customiseHLTForAlpaka
'--procModifiers': 'alpaka',
'--customise' : 'HeterogeneousCore/AlpakaServices/customiseAlpakaServiceMemoryFilling.customiseAlpakaServiceMemoryFilling',
},
reco = {
Expand All @@ -1352,7 +1352,7 @@ def setup_(self, step, stepName, stepDict, k, properties):
# - HCAL-only reconstruction using Alpaka
upgradeWFs['PatatrackHCALOnlyAlpakaProfiling'] = PatatrackWorkflow(
digi = {
'--procModifiers': 'alpaka', # alpaka modifier activates customiseHLTForAlpaka
'--procModifiers': 'alpaka',
},
reco = {
'-s': 'RAW2DIGI:RawToDigi_hcalOnly,RECO:reconstruction_hcalOnly',
Expand All @@ -1369,13 +1369,13 @@ def setup_(self, step, stepName, stepDict, k, properties):
# - harvesting
upgradeWFs['PatatrackFullRecoAlpaka'] = PatatrackWorkflow(
digi = {
'--procModifiers': 'alpaka', # alpaka modifier activates customiseHLTForAlpaka
'--procModifiers': 'alpaka',
'--customise' : 'HeterogeneousCore/AlpakaServices/customiseAlpakaServiceMemoryFilling.customiseAlpakaServiceMemoryFilling',
},
reco = {
# skip the @pixelTrackingOnlyValidation which cannot run together with the full reconstruction
'-s': 'RAW2DIGI:RawToDigi+RawToDigi_pixelOnly,L1Reco,RECO:reconstruction+reconstruction_pixelTrackingOnly,RECOSIM,PAT,VALIDATION:@standardValidation+@miniAODValidation,DQM:@standardDQM+@ExtraHLT+@miniAODDQM+@pixelTrackingOnlyDQM',
'--procModifiers': 'alpaka,pixelNtupletFit',
'--procModifiers': 'alpaka',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will this still run the pixel ntuplet reconstruction ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep. Since the transition to Alpaka all the chain is activateed just using the alpaka modifier. The pixelNtupletFit one was there because we had the two different chains CPU (from clusters) and GPU (from RAW) and needed a way to distinguish them. When we clean up the CUDA part, pixelNtupletFit can go.

'--customise' : 'HeterogeneousCore/AlpakaServices/customiseAlpakaServiceMemoryFilling.customiseAlpakaServiceMemoryFilling',
},
harvest = {
Expand All @@ -1385,6 +1385,28 @@ def setup_(self, step, stepName, stepDict, k, properties):
offset = 0.492,
)

# Workflow running the Pixel triplets, ECAL and HCAL reconstruction on GPU (optional), PF using Alpaka, together with the full offline reconstruction on CPU
# - HLT on GPU (optional)
# - reconstruction on Alpaka, with DQM and validation
# - harvesting
upgradeWFs['PatatrackFullRecoAlpaka'] = PatatrackWorkflow(
digi = {
'--procModifiers': 'alpaka',
'--customise' : 'HeterogeneousCore/AlpakaServices/customiseAlpakaServiceMemoryFilling.customiseAlpakaServiceMemoryFilling',
},
reco = {
# skip the @pixelTrackingOnlyValidation which cannot run together with the full reconstruction
'-s': 'RAW2DIGI:RawToDigi+RawToDigi_pixelOnly,L1Reco,RECO:reconstruction+reconstruction_pixelTrackingOnly,RECOSIM,PAT,VALIDATION:@standardValidation+@miniAODValidation,DQM:@standardDQM+@ExtraHLT+@miniAODDQM+@pixelTrackingOnlyDQM',
'--procModifiers': 'alpaka',
'--customise' : 'RecoTracker/Configuration/customizePixelTracksForTriplets.customizePixelTracksForTriplets,HeterogeneousCore/AlpakaServices/customiseAlpakaServiceMemoryFilling.customiseAlpakaServiceMemoryFilling',
},
harvest = {
# skip the @pixelTrackingOnlyDQM harvesting
},
suffix = 'Patatrack_FullRecoAlpakaTriplets',
offset = 0.496,
)

# Workflow running the Pixel quadruplets, ECAL and HCAL reconstruction on CPU
# - HLT on CPU
# - reconstruction on CPU, with DQM and validation
Expand Down Expand Up @@ -1691,7 +1713,7 @@ def setup_(self, step, stepName, stepDict, k, properties):

upgradeWFs['PatatrackPixelOnlyAlpaka'] = PatatrackWorkflow(
digi = {
'--procModifiers': 'alpaka', # alpaka modifier activates customiseHLTForAlpaka
'--procModifiers': 'alpaka',
'--customise' : 'HeterogeneousCore/AlpakaServices/customiseAlpakaServiceMemoryFilling.customiseAlpakaServiceMemoryFilling',
},
reco = {
Expand All @@ -1708,7 +1730,7 @@ def setup_(self, step, stepName, stepDict, k, properties):

upgradeWFs['PatatrackPixelOnlyAlpakaValidation'] = PatatrackWorkflow(
digi = {
'--procModifiers': 'alpaka', # alpaka modifier activates customiseHLTForAlpaka
'--procModifiers': 'alpaka',
'--customise' : 'HeterogeneousCore/AlpakaServices/customiseAlpakaServiceMemoryFilling.customiseAlpakaServiceMemoryFilling',
},
reco = {
Expand All @@ -1725,7 +1747,7 @@ def setup_(self, step, stepName, stepDict, k, properties):

upgradeWFs['PatatrackPixelOnlyAlpakaProfiling'] = PatatrackWorkflow(
digi = {
'--procModifiers': 'alpaka', # alpaka modifier activates customiseHLTForAlpaka
'--procModifiers': 'alpaka',
},
reco = {
'-s': 'RAW2DIGI:RawToDigi_pixelOnly,RECO:reconstruction_pixelTrackingOnly',
Expand All @@ -1737,6 +1759,54 @@ def setup_(self, step, stepName, stepDict, k, properties):
offset = 0.404,
)

upgradeWFs['PatatrackPixelOnlyTripletsAlpaka'] = PatatrackWorkflow(
digi = {
'--procModifiers': 'alpaka',
'--customise' : 'HeterogeneousCore/AlpakaServices/customiseAlpakaServiceMemoryFilling.customiseAlpakaServiceMemoryFilling',
},
reco = {
'-s': 'RAW2DIGI:RawToDigi_pixelOnly,RECO:reconstruction_pixelTrackingOnly,VALIDATION:@pixelTrackingOnlyValidation,DQM:@pixelTrackingOnlyDQM',
'--procModifiers': 'alpaka',
'--customise' : 'RecoTracker/Configuration/customizePixelTracksForTriplets.customizePixelTracksForTriplets,HeterogeneousCore/AlpakaServices/customiseAlpakaServiceMemoryFilling.customiseAlpakaServiceMemoryFilling'
},
harvest = {
'-s': 'HARVESTING:@trackingOnlyValidation+@pixelTrackingOnlyDQM'
},
suffix = 'Patatrack_PixelOnlyTripletsAlpaka',
offset = 0.406,
)

upgradeWFs['PatatrackPixelOnlyTripletsAlpakaValidation'] = PatatrackWorkflow(
digi = {
'--procModifiers': 'alpaka',
'--customise' : 'HeterogeneousCore/AlpakaServices/customiseAlpakaServiceMemoryFilling.customiseAlpakaServiceMemoryFilling',
},
reco = {
'-s': 'RAW2DIGI:RawToDigi_pixelOnly,RECO:reconstruction_pixelTrackingOnly,VALIDATION:@pixelTrackingOnlyValidation,DQM:@pixelTrackingOnlyDQM',
'--procModifiers': 'alpakaValidation',
'--customise' : 'RecoTracker/Configuration/customizePixelTracksForTriplets.customizePixelTracksForTriplets,HeterogeneousCore/AlpakaServices/customiseAlpakaServiceMemoryFilling.customiseAlpakaServiceMemoryFilling'
},
harvest = {
'-s': 'HARVESTING:@trackingOnlyValidation+@pixelTrackingOnlyDQM'
},
suffix = 'Patatrack_PixelOnlyTripletsAlpaka_Validation',
offset = 0.407,
)

upgradeWFs['PatatrackPixelOnlyTripletsAlpakaProfiling'] = PatatrackWorkflow(
digi = {
'--procModifiers': 'alpaka',
},
reco = {
'-s': 'RAW2DIGI:RawToDigi_pixelOnly,RECO:reconstruction_pixelTrackingOnly',
'--procModifiers': 'alpaka',
'--customise' : 'RecoTracker/Configuration/customizePixelTracksForTriplets.customizePixelTracksForTriplets,RecoTracker/Configuration/customizePixelOnlyForProfiling.customizePixelOnlyForProfilingGPUOnly'
},
harvest = None,
suffix = 'Patatrack_PixelOnlyTripletsAlpaka_Profiling',
offset = 0.408,
)

# end of Patatrack workflows

class UpgradeWorkflow_ProdLike(UpgradeWorkflow):
Expand Down
19 changes: 17 additions & 2 deletions DQM/SiPixelHeterogeneous/python/SiPixelHeterogenousDQM_FirstStep_cff.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -239,7 +239,7 @@
siPixelVertexSoAMonitorDevice *
siPixelCompareVertices )

# Phase-2 sequence
# Phase-2 sequence ...
_monitorpixelSoACompareSource = cms.Sequence(siPixelPhase2MonitorRecHitsSoACPU *
siPixelPhase2MonitorRecHitsSoAGPU *
siPixelPhase2CompareRecHitsSoA *
Expand All @@ -250,6 +250,18 @@
siPixelMonitorVertexSoAGPU *
siPixelCompareVertexSoA)

# ...and the Alpaka version
_monitorpixelSoACompareSourceAlpakaPhase2 = cms.Sequence(
siPixelRecHitsSoAMonitorSerial *
siPixelRecHitsSoAMonitorDevice *
siPixelPhase1CompareRecHits *
siPixelTrackSoAMonitorSerial *
siPixelTrackSoAMonitorDevice *
siPixelPhase1CompareTracks *
siPixelVertexSoAMonitorSerial *
siPixelVertexSoAMonitorDevice *
siPixelCompareVertices )

# HIon sequence
_monitorpixelSoACompareSourceHIonPhase1 = cms.Sequence(siPixelHIonPhase1MonitorRecHitsSoACPU *
siPixelHIonPhase1MonitorRecHitsSoAGPU *
Expand All @@ -262,9 +274,12 @@
siPixelCompareVertexSoA)

phase2_tracker.toReplaceWith(monitorpixelSoACompareSource,_monitorpixelSoACompareSource)
phase2_tracker.toReplaceWith(monitorpixelSoACompareSourceAlpaka,_monitorpixelSoACompareSourceAlpakaPhase2)

from Configuration.ProcessModifiers.gpuValidationPixel_cff import gpuValidationPixel
gpuValidationPixel.toReplaceWith(monitorpixelSoASource, monitorpixelSoACompareSource)

from Configuration.ProcessModifiers.alpakaValidationPixel_cff import alpakaValidationPixel
(alpakaValidationPixel & ~gpuValidationPixel).toReplaceWith(monitorpixelSoASource, monitorpixelSoACompareSourceAlpaka)
(alpakaValidationPixel & ~gpuValidationPixel ).toReplaceWith(monitorpixelSoASource, monitorpixelSoACompareSourceAlpaka)


15 changes: 15 additions & 0 deletions DataFormats/VertexSoA/interface/ZVertexDefinitions.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,21 @@ namespace zVertex {
constexpr uint32_t MAXTRACKS = 32 * 1024;
constexpr uint32_t MAXVTX = 1024;

//
// FIXME: MAXTRACKS is low for Phase2 pixel triplets with PU=200
// and the txiplets wfs in those conditions will fail.
//
// Not rising it since to the needed 128*1024 since it causes
// a 4-5% jump in memory usage.
//
// The original sin is that, for the moment, the VertexSoA
// has a unique index for the tracks and the vertices and
// so it needs to be sized for the bigger of the two (the tracks, of course).
// This means we are wasting memory (this is true also for Phase1).
// We need to split the two indices (as is for CUDA, since we were not using
// the PortableCollection + Layout was not yet used).
//

} // namespace zVertex

#endif
Loading