Enable GPU exection of atm_bdy_adjust_scalars_work via OpenACC #1266
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Small whitespace changes. Also change implicit loop to an explicit loop to better parallelize. Implicit loops can be ported with 'acc kernels', but we prefer more proscribed 'acc parallel ...' constructs.
This commit adds an initial port of this routine using OpenACC. More changes are needed for performance and data management.
Ensures the fields which don't change while the model is running are present on the device from model startup to model shutdown.
Ensure that the other, non-invariant fields are available for this routine. Variables that are overwritten during this routine are only created while others are copied in. Any variables overwritten by this routine are copied out at the end. Timing for these transfers are reported in the output log file in the new timer: 'atm_bdy_adjust_scalars [ACC_data_xfer]'. Also add default(present) to parallel directives to ensure data movement is correct and prevent any implicit data movements from the compiler.
jim-p-w
approved these changes
Jan 23, 2025
abishekg7
reviewed
Feb 1, 2025
| do iCell = cellSolveStart, cellSolveEnd ! threaded over cells | ||
|
|
||
| if ( (bdyMaskCell(iCell) > 1) .and. (bdyMaskCell(iCell) <= nRelaxZone) ) then ! relaxation zone | ||
|
|
||
| laplacian_filter_coef = dt_rk*(real(bdyMaskCell(iCell)) - 1.)/real(nRelaxZone)/(10.*dt*meshScalingRegionalCell(iCell)) | ||
| rayleigh_damping_coef = laplacian_filter_coef/5.0 | ||
| scalars_tmp(1:num_scalars,1:nVertLevels,iCell) = scalars_new(1:num_scalars,1:nVertLevels,iCell) | ||
|
|
There was a problem hiding this comment.
Do we need a newline here by style conventions?
There was a problem hiding this comment.
I'm not sure about style conventions, I just thought it helped it read easier. It's removed to keep the changes minimal
abishekg7
reviewed
Feb 1, 2025
| @@ -6799,6 +6821,7 @@ subroutine atm_bdy_adjust_scalars_work( scalars_new, scalars_driving, dt, dt_rk, | |||
| cell1 = cellsOnEdge(1,iEdge) | |||
| cell2 = cellsOnEdge(2,iEdge) | |||
| !DIR$ IVDEP | |||
| !$acc loop collapse(2) | |||
There was a problem hiding this comment.
I guess the compiler will implicitly assume a vector here, but wondering if we might need to also explicitly state that here for sake of convention.
There was a problem hiding this comment.
Thanks for the catch. It's been addressed in the fixup commits pushed today.
|
Other than my comments, this PR seems to be bit identical with the previous version of develop. |
Keep consistency with explicit loop schedules
Remove newline, keep changes minimal
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR makes small code modifications and adds OpenACC directives so the atm_bdy_adjust_scalars_work routine can execute on GPU(s).
Timing information for the OpenACC data transfers in this routine is captured in the log file by a new timer: atm_bdy_adjust_scalars [ACC_data_xfer].
Invariant fields used in this routine are also copied to the device within mpas_atm_dynamics_init and are deleted in mpas_atm_dynamics_finalize.