remove unneeded fields for dev/ufs-weather-model #1317

DeniseWorthen · 2024-11-08T19:49:13Z

Pull Request Summary

Removes un-needed allocations and time-interpolation of import fields when CMEPS is used. Note this PR is built on top of the PIO PR. Changes are minimal when compared to that PR.

Description

Introduces a logical flag (use_cmeps) which is false by default. When true, arrays that are not needed at the global domain resolution are minimally allocated and time-interpolation w/in the update routines is skipped.

To test the impact, three cases were run using the same DATM-S2SW configuration used for testing the PIO work on Gaea. Each case was run out 48 hours on 816 WAV tasks, with WW3 restarts written every 2 hours.

The ProfileMemory configuration flag was turned on and memory statistics for the WW3 PETs were pulled out for every 50th PE (18 total traces). Three values were plotted: VmPeak, VmRSS and CommittedPercent. The first is the max resident size, the second is the mean resident size and the third is (as defined here) an interesting number I happened to notice in the output. I'm not sure of this last value, but it seems relevant---for example, if I use too few tasks for WAV, the model crashes and I also see values like CommittedPercent: 166.939 %.

Each of the following figures show the current dev/ufs-weather-model branch (top), the feature/pio4ww3 branch (middle) and this feature branch (bottom):

VmPeak:

VmRss:

Note here that the highest trace on the top figure shows the values for the last (naprst) PET.

CommittedPercent:

Note here that the lowest trace on all figures is the first PET.

My interpretation of these results is that a) the PIO feature itself reduces memory usage by reducing the high-memory cost of the naprst task and b) this feature has a measurable impact on the memory footprint of the model. However, no consistent impact on model execution time was found in my limited testing.

Issue(s) addressed

fixes allocation of un-needed arrays in dev/ufs-weather-model #1315

Commit Message

minimally allocate unneeded arrays when CMEPS is in use. Skip time-interpolation for imported fields.

Check list

Branch is up to date with the authoritative repository (NOAA-EMC) develop branch.
Checked the checklist for a developer submitting to develop.
If a version number update is required, checked the updating version number checklist.
If a new feature was added, a regression test for testing the new feature is added.

Testing

How were these changes tested?
Are the changes covered by regression tests? (If not, why? Do new tests need to be added?)
Have the matrix regression tests been run (if yes, please note HPC and compiler)?
Please indicate the expected changes in the regression test output, (Note the list of known non-identical tests.)
Please provide the summary output of matrix.comp (matrix.Diff.txt, matrixCompFull.txt and matrixCompSummary.txt):

I ran this PR against a baseline created with PR #1303. All baselines passed.

* remove wav_grdout routine, now moved into wav_history_mod

* remove more cap stuff from w3iorsmd. only ww3 native filenaming is possible w/ binary restarts * remove ifdef w3_ascii from w3wavemd, since the ascii commit is not yet present in mesh cap branch * clean up config variable logic for filenaming

* nrqrs can be non-zero if also using the restart from binary option

* flpart needs to be set for either historync or not

* move restart and history nc blocks outside of ww3 time testing block. * add log print messages for reading and writing restarts

* get logging working correctly for ufs * fix noclobber variable and file name in wav_history * clarify some comments

* make binary history files match when restartnc=true * rework wav_restart_mod, which originally was designed to be able to read and write restarts for testing purposes from inside wav_comp_nuopc.

* verboselog is true by default, but can be set false by config * the header text for ww3 logging as it steps through time is now turned off in w3init and placed into the mesh cap. this allows the mesh cap to order the logging correctly

* move block where addrstflds was set to before call to w3init since restarts are read in w3init * ensure that if nml file lacks a specification of extra fields, the default value of "unset" will not be returned as a field name

* only ice is added for now * tab cleanup in w3grid

* need to send explicit array bounds for ice array since it is 0:nsea * all baselines b4b against f9531d0

* intialize floutg and floutg2 which are can be unintialized when waves are in slow loop and historync is true

* local arrays dimension nx or greater are locally allocated * deallocations added, even though not required for user clarity

…o feature/pio4ww3

* global_input and _output can be allocatable

This reverts commit 7244b76.

* remove unused variables

* minimally allocate fields used in w3updtmd when using cmeps * use fields provided by cmeps w/o fake time-interpolation

…nto feature/nodupfields

DeniseWorthen · 2025-05-16T16:18:31Z

An update on this PR; I've known for quite some time that I was seeing non-debug gnu test failures with this branch. GNU +debug itself passes, as do all the INTEL,INTEL+DEBUG. I've just tested on Hera w/ the newly re-enabled GNU tests and I get the same result. On Hera, these two tests fail comparison.

cpld_control_p8_gnu
cpld_control_pdlib_p8_gnu

The same two tests pass on Hercules.

EDIT: The Hera+GNU tests do pass against a generated baselines. Since the DEBUG tests pass against the existing baseline, I think the Hera failures are due to a combination of compiler, along w/ order-of-operation differences in the PR branch.

DeniseWorthen · 2025-05-22T22:36:16Z

I repeated the test cases above using the current develop and this feature branch. I ran two test cases on C6, using the DATM-S2SW configuration and the mesh.uglo_15km.nc. All I/O was off for all components. The cases were identical, differing only in the WW3 branch.

The VmRSS for the two cases over a 4-day integration is shown for the develop branch (upper) and this feature branch (lower)

The difference in the CommittedPercent values was ~3% over the course of the integration.

JessicaMeixner-NOAA · 2025-05-27T20:39:34Z

@DeniseWorthen -
I have run the standalone WW3 tests and we just get the expected non B4B so I can confirm no unintentional consequences on the standalone side. I also went through the code itself for double checking and thought things looked good.

I'll run these again when it's closer to time, so I'm not putting the full logs but the standalone regtest diffs:

**********************************************************************
********************* non-identical cases ****************************
**********************************************************************
mww3_test_03/./work_PR3_UQ_MPI_e_c                     (1 files differ)
mww3_test_03/./work_PR3_UNO_MPI_e                     (1 files differ)
mww3_test_03/./work_PR2_UQ_MPI_e                     (1 files differ)
mww3_test_03/./work_PR2_UNO_MPI_d2                     (8 files differ)
mww3_test_03/./work_PR1_MPI_d2                     (10 files differ)
mww3_test_03/./work_PR3_UNO_MPI_d2_c                     (13 files differ)
mww3_test_03/./work_PR3_UQ_MPI_d2_c                     (15 files differ)
mww3_test_03/./work_PR3_UNO_MPI_d2                     (8 files differ)
mww3_test_03/./work_PR2_UQ_MPI_d2                     (15 files differ)
mww3_test_03/./work_PR3_UNO_MPI_e_c                     (1 files differ)
mww3_test_03/./work_PR3_UQ_MPI_d2                     (16 files differ)
mww3_test_09/./work_MPI_ASCII                     (0 files differ)
ww3_tp2.10/./work_MPI_OMPH                     (6 files differ)
ww3_tp2.16/./work_MPI_OMPH                     (4 files differ)
ww3_tp2.6/./work_ST4_ASCII                     (0 files differ)
ww3_ufs1.3/./work_a                     (3 files differ)

DeniseWorthen · 2025-05-27T20:47:38Z

@JessicaMeixner-NOAA One thing I noticed (but did not make a change for) is that there seem to be some missing OMP statements in w3updtmd.F90

WW3/model/src/w3updtmd.F90

Lines 999 to 1022 in f27fe03

    
           #ifdef W3_OMPG 
        
               !$OMP PARALLEL DO PRIVATE (ISEA,MI2,MXR,MYR,MDARC) 
        
           #endif 
        
               ! 
        
               DO ISEA=1, NSEA 
        
                 ! 
        
                 TAUA(ISEA) = MA0(ISEA) + RD * MAI(ISEA) 
        
           #ifdef W3_WNT2 
        
                 MI2      = SQRT ( RD2 *      MA0(ISEA)**2 +             & 
        
                      RD  *(MA0(ISEA)+MAI(ISEA))**2 ) 
        
                 TAUA(ISEA) = TAUA(ISEA) * MIN(1.25,MI2/MAX(1.E-7,TAUA(ISEA))) 
        
           #endif 
        
                 TAUADIR(ISEA) = MD0(ISEA) + RD * MDI(ISEA) 
        
           #ifdef W3_SMC 
        
                 !Li   Rotate momentum direction by ANGARC for Arctic part cells. 
        
                 IF( ARCTC .AND. (ISEA .GT. NGLO) ) THEN 
        
                   MDARC = TAUADIR(ISEA) + ANGARC( ISEA - NGLO )*DERA 
        
                   TAUADIR(ISEA) = MOD ( TPI + MDARC, TPI ) 
        
                 ENDIF 
        
           #endif 
        
                 ! 
        
               END DO 
        
               ! 
        
               RETURN

and

WW3/model/src/w3updtmd.F90

Lines 2697 to 2707 in f27fe03

    
           #ifdef W3_OMPG 
        
               !$OMP PARALLEL DO PRIVATE (ISEA,RA0,RAI) 
        
           #endif 
        
               ! 
        
               DO ISEA=1, NSEA 
        
                 ! 
        
                 RHOAIR(ISEA) = RA0(ISEA) + RD * RAI(ISEA) 
        
                 ! 
        
               END DO 
        
               ! 
        
               RETURN

JessicaMeixner-NOAA · 2025-05-27T23:41:08Z

Thanks for pointing that out. I made an issue: #1444 so we can make sure to go back to that.

JessicaMeixner-NOAA

**********************************************************************
********************* non-identical cases ****************************
**********************************************************************
mww3_test_03/./work_PR1_MPI_e                     (1 files differ)
mww3_test_03/./work_PR3_UQ_MPI_e_c                     (1 files differ)
mww3_test_03/./work_PR3_UNO_MPI_e                     (1 files differ)
mww3_test_03/./work_PR2_UQ_MPI_e                     (1 files differ)
mww3_test_03/./work_PR2_UNO_MPI_d2                     (15 files differ)
mww3_test_03/./work_PR1_MPI_d2                     (10 files differ)
mww3_test_03/./work_PR3_UNO_MPI_d2_c                     (16 files differ)
mww3_test_03/./work_PR3_UQ_MPI_d2_c                     (15 files differ)
mww3_test_03/./work_PR3_UNO_MPI_d2                     (15 files differ)
mww3_test_03/./work_PR2_UQ_MPI_d2                     (15 files differ)
mww3_test_03/./work_PR3_UQ_MPI_e                     (1 files differ)
mww3_test_03/./work_PR3_UNO_MPI_e_c                     (1 files differ)
mww3_test_03/./work_PR3_UQ_MPI_d2                     (15 files differ)
mww3_test_09/./work_MPI_ASCII                     (0 files differ)
ww3_tp2.10/./work_MPI_OMPH                     (7 files differ)
ww3_tp2.16/./work_MPI_OMPH                     (4 files differ)
ww3_tp2.6/./work_ST4_ASCII                     (0 files differ)
ww3_ufs1.3/./work_a                     (2 files differ)

matrixCompFull.txt
matrixCompSummary.txt
matrixDiff.txt

Thank you @DeniseWorthen !!

DeniseWorthen and others added 30 commits August 28, 2024 08:06

clean branch for pio work

31d7f36

add new files

0fe584a

rename w3iogoncmd as wav_history_mod

099645b

rework order to enable PIO initialization

c7a9ee3

* remove wav_grdout routine, now moved into wav_history_mod

prevent mpi_startalls if restartnc is active

efd5774

* nrqrs can be non-zero if also using the restart from binary option

move setting of flpart flag

3b038ff

* flpart needs to be set for either historync or not

remove unneeded CESMCOUPLED from w3iorsmd

6e32b5f

update nc block location, add log messages

9976c4d

* move restart and history nc blocks outside of ww3 time testing block. * add log print messages for reading and writing restarts

use pio_ioformat for create mode

32c5f25

make iosys a pointer, add debug level config

4edfbc6

fix ww3 thrd cases

fbeb83e

remove stray bin2nc ifdef

5aedf8c

fixes for logs and wav_history

4f62341

* get logging working correctly for ufs * fix noclobber variable and file name in wav_history * clarify some comments

resolve issue for binary history files

553f1b1

* make binary history files match when restartnc=true * rework wav_restart_mod, which originally was designed to be able to read and write restarts for testing purposes from inside wav_comp_nuopc.

fix multiple writes when waves are in slow loop

8d0f170

rename oldst2 variable in wav_restart

9cb49d1

add verbose log option

4674dae

* verboselog is true by default, but can be set false by config * the header text for ww3 logging as it steps through time is now turned off in w3init and placed into the mesh cap. this allows the mesh cap to order the logging correctly

add optional restart files via nml

7c43376

resolve issue w/ nml for extra fields

f9531d0

* move block where addrstflds was set to before call to w3init since restarts are read in w3init * ensure that if nml file lacks a specification of extra fields, the default value of "unset" will not be returned as a field name

add generic read/write for added restart fields

ad51cd6

* only ice is added for now * tab cleanup in w3grid

fix generic read/write in wav_restart

d92e9ff

* need to send explicit array bounds for ice array since it is 0:nsea * all baselines b4b against f9531d0

initialize variables

e0134de

* intialize floutg and floutg2 which are can be unintialized when waves are in slow loop and historync is true

replace statically allocated arrays to reduce memory

6465186

* local arrays dimension nx or greater are locally allocated * deallocations added, even though not required for user clarity

Merge remote-tracking branch 'origin/feature/reducememuse_triamd' int…

e556ea8

…o feature/pio4ww3

in wav_restart, make arrays allocatable

008272c

* global_input and _output can be allocatable

write va in nspec loop

9b0f44c

fix compile error

e690219

Merge branch 'dev/ufs-weather-model' into feature/pio4ww3

91f1111

Merge branch 'dev/ufs-weather-model' into feature/debugsubset

c25510f

DeniseWorthen added 6 commits January 22, 2025 16:04

Revert "trigp test branch"

93cc635

This reverts commit 7244b76.

Merge branch 'NOAA-EMC:dev/ufs-weather-model' into dev/ufs-weather-model

51c2d29

Merge branch 'dev/ufs-weather-model' into feature/nodupfields

d31915b

Merge branch 'NOAA-EMC:dev/ufs-weather-model' into dev/ufs-weather-model

60a7b1b

Merge branch 'dev/ufs-weather-model' into feature/nodupfields

09c1f2f

remove extra blank line

b780d99

DeniseWorthen mentioned this pull request Feb 26, 2025

Reduce WW3 memory use ufs-community/ufs-weather-model#2624

Merged

13 tasks

DeniseWorthen added 2 commits April 15, 2025 12:40

fix type conversions found w/ GNU+debug on Hera

5ce8b54

* remove unused variables

Merge branch 'dev/ufs-weather-model' into feature/nodupfields

5647bed

DeniseWorthen force-pushed the feature/nodupfields branch from 6ffebef to a64f2dd Compare April 22, 2025 22:56

reduce allocated forcing fields when using cmeps

525e749

* minimally allocate fields used in w3updtmd when using cmeps * use fields provided by cmeps w/o fake time-interpolation

DeniseWorthen force-pushed the feature/nodupfields branch from 5f370a4 to 525e749 Compare April 23, 2025 15:04

DeniseWorthen and others added 5 commits April 23, 2025 17:57

Merge remote-tracking branch 'origin/bugfix/w3sdb1_typeconversions' i…

1ae507f

…nto feature/nodupfields

control_pdlib_p8_gnu fails even w/ sdb1 fix, all others B4B

5abb625

Merge branch 'dev/ufs-weather-model' into feature/nodupfields

d9c1b5d

remove duplicate code block from merge

fbd5452

Merge branch 'dev/ufs-weather-model' into feature/nodupfields

57f9c45

DeniseWorthen added 2 commits May 27, 2025 06:13

tidy up

5949516

revert unintentional blank line changes

356aa9c

DeniseWorthen marked this pull request as ready for review May 27, 2025 14:56

Merge branch 'dev/ufs-weather-model' into feature/nodupfields

aa508be

JessicaMeixner-NOAA mentioned this pull request May 27, 2025

Missing OMP statements in w3updatmd #1444

Open

JessicaMeixner-NOAA self-requested a review May 29, 2025 18:35

JessicaMeixner-NOAA approved these changes May 29, 2025

View reviewed changes

JessicaMeixner-NOAA merged commit 9939046 into NOAA-EMC:dev/ufs-weather-model Jul 1, 2025
3 of 6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

remove unneeded fields for dev/ufs-weather-model #1317

remove unneeded fields for dev/ufs-weather-model #1317

Uh oh!

DeniseWorthen commented Nov 8, 2024 •

edited

Loading

Uh oh!

DeniseWorthen commented May 16, 2025 •

edited

Loading

Uh oh!

DeniseWorthen commented May 22, 2025 •

edited

Loading

Uh oh!

JessicaMeixner-NOAA commented May 27, 2025

Uh oh!

DeniseWorthen commented May 27, 2025

Uh oh!

JessicaMeixner-NOAA commented May 27, 2025

Uh oh!

JessicaMeixner-NOAA left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

remove unneeded fields for dev/ufs-weather-model #1317

remove unneeded fields for dev/ufs-weather-model #1317

Uh oh!

Conversation

DeniseWorthen commented Nov 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Summary

Description

Issue(s) addressed

Commit Message

Check list

Testing

Uh oh!

DeniseWorthen commented May 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DeniseWorthen commented May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JessicaMeixner-NOAA commented May 27, 2025

Uh oh!

DeniseWorthen commented May 27, 2025

Uh oh!

JessicaMeixner-NOAA commented May 27, 2025

Uh oh!

JessicaMeixner-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

DeniseWorthen commented Nov 8, 2024 •

edited

Loading

DeniseWorthen commented May 16, 2025 •

edited

Loading

DeniseWorthen commented May 22, 2025 •

edited

Loading