Updates to progomega option of saSAS convection scheme

[lisa-bengtsson]

Commit Queue Requirements:

• This PR addresses a relevant WM issue (if not, create an issue).
• All subcomponent pull requests (if any) have been reviewed by their code managers.
• Run the full Intel+GNU RT suite (compared to current baselines), preferably on Ursa (Derecho or Hercules are acceptable alternatives). Exceptions: documentation-only PRs, CI-only PRs, etc.
  • Commit log file w/full results from RT suite run (if applicable).
  • Verify that test_changes.list indicates which tests, if any, are changed by this PR. Commit test_changes.list, even if it is empty.
• Fill out all sections of this template.

---

Description:

This PR brings in improvements, cleaning and tuning of the prognostic updraft velocity parameterization under the progomega flag in saSAS. The new parameterization improves the representation of updrafts and cloud base mass-flux compared with lidar observations.

Commit Message:

* UFSWM - Update submodule pointer to UFSATM
  * UFSATM -  Update submodule pointer to ccpp-physics
    * ccpp-physics - Include new code under progomega option in saSAS convection

Priority:

• Critical Bugfix: Reason
• High: Reason
• [x ] Normal

Git Tracking

UFSWM:

• Closes # Prognostic Updraft Velocity refinements #3074

Sub component Pull Requests:

• UFSATM: Updates to progomega option of saSAS convection scheme NOAA-EMC/ufsatm#1077
  • ccpp-physics: Updates to progomega option of saSAS convection scheme ccpp-physics#364

UFSWM Blocking Dependencies:

• Blocked by #
• None

Documentation:

• Documentation update required.
  • Relevant updates are included with this PR.
  • A WM issue has been opened to track the need for a documentation update; a person responsible for submitting the update has been assigned to the issue (link issue).
• Documentation update NOT required.
  • Explanation: Just tuning, cleaning, refinement of existing scheme

---

Changes

Regression Test Changes (Please commit test_changes.list):

• PR Adds New Tests/Baselines.
• PR Updates/Changes Baselines.
• No Baseline Changes.

Input data Changes:

• None.
• PR adds input data.
• PR changes existing input data.

Library Changes/Upgrades:

• Required
  • Library names w/versions:
  • Git Stack Issue (JCSDA/spack-stack#)
• No Updates

---

Testing Log:

• RDHPCS
  • Orion
  • Hercules
  • GaeaC6
  • Derecho
  • Ursa
• WCOSS2
  • Dogwood/Cactus
  • Acorn
• CI
• opnReqTest (complete task if unnecessary)

[gspetro-NOAA]

@lisa-bengtsson If work on this PR is complete, please make sure to run the full RT suite, preferably on Ursa, and push the logs and the test_changes.list file, which will show which baselines will change.

[lisa-bengtsson]

@lisa-bengtsson If work on this PR is complete, please make sure to run the full RT suite, preferably on Ursa, and push the logs and the test_changes.list file, which will show which baselines will change.

@gspetro-NOAA I have now updated the logfile from Ursa and the test_changes.list I think the main CCPP PR still needs to be reviewed.

[gspetro-NOAA]

@gspetro-NOAA I have now updated the logfile from Ursa and the test_changes.list I think the main CCPP PR still needs to be reviewed.

Great, thanks! I'll see if I can prompt folks to review.

[gspetro-NOAA]

@lisa-bengtsson Please sync with develop whenever you have a chance. We'd like to process your PR next. :)

[lisa-bengtsson]

@gspetro-NOAA I have synced with develop in all the submodules, let me know if there's anything else I should do. Thanks!

[gspetro-NOAA]

@lisa-bengtsson Baseline creation failed for three tests:

• cpld_debug_gfsv17_intel (Hercules/Orion/Derecho)
• cpld_debug_sfs_intel (Hercules/Orion/Derecho)
• hafs_global_multiple_4nests_atm_intel (Derecho only)

The first errors I come across in the cpld_debug_gfsv17_intel err file are several like: forrtl: error (65): floating invalid; then there are a bunch of forrtl: error (78): process killed (SIGTERM).
The out file stops after 0: in radiation_clouds_prop= 8 F 4 F F 2 1.
cpld_debug_sfs_intel doesn't have the floating point invalid error, but it stops in a similar place with radiation_clouds_prop. I can send you my run_dir, but let me know what platform is best for you of those three.

For the hafs case, there are several SIGTERM errors, and the out file stops at enter get_nggps_ic is= 81 ie= 120 js= 217 je= 240 isd= 78 ied= 123 jsd= 214 jed= 243. That case only failed on Derecho, but idk if you have access to that platform?

[DeniseWorthen]

The log file posted at a217654 showed multiple RUN DID NOT COMPLETE messages.

[gspetro-NOAA]

The log file posted at a217654 showed multiple RUN DID NOT COMPLETE messages.

Oof! You're right 😞
@lisa-bengtsson On your Ursa log, I see the following runs did not complete:

• cpld_debug_sfs_intel
• cpld_debug_gfsv17_intel
• hafs_global_multiple_4nests_atm_intel
• cpld_debug_gfsv17_intelllvm
  I think we may postpone this PR to give you time to troubleshoot those failing tests.

@DeniseWorthen I might also try to add a CI check for FAILED: RUN DID NOT COMPLETE messages (in addition to the warnings/remarks increases) so that they don't get lost among all the expected comparison failures and restart "did not start" messages. It seems like maybe I should also check for: FAILED: UNABLE TO START COMPILE and FAILED: UNABLE TO FINISH COMPILE. It seems less necessary to check for the Disk Quota and timeout failures, but not sure if you have thoughts on that.

[DeniseWorthen]

I think adding auto-checking would be a good idea. I'm guessing that if someone is expecting the baselines to fail, they might not notice when it fails for something other than not comparing.

The disk quota issue has maybe lessened since they implemented the scrubber on Ursa. But I wouldn't remove it, even so. I know I've spent time scratching my head why something failed, only to finally figure out it was quota.

[lisa-bengtsson]

I think adding auto-checking would be a good idea. I'm guessing that if someone is expecting the baselines to fail, they might not notice when it fails for something other than not comparing.

The disk quota issue has maybe lessened since they implemented the scrubber on Ursa. But I wouldn't remove it, even so. I know I've spent time scratching my head why something failed, only to finally figure out it was quota.

Ah, my apologies, Denise is right I only checked for failed RT's. Looks like something is not working in debug mode, let me see what I can find.

[lisa-bengtsson]

I think adding auto-checking would be a good idea. I'm guessing that if someone is expecting the baselines to fail, they might not notice when it fails for something other than not comparing.
The disk quota issue has maybe lessened since they implemented the scrubber on Ursa. But I wouldn't remove it, even so. I know I've spent time scratching my head why something failed, only to finally figure out it was quota.

Ah, my apologies, Denise is right I only checked for failed RT's. Looks like something is not working in debug mode, let me see what I can find.

I found the problem, and re-running the full regression test on Ursa now.

[gspetro-NOAA]

I found the problem, and re-running the full regression test on Ursa now.

Great! If all goes well, we can try again on Monday/Tuesday. 🙂

[lisa-bengtsson]

I have updated the regression log and test_changes after resolving the "debug" and "hafs 4 nest" test issues. Could you let me know if the log file looks correct now?

[gspetro-NOAA]

I have updated the regression log and test_changes after resolving the "debug" and "hafs 4 nest" test issues. Could you let me know if the log file looks correct now?

@lisa-bengtsson Thanks! Could you rerun just cpld_debug_gfsv17_gnu? It has a test timed out message, which is usually harmless, but it should probably have an "unable to complete comparison" message instead I'd think (or a pass)? Better to check.

[lisa-bengtsson]

I have updated the regression log and test_changes after resolving the "debug" and "hafs 4 nest" test issues. Could you let me know if the log file looks correct now?

@lisa-bengtsson Thanks! Could you rerun just cpld_debug_gfsv17_gnu? It has a test timed out message, which is usually harmless, but it should probably have an "unable to complete comparison" message instead I'd think (or a pass)? Better to check.

@gspetro-NOAA Ok! I re-ran only the cpld_debug_gfsv17_gnu and now it finished in time and the RT failed as expected: /scratch4/BMC/gsienkf/Lisa.Bengtsson/ufs-weather-model-updraft/tests/logs/RegressionTests_ursa.log

[gspetro-NOAA]

@gspetro-NOAA Ok! I re-ran only the cpld_debug_gfsv17_gnu and now it finished in time and the RT failed as expected: /scratch4/BMC/gsienkf/Lisa.Bengtsson/ufs-weather-model-updraft/tests/logs/RegressionTests_ursa.log

Awesome, thanks! I think we can process this one next. Will let the other CMs know, since I'll be traveling.

[FernandoAndrade-NOAA]

This PR is next for testing, please sync / resolve conflicts, thank you.

[lisa-bengtsson]

Ok - I have synced this PR now, let me know if there is anything else I should do.

[FernandoAndrade-NOAA]

We should be good to go for starting RTs, thank you! @gspetro-NOAA @BrianCurtis-NOAA