Skip to content

Add NOAA UFS UGWP suite of physics parameterizations #1276

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 16 commits into from
May 12, 2025

Conversation

mdtoyNOAA
Copy link
Contributor

@mdtoyNOAA mdtoyNOAA commented Jan 24, 2025

This PR incorporates NOAA's Unified Forecast System (UFS) Unified Gravity Wave Physics (UGWP) suite of physics parameterizations into MPAS: https://ufs.epic.noaa.gov/wp-content/uploads/2021/07/UGWP_in_UFS_July_2021-1.pdf.

The following is a brief overview on using the suite:

  1. In the init_atmosphere core, there is a new static pre-processing option, 'config_native_gwd_gsl_static', that is used to generate a new static file used by the Unified Gravity Wave Physics. The static file is written by the ugwp_oro_data stream, and requires two new datasets in the geographical dataset directory: 'topo_ugwp_30s' and 'topo_ugwp_2.5m'.
  2. In the atmosphere core, there are several new options when config_gwdo_scheme = 'bl_ugwp_gwdo': config_ngw_scheme, config_knob_ugwp_tauamp, and config_ugwp_diags.
  3. If config_ngw_scheme = true, an additional input file, ugwp_limb_tau.nc is needed. This namelist options activates the non-stationary gravity wave drag scheme.
  4. If config_ugwp_diags = true, the diag_ugwp output stream will be written.

NOTE: This physics package is the "NOAA/GSL" orographic gravity wave drag (GWD) suite introduced in WRF Version 4.3 (activated by WRF namelist option "gwd_opt=3"), but with the addition of a non-stationary GWD parameterization that represents gravity wave sources such as deep convection and frontal instability.

Thanks to @mgduda for help and advice on how to incorporate the NOAA/GSL code into the MPAS-Dev repository.

…tmosphere

This commit adds three new source files

 mpas_gsl_oro_data_sm_scale.F
 mpas_gsl_oro_data_lg_scale.F
 mpas_init_atm_gwd_gsl.F

to the init_atmosphere core, along with modifications to the main
core_init_atmosphere Makefile to compile these source files.

At present, none of the code in these files is called from the init_atmosphere
core.
…re core

This commit defines new static fields, a new package, a new namelist option, and
a new output stream for the GSL GWDO scheme. Also included in this commit are
changes to the init_atm_setup_packages() routine to set the new
'gwd_gsl_stage_out' package if and only if the new namelist option
'config_native_gwd_gsl_static' is true.

When 'config_native_gwd_gsl_static' is set in the namelist, the init_atmosphere
core will write the twenty new static fields (var2d, con, oa{1,2,3,4}, and
ol{1,2,3,4}) for large scale (ls) and small scale (ss) to a new 'ugwp_oro_data'
output stream.

As of this commit, code to actually compute the twenty new static fields is not
being called, and so the fields are expected to be zero everywhere.
When the namelist option config_native_gwd_gsl_static is .true., the
init_atm_setup_case() routine now calls thecalc_gsl_oro_data() routine to
compute the twenty GSL GWDO static fields.
This commit adds an entry to the Externals.cfg file to obtain the GSL UGWP code
from a GitHub repository. Also included in this commit are changes to the main
physics Makefile to compile the UGWP code.

At present, none of the UGWP code is called.
Also included in this commit is logic for setting up packages related to the GSL
UGWP scheme in the atm_setup_packages() routine.
…emes

This commit adds a new module, module_bl_ugwp_gwdo, in the
core_atmosphere/physics/physics_wrf directory. This module serves as a wrapper
for the UGWP physics that is compatible with the MPAS-A physics drivers.

Changes to the Makefile in the physics_wrf directory ensure that the
module_bl_ugwp_gwdo.F file is compiled, but at present the code in that module
is not actually called.
With this commit, the UGWP physics can now be used in MPAS-A simulations

* Setting the 'config_gwdo_scheme' namelist option to 'bl_ugwp_gwdo' selects the
  GSL GWDO scheme.

  NOTE: If the GSL GWDO scheme is selected, the 'ugwp_oro_data_in' input stream
        must specify a valid input file with UGWP static fields.

* Setting the 'config_ngw_scheme' namelist option to 'true' will activate the
  non-stationary gravity wave drag scheme.

  NOTE: If the NGW scheme is activated, the 'ugwp_ngw_in' input stream
        must specify a valid input file.

* Setting the 'config_ugwp_diags' namelist option to 'true' will compute UGWP
  diagnostic fields, which will be written by the 'diag_ugwp' output stream.
@mgduda
Copy link
Contributor

mgduda commented Jan 31, 2025

It looks like we may need to add -I./physics/UGWP to the build commands in src/core_atmosphere/Makefile to prevent issues with the nvfortran compiler not finding modules. This seems to be a general issue with the nvfortran compiler -- that search paths for all modules that are used directly and indirectly are required.

--- a/src/core_atmosphere/Makefile
+++ b/src/core_atmosphere/Makefile
@@ -81,7 +81,7 @@ clean:
        $(RM) $@ $*.mod
 ifeq "$(GEN_F90)" "true"
        $(CPP) $(CPPFLAGS) $(PHYSICS) $(CPPINCLUDES) -I./inc $< > $*.f90
-       $(FC) $(FFLAGS) -c $*.f90 $(FCINCLUDES) -I../framework -I../operators -I./physics -I./dynamics -I./diagnostics -I./physics/physics_wrf -I./physics/physics_mmm -I../external/esmf_time_f90
+       $(FC) $(FFLAGS) -c $*.f90 $(FCINCLUDES) -I../framework -I../operators -I./physics -I./dynamics -I./diagnostics -I./physics/physics_wrf -I./physics/physics_mmm -I./physics/UGWP -I../external/esmf_time_f90
 else
-       $(FC) $(CPPFLAGS) $(PHYSICS) $(FFLAGS) -c $*.F $(CPPINCLUDES) $(FCINCLUDES) -I./inc -I../framework -I../operators -I./physics -I./dynamics -I./diagnostics -I./physics/physics_wrf -I./physics/physics_mmm -I../external/esmf_time_f90
+       $(FC) $(CPPFLAGS) $(PHYSICS) $(FFLAGS) -c $*.F $(CPPINCLUDES) $(FCINCLUDES) -I./inc -I../framework -I../operators -I./physics -I./dynamics -I./diagnostics -I./physics/physics_wrf -I./physics/physics_mmm -I./physics/UGWP -I../external/esmf_time_f90
 endif

Modified src/core_atmosphere/Makefile to avoid endless loop if running
'make clean CORE=atmospere' before the UGWP directory has been created.

Added '-I./physics/UGWP' to src/core_atmosphere/physics/Makefile to
prevent issues with the nvfortran compiler not finding modules.  This
seems to be a general issue with the nvfortran compiler -- that search
paths for all modules that are used directly and indirectly are required.
@mdtoyNOAA
Copy link
Contributor Author

@mgduda I added -I./physics/UGWP to the physics/Makefile, did a test compile, and pushed new commit.

@mgduda mgduda self-requested a review February 27, 2025 22:30
Copy link
Contributor

@mgduda mgduda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've just left one request to add pre-processing directives in the Registry.xml file so that we can compile stand-alone MPAS-A as a dynamics-only model. I think just pushing an additional commit to the PR branch with this change should be all that's needed.

@mgduda mgduda self-requested a review February 28, 2025 19:58
Copy link
Contributor

@mgduda mgduda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code changes look good to me. Thanks for all of your work on this, @mdtoyNOAA!

I have one small request: could the PR description be updated to document some of the practical aspects of the changes in this PR? For example, it would be nice to note that:

  1. In the init_atmosphere core, there is a new static pre-processing option, config_native_gwd_gsl_static, that is used to generate a new static file used by the Unified Gravity Wave Physics. The static file is written by the ugwp_oro_data stream, and requires two new datasets in the geographical dataset directory: topo_ugwp_30s and topo_ugwp_2.5m.
  2. In the atmosphere core, there are several new options when config_gwdo_scheme = 'bl_ugwp_gwdo': config_ngw_scheme, config_knob_ugwp_tauamp, and config_ugwp_diags.
  3. If config_ngw_scheme = true, an additional input file, ugwp_limb_tau.nc is needed.
  4. If config_ugwp_diags = true, the diag_ugwp output stream will be written.

I can imagine members of the community asking about the new UGWP option, so having some of this documentation to point to in the PR description (perhaps later to be migrated to more formal user documentation) could be helpful.

@mdtoyNOAA
Copy link
Contributor Author

mdtoyNOAA commented Mar 6, 2025 via email

GSL_UGWP GitHub repo URL updated from 'mdtoyNOAA' to 'NOAA-GSL'
in src/core_atmosphere/Externals.cfg
This commit prevents model termination in subroutine physics_compatibility_check
in src/core_atmosphere/physics/mpas_atmphys_control.F when the 'bl_ugwp_gwdo'
physics scheme is selected and the 'bl_ysu_gwdo' (var2d, con, etc.)  static
variables are all 'zero' in the 'init.nc' file.  Note that these variables are
not used by the 'bl_ugwp_gwdo' orographic gravity wave drag physics.
Changed the name of the UGWP branch from 'main_knob_ugwp_tauamp_namelist'
to 'main_MPAS_2024_12_23' in Externals.cfg.  Note that the contents of
the branch has not changed, and this commit does not change the model
results.
@ldfowler58
Copy link
Contributor

Here are my few suggestions:

  1. I would suggest to change the name of the directory UGWP to, for instance, something like physics_gsl, for at least two reasons: a) Assuming that other GSL parameterizations will be implemented in the near future, it would make sense to me to not have individual GSL parameterizations sitting in their own directories, but instead grouped in one directory containing all the GSL physics as we have pioneered with the physics_mmm, physics_noahmp, and physics_wrf. Otherwise, we are going to have GSL directories scattered all over the physics directory to which I am totally opposed.

I suggest that we do that now with one parameterization to pave the way for and entice other GSL developers to do the same in the future. This will also helps Users who are familiar (or not) with the organization of physics.

@ldfowler58
Copy link
Contributor

  1. In mpas_atmphys_vars.F, can we have the units of all variables for UGWP orographic gravity wave drag be "right" justified instead of "left" justified (and realign the description of the variables. Thanks.

  2. In mpas_atmphys_driver.F, can we realign the variables in the call to gwdo_ugwp (dusfc_ls to errflg) as done between p3d and zi. Thanks.

@mdtoyNOAA
Copy link
Contributor Author

@ldfowler58 Thank you for you suggestions. I will work on incorporating them and re-commit.

@mdtoyNOAA
Copy link
Contributor Author

Here are my few suggestions:

  1. I would suggest to change the name of the directory UGWP to, for instance, something like physics_gsl, for at least two reasons: a) Assuming that other GSL parameterizations will be implemented in the near future, it would make sense to me to not have individual GSL parameterizations sitting in their own directories, but instead grouped in one directory containing all the GSL physics as we have pioneered with the physics_mmm, physics_noahmp, and physics_wrf. Otherwise, we are going to have GSL directories scattered all over the physics directory to which I am totally opposed.

I suggest that we do that now with one parameterization to pave the way for and entice other GSL developers to do the same in the future. This will also helps Users who are familiar (or not) with the organization of physics.

@ldfowler58 and @mgduda I like the idea of having a new directory '/physics/physics_gsl' for depositing GSL physics, however I would like to further suggest that the physics be grouped under their own directories under the 'physics_gsl' directory in order to facilitate submodules. For example, the UGWP will be in the directory 'physics/physics_gsl/UGWP', MYNN will be 'physics/physics_gsl/MYNN', etc.

@ldfowler58
Copy link
Contributor

I do not have any objections about having the GSL physics organized in their own directories under the directory physics_gsl.

@mgduda
Copy link
Contributor

mgduda commented Apr 25, 2025

@ldfowler58 and @mgduda I like the idea of having a new directory '/physics/physics_gsl' for depositing GSL physics, however I would like to further suggest that the physics be grouped under their own directories under the 'physics_gsl' directory in order to facilitate submodules. For example, the UGWP will be in the directory 'physics/physics_gsl/UGWP', MYNN will be 'physics/physics_gsl/MYNN', etc.

I think this sounds good -- having a physics/physics_gsl directory, with further sub-directories for individual packages, e.g., physics/physics_gsl/UGWP.

1)  Updated UGWP submodule tag in core_atmosphere/Externals.cfg
2)  Moved UGWP submodule from 'physics/' to 'physics/physics_gsl' subdirectory
3)  Updated formatting in mpas_atmphys_driver_gwdo.F and mpas_atmphys_vars.F
Changed location of UGWP submodule directory from
'src/core_atmosphere/physics/physics_gsl/UGWP' to
'src/core_atmosphere/physics/physics_noaa/UGWP'
@snyderdad
Copy link

@mdtoyNOAA
A question separate from your code: Is this the same gravity-wave drag that is used in global UFS?
I would have started with the pdf referenced in the PR description, but the link is stale.

@mdtoyNOAA
Copy link
Contributor Author

@mdtoyNOAA A question separate from your code: Is this the same gravity-wave drag that is used in global UFS? I would have started with the pdf referenced in the PR description, but the link is stale.

@snyderdad Thank you for bringing this up. I had linked the wrong URL. It's now fixed, and yes, it is the same GWD that is used in the global UFS.

@mgduda
Copy link
Contributor

mgduda commented May 12, 2025

It looks to me that all review comments have been addressed, so I think we're all set!

@mgduda mgduda merged commit d2c0680 into MPAS-Dev:develop May 12, 2025
jim-p-w added a commit to jim-p-w/MPAS-Model that referenced this pull request May 15, 2025
…ecast

System Unified Gravity Wave Physics suite of physics parameterizations.

git commit d2c0680  (PR MPAS-Dev#1276) introduced a new external dependency for
the physics code.  While the Makefiles were updated accordingly, the cmake
files were not.  This resulted in compile failures when building with cmake:
MPAS-Model-develop/src/core_atmosphere/physics/mpas_atmphys_finalize.F:14:6:
14 |  use cires_ugwpv1_module
|      1
Fatal Error: Cannot open module file 'cires_ugwpv1_module.mod' for reading at
(1): No such file or directory
mgduda added a commit that referenced this pull request May 15, 2025
This merge updates CMake files to reflect the incorporation of NOAA's Unified
Gravity Wave Physics suite.

Commit d2c0680 (PR #1276) introduced a new external dependency for the UGWP
code. While the Makefiles were updated accordingly, the CMakeLists.txt files
were not. This resulted in compile failures when building with cmake:

  MPAS-Model-develop/src/core_atmosphere/physics/mpas_atmphys_finalize.F:14:6:
  14 |  use cires_ugwpv1_module
  |      1
  Fatal Error: Cannot open module file 'cires_ugwpv1_module.mod' for reading at
  (1): No such file or directory
jim-p-w added a commit to jim-p-w/MPAS-Model that referenced this pull request May 15, 2025
added in commit d2c0680  (PR MPAS-Dev#1276).
That PR updated the makefiles, but not the cmake files. That resulted in
compile errors when building with cmake:
src/core_init_atmosphere/mpas_init_atm_cases.F:48:11:

   48 |       use mpas_init_atm_gwd_gsl, only : calc_gsl_oro_data
      |           1
Fatal Error: Cannot open module file 'mpas_init_atm_gwd_gsl.mod' for reading at
(1): No such file or directory
mgduda added a commit that referenced this pull request May 16, 2025
This merge updates the core_init_atmosphere CMakeLists.txt file to include new
files added in commit d2c0680 (PR #1276).  That PR updated the Makefile, but not
the CMakeLists.txt file, resulting in compilation errors when building with
cmake:

  src/core_init_atmosphere/mpas_init_atm_cases.F:48:11:

  48 | use mpas_init_atm_gwd_gsl, only : calc_gsl_oro_data
  | 1
  Fatal Error: Cannot open module file 'mpas_init_atm_gwd_gsl.mod' for reading at
  (1): No such file or directory
mgduda added a commit that referenced this pull request Jun 3, 2025
MPAS Version 8.3.0

This release of MPAS introduces new capabilities and improvements in the
MPAS-Atmosphere model and its supporting software infrastructure. Notable
changes are listed below.

Initialization:

 * Addition of support for 30" BNU soil category dataset. The 30" BNU soil
   category dataset can be selected by setting the new namelist option
   config_soilcat_data to 'BNU' in the &data_sources namelist group. Use of this
   dataset requires a separate static dataset download. (PR #1322)

 * Addition of support for 15" MODIS land use dataset. The 15" MODIS land use
   dataset may be selected by setting the existing namelist option
   config_landuse_data to 'MODIFIED_IGBP_MODIS_NOAH_15s' in the &data_sources
   namelist group. Use of this dataset requires a separate static dataset
   download.  (PR #1322)

 * Introduction of a new namelist option, config_lu_supersample_factor, to
   control the super-sampling of land use data, which may now be on either a 30"
   or a 15" grid, depending on the choice of dataset. The existing namelist
   option config_30s_supersample_factor now controls the super-sampling for 30"
   terrain, soil category, and MODIS FPAR monthly vegetation fraction data only.
   (PR #1322)

 * A change in the horizontal interpolation from a four-point bilinear
   interpolation to a sixteen-point overlapping parabolic interpolation for both
   initial conditions and lateral boundary conditions. (PR #1303)

 * Ability to use ICON soil moisture and soil temperature fields. (PR #1298)

 * Addition of an option to skip processing of Noah-MP-only static fields in the
   init_atmosphere core. Setting the new config_noahmp_static namelist option to
   false in the &data_sources namelist group prevents the Noah-MP static fields
   from being processed when config_static_interp = true in the
   namelist.init_atmosphere file; this also permits existing static files that
   lack the Noah-MP fields 'soilcomp', 'soilcl1', 'soilcl2', 'soilcl3', and
   'soilcl4' to be used by the init_atmosphere_model program. (PR #1239)

 * Memory scaling improvements to the gravity wave drag (GWD) static field
   processing in the init_atmosphere core (when 'config_native_gwd_static =
   true') to reduce memory usage when multiple MPI ranks are used. In many
   cases, these changes eliminate the need to undersubscribe computing
   resources, which was previously required in order to work around lack of
   memory scaling in the GWD static field processing. (PR #1235)

Physics:

 * Update of the RRTMG LW and SW schemes, most notably with the addition of the
   exponential and exponential_random cloud overlap assumptions. The cloud
   overlap assumption and decorrelation length are now available as namelist
   options (config_radt_cld_overlap and config_radt_cld_dcorrlen, respectively).
   (PR #1296 and PR #1297)

 * The incorporation of NOAA's Unified Forecast System (UFS) Unified Gravity
   Wave Physics (UGWP) suite of physics parameterizations. This physics package
   is the "NOAA/GSL" orographic gravity wave drag (GWD) suite introduced in WRF
   Version 4.3 (activated by WRF namelist option 'gwd_opt=3'), but with the
   addition of a non-stationary GWD parameterization that represents gravity
   wave sources such as deep convection and frontal instability. The use of the
   UGWP suite requires additional static field downloads. (PR #1276)

Dynamics:

 * Complete port of all routines in the dynamical core to GPUs using OpenACC
   directives, including routines used by limited-area simulations. Not included
   in this release, though, is the optimization of data movement between the CPU
   and GPU memory, and the profiling and optimization of the computational
   kernels.

 * A change in the zero-gradient LBC for w to a constant value of w=0 in the
   specified zone. For limited-area configurations, the change from a
   zero-gradient boundary condition for the vertical velocity, w, to a setting
   of the vertical velocity to zero in the specified region alleviates spurious
   streamers and instabilities that appeared near the boundaries in regions of
   strong inflow.  (PR #1304)

Infrastructure:

 * Implementation of a new capability to automatically generate package logic
   code, which determines when a package is active. This package logic is
   generated by the registry at build time through the use of a new XML
   attribute, active_when, for <package> elements. (PR #1321)

Other:

 * Addition of a new Python script for setting up MPAS-Atmosphere run
   directories.  (PR #1326)

 * Addition of 3-d 10 cm radar reflectivity (refl10cm) to the 'da_state' stream,
   useful for radar DA and radar obs comparison purposes. (PR #1323)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants