Merge branch 'release-v8.3.0'

MPAS Version 8.3.0

This release of MPAS introduces new capabilities and improvements in the
MPAS-Atmosphere model and its supporting software infrastructure. Notable
changes are listed below.

Initialization:

 * Addition of support for 30" BNU soil category dataset. The 30" BNU soil
   category dataset can be selected by setting the new namelist option
   config_soilcat_data to 'BNU' in the &data_sources namelist group. Use of this
   dataset requires a separate static dataset download. (PR #1322)

 * Addition of support for 15" MODIS land use dataset. The 15" MODIS land use
   dataset may be selected by setting the existing namelist option
   config_landuse_data to 'MODIFIED_IGBP_MODIS_NOAH_15s' in the &data_sources
   namelist group. Use of this dataset requires a separate static dataset
   download.  (PR #1322)

 * Introduction of a new namelist option, config_lu_supersample_factor, to
   control the super-sampling of land use data, which may now be on either a 30"
   or a 15" grid, depending on the choice of dataset. The existing namelist
   option config_30s_supersample_factor now controls the super-sampling for 30"
   terrain, soil category, and MODIS FPAR monthly vegetation fraction data only.
   (PR #1322)

 * A change in the horizontal interpolation from a four-point bilinear
   interpolation to a sixteen-point overlapping parabolic interpolation for both
   initial conditions and lateral boundary conditions. (PR #1303)

 * Ability to use ICON soil moisture and soil temperature fields. (PR #1298)

 * Addition of an option to skip processing of Noah-MP-only static fields in the
   init_atmosphere core. Setting the new config_noahmp_static namelist option to
   false in the &data_sources namelist group prevents the Noah-MP static fields
   from being processed when config_static_interp = true in the
   namelist.init_atmosphere file; this also permits existing static files that
   lack the Noah-MP fields 'soilcomp', 'soilcl1', 'soilcl2', 'soilcl3', and
   'soilcl4' to be used by the init_atmosphere_model program. (PR #1239)

 * Memory scaling improvements to the gravity wave drag (GWD) static field
   processing in the init_atmosphere core (when 'config_native_gwd_static =
   true') to reduce memory usage when multiple MPI ranks are used. In many
   cases, these changes eliminate the need to undersubscribe computing
   resources, which was previously required in order to work around lack of
   memory scaling in the GWD static field processing. (PR #1235)

Physics:

 * Update of the RRTMG LW and SW schemes, most notably with the addition of the
   exponential and exponential_random cloud overlap assumptions. The cloud
   overlap assumption and decorrelation length are now available as namelist
   options (config_radt_cld_overlap and config_radt_cld_dcorrlen, respectively).
   (PR #1296 and PR #1297)

 * The incorporation of NOAA's Unified Forecast System (UFS) Unified Gravity
   Wave Physics (UGWP) suite of physics parameterizations. This physics package
   is the "NOAA/GSL" orographic gravity wave drag (GWD) suite introduced in WRF
   Version 4.3 (activated by WRF namelist option 'gwd_opt=3'), but with the
   addition of a non-stationary GWD parameterization that represents gravity
   wave sources such as deep convection and frontal instability. The use of the
   UGWP suite requires additional static field downloads. (PR #1276)

Dynamics:

 * Complete port of all routines in the dynamical core to GPUs using OpenACC
   directives, including routines used by limited-area simulations. Not included
   in this release, though, is the optimization of data movement between the CPU
   and GPU memory, and the profiling and optimization of the computational
   kernels.

 * A change in the zero-gradient LBC for w to a constant value of w=0 in the
   specified zone. For limited-area configurations, the change from a
   zero-gradient boundary condition for the vertical velocity, w, to a setting
   of the vertical velocity to zero in the specified region alleviates spurious
   streamers and instabilities that appeared near the boundaries in regions of
   strong inflow.  (PR #1304)

Infrastructure:

 * Implementation of a new capability to automatically generate package logic
   code, which determines when a package is active. This package logic is
   generated by the registry at build time through the use of a new XML
   attribute, active_when, for <package> elements. (PR #1321)

Other:

 * Addition of a new Python script for setting up MPAS-Atmosphere run
   directories.  (PR #1326)

 * Addition of 3-d 10 cm radar reflectivity (refl10cm) to the 'da_state' stream,
   useful for radar DA and radar obs comparison purposes. (PR #1323)

Author: mgduda

CMakeLists.txt

@@ -21,12 +21,14 @@ option(MPAS_DOUBLE_PRECISION "Use double precision 64-bit Floating point." TRUE)
 option(MPAS_PROFILE "Enable GPTL profiling" OFF)
 option(MPAS_OPENMP "Enable OpenMP" OFF)
 option(BUILD_SHARED_LIBS "Build shared libraries" ON)
+option(MPAS_USE_PIO "Build with PIO I/O library" OFF)
 
 message(STATUS "[OPTION] MPAS_CORES: ${MPAS_CORES}")
 message(STATUS "[OPTION] MPAS_DOUBLE_PRECISION: ${MPAS_DOUBLE_PRECISION}")
 message(STATUS "[OPTION] MPAS_PROFILE: ${MPAS_PROFILE}")
 message(STATUS "[OPTION] MPAS_OPENMP: ${MPAS_OPENMP}")
 message(STATUS "[OPTION] BUILD_SHARED_LIBS: ${BUILD_SHARED_LIBS}")
+message(STATUS "[OPTION] MPAS_USE_PIO: ${MPAS_USE_PIO}")
 
 # Build product output locations
 set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/bin)
@@ -56,10 +58,20 @@ endif()
 
 ### Dependencies
 find_package(OpenMP COMPONENTS Fortran)
+
+# use mpifort options with nvfortran
+if(CMAKE_Fortran_COMPILER_ID MATCHES NVHPC)
+    find_program(CMAKE_Fortran_COMPILER_MPI NAMES mpifort REQUIRED)
+    message(VERBOSE "CMAKE_Fortran_COMPILER_MPI: ${CMAKE_Fortran_COMPILER_MPI}")
+    set(MPI_Fortran_COMPILER mpifort)
+endif()
 find_package(MPI REQUIRED COMPONENTS Fortran)
-find_package(NetCDF REQUIRED COMPONENTS Fortran C)
+
 find_package(PnetCDF REQUIRED COMPONENTS Fortran)
-find_package(PIO REQUIRED COMPONENTS Fortran C)
+if(MPAS_USE_PIO)
+    find_package(PIO REQUIRED COMPONENTS Fortran C)
+    find_package(NetCDF REQUIRED COMPONENTS Fortran C)
+endif()
 if(MPAS_PROFILE)
     find_package(GPTL REQUIRED)
 endif()
@@ -70,6 +82,8 @@ if(CMAKE_C_COMPILER_ID MATCHES GNU)
     set(CPP_EXTRA_FLAGS -traditional)
 elseif(CMAKE_C_COMPILER_ID MATCHES "(Apple)?Clang" )
     find_program(CPP_EXECUTABLE NAMES cpp REQUIRED)
+elseif(CMAKE_C_COMPILER_ID MATCHES "NVHPC" )
+    find_program(CPP_EXECUTABLE NAMES cpp REQUIRED)
 else()
     message(STATUS "Unknown compiler: ${CMAKE_C_COMPILER_ID}")
     set(CPP_EXECUTABLE ${CMAKE_C_COMPILER})
@@ -88,6 +102,9 @@ set(MPAS_SUBDRIVER_SRC  ${CMAKE_CURRENT_SOURCE_DIR}/src/driver/mpas_subdriver.F)
 
 ## Create targets
 add_subdirectory(src/external/ezxml) # Target: MPAS::external::ezxml
+if(NOT MPAS_USE_PIO)
+    add_subdirectory(src/external/SMIOL) # Target: MPAS::external::smiol
+endif()
 if(ESMF_FOUND)
   message(STATUS "Configure MPAS for external ESMF")
   add_definitions(-DMPAS_EXTERNAL_ESMF_LIB -DMPAS_NO_ESMF_INIT)

Makefile

@@ -667,10 +667,10 @@ intel:   # BUILDTARGET Intel oneAPI Fortran, C, and C++ compiler suite
 	"CFLAGS_OPT = -O3" \
 	"CXXFLAGS_OPT = -O3" \
 	"LDFLAGS_OPT = -O3" \
-	"FFLAGS_DEBUG = -g -convert big_endian -free -check all -fpe0 -traceback" \
+	"FFLAGS_DEBUG = -g -convert big_endian -free -check bounds,pointers,arg_temp_created,format,shape,contiguous -fpe0 -traceback" \
 	"CFLAGS_DEBUG = -g -traceback" \
 	"CXXFLAGS_DEBUG = -g -traceback" \
-	"LDFLAGS_DEBUG = -g -check all -fpe0 -traceback" \
+	"LDFLAGS_DEBUG = -g -traceback" \
 	"FFLAGS_OMP = -qopenmp" \
 	"CFLAGS_OMP = -qopenmp" \
 	"PICFLAG = -fpic" \
@@ -1266,7 +1266,7 @@ ifeq "$(OPENACC)" "true"
 endif # OPENACC eq true
 
 
-pio_test: openmp_test openacc_test
+pio_test: openmp_test openacc_test pnetcdf_test
 	@#
 	@# PIO_VERS will be set to:
 	@#  0 if no working PIO library was detected (and .piotest.log will contain error messages)
@@ -1374,11 +1374,52 @@ mpi_f08_test:
 	$(if $(findstring 1,$(MPAS_MPI_F08)), $(eval MPI_F08_MESSAGE = "Using the mpi_f08 module."), )
 	$(if $(findstring 1,$(MPAS_MPI_F08)), $(info mpi_f08 module detected.))
 
+
+pnetcdf_test:
+	@#
+	@# Create test C programs that look for PNetCDF header file and some symbols in it
+	@#
+ifneq "$(PNETCDF)" ""
+	@echo "Checking for a working PnetCDF library..."
+	@printf "#include \"pnetcdf.h\"\n\
+			&#include \"mpi.h\"\n\
+			&int main(){\n\
+			&    int err, ncid;\n\
+			&    err = ncmpi_create(MPI_COMM_WORLD, \"foo.nc\", NC_NOCLOBBER, MPI_INFO_NULL, &ncid);\n\
+			&    return 0;\n\
+			&}\n" | sed 's/&/ /' > pnetcdf.c
+	@( $(CC) pnetcdf.c $(CPPINCLUDES) $(CFLAGS) $(LDFLAGS) -L$(PNETCDF)/$(PNETCDFLIBLOC) -lpnetcdf  -o pnetcdf.out > pnetcdf.log 2>&1; \
+	   if [ $$? -eq 0 ] ; then \
+	       echo "$(CC) can compile test PnetCDF C program."; \
+	   else \
+	       echo "*********************************************************"; \
+	       echo "ERROR: Test PnetCDF C program could not be compiled by $(CC)."; \
+	       echo "Please ensure you have a working PnetCDF library installed."; \
+	       echo ""; \
+	       echo "The following compilation command failed with errors:" ; \
+	       echo "$(CC) pnetcdf.c $(CPPINCLUDES) $(CFLAGS) $(LDFLAGS) -L$(PNETCDF)/$(PNETCDFLIBLOC) -lpnetcdf -o pnetcdf.out"; \
+	       echo ""; \
+	       echo "Test program pnetcdf.c and output pnetcdf.log have been left"; \
+	       echo "in the top-level MPAS directory for further debugging"; \
+	       echo "*********************************************************"; \
+	       rm -f pnetcdf.out; exit 1; \
+	   fi )
+
+	@rm -f pnetcdf.c pnetcdf.out pnetcdf.log
+else
+	@echo "*********************************************************"; \
+	 echo "ERROR: The PNETCDF environment variable isn't set."; \
+	 echo "Please set this variable to where PnetCDF is installed."; \
+	 echo "*********************************************************"; \
+	 exit 1
+endif
+
+
 ifneq "$(PIO)" ""
-MAIN_DEPS = rebuild_check openmp_test openacc_test pio_test mpi_f08_test
+MAIN_DEPS = rebuild_check openmp_test openacc_test pnetcdf_test pio_test mpi_f08_test
 override CPPFLAGS += "-DMPAS_PIO_SUPPORT"
 else
-MAIN_DEPS = rebuild_check openmp_test openacc_test mpi_f08_test
+MAIN_DEPS = rebuild_check openmp_test openacc_test pnetcdf_test mpi_f08_test
 IO_MESSAGE = "Using the SMIOL library."
 override CPPFLAGS += "-DMPAS_SMIOL_SUPPORT"
 endif

README.md

@@ -1,4 +1,4 @@
-MPAS-v8.2.3
+MPAS-v8.3.0
 ====
 
 The Model for Prediction Across Scales (MPAS) is a collaborative project for

cmake/Functions/MPAS_Functions.cmake

@@ -80,10 +80,16 @@ function(mpas_fortran_target target)
 
     # Global Fortran configuration
     set_target_properties(${target} PROPERTIES Fortran_FORMAT FREE)
-    set(MPAS_FORTRAN_TARGET_COMPILE_DEFINITIONS
-        _MPI=1
-        USE_PIO2=1
-    )
+    if(MPAS_USE_PIO)
+        set(MPAS_FORTRAN_TARGET_COMPILE_DEFINITIONS
+            USE_PIO2=1
+        )
+    else()
+        set(MPAS_FORTRAN_TARGET_COMPILE_DEFINITIONS
+            MPAS_SMIOL_SUPPORT=1
+        )
+    endif()
+    list(APPEND MPAS_FORTRAN_TARGET_COMPILE_DEFINITIONS _MPI=1)
     # Enable OpenMP support
     if(MPAS_OPENMP)
         target_link_libraries(${target} PUBLIC OpenMP::OpenMP_Fortran)
@@ -123,6 +129,17 @@ function(mpas_fortran_target target)
         else()
             list(APPEND MPAS_FORTRAN_TARGET_COMPILE_DEFINITIONS SINGLE_PRECISION)
         endif()
+    elseif(CMAKE_Fortran_COMPILER_ID MATCHES NVHPC)
+
+        list(APPEND MPAS_FORTRAN_TARGET_COMPILE_DEFINITIONS
+            $<$<COMPILE_LANGUAGE:Fortran>:-DCPRPGI -DMPAS_NAMELIST_SUFFIX=atmosphere -DMPAS_EXE_NAME=atmosphere_model>
+            $<$<COMPILE_LANGUAGE:Fortran>:-DMPAS_OPENACC -DSINGLE_PRECISION -DMPAS_BUILD_TARGET=nvhpc>
+        )
+        list(APPEND MPAS_FORTRAN_TARGET_COMPILE_OPTIONS_PRIVATE
+            $<$<COMPILE_LANGUAGE:Fortran>: -Mnofma -acc -gpu=math_uniform,cc70,cc80 -Minfo=accel -byteswapio>
+        )
+        message(VERBOSE "${target} options: ${MPAS_FORTRAN_TARGET_COMPILE_OPTIONS_PRIVATE}")
+        message(VERBOSE "${target} defines: ${MPAS_FORTRAN_TARGET_COMPILE_DEFINITIONS}")
     endif()
     target_compile_definitions(${target} PRIVATE ${MPAS_FORTRAN_TARGET_COMPILE_DEFINITIONS})
     target_compile_options(${target} PRIVATE ${MPAS_FORTRAN_TARGET_COMPILE_OPTIONS_PRIVATE})
@@ -244,4 +261,4 @@ function(set_MPAS_DEBUG_flag target)
     if(CMAKE_BUILD_TYPE MATCHES Debug)
         target_compile_definitions(${target} PRIVATE MPAS_DEBUG)
     endif()
-endfunction()
+endfunction()

src/core_atmosphere/CMakeLists.txt

@@ -20,6 +20,7 @@ set(ATMOSPHERE_CORE_PHYSICS_SOURCES
     mpas_atmphys_o3climatology.F
     mpas_atmphys_rrtmg_lwinit.F
     mpas_atmphys_rrtmg_swinit.F
+    mpas_atmphys_sfc_diagnostics.F
     mpas_atmphys_update.F
     mpas_atmphys_update_surface.F
     mpas_atmphys_utilities.F
@@ -51,6 +52,7 @@ set(ATMOSPHERE_CORE_PHYSICS_WRF_SOURCES
     cu_ntiedtke_pre.F
     module_bep_bem_helper.F
     module_bl_gwdo.F
+    module_bl_ugwp_gwdo.F
     module_bl_ysu.F
     module_cam_error_function.F
     module_cam_shr_kind_mod.F
@@ -128,9 +130,36 @@ set(ATMOSPHERE_CORE_PHYSICS_MMM_SOURCES
 
 list(TRANSFORM ATMOSPHERE_CORE_PHYSICS_MMM_SOURCES PREPEND physics/physics_mmm/)
 
+set(ATMOSPHERE_CORE_PHYSICS_NOAA_DIR ${CMAKE_CURRENT_SOURCE_DIR}/physics/physics_noaa/UGWP)
+
+if(NOT EXISTS ${ATMOSPHERE_CORE_PHYSICS_NOAA_DIR})
+    set(PHYSICS_NOAA_REPO_URL "https://github.com/NOAA-GSL/UGWP.git")
+    execute_process(COMMAND git clone ${PHYSICS_NOAA_REPO_URL} ${ATMOSPHERE_CORE_PHYSICS_NOAA_DIR}
+                    RESULT_VARIABLE GIT_CLONE_RESULT
+                    OUTPUT_VARIABLE GIT_CLONE_OUTPUT
+                    ERROR_VARIABLE GIT_CLONE_ERROR)
+    if(NOT GIT_CLONE_RESULT EQUAL 0)
+        message(FATAL_ERROR "Git clone failed with error: ${GIT_CLONE_ERROR}")
+    endif()
+
+else()
+    message(STATUS "Directory ${DIR_TO_CHECK} already exists, skipping clone")
+endif()
+
+set(ATMOSPHERE_CORE_PHYSICS_NOAA_SOURCES
+    bl_ugwp.F
+    bl_ugwpv1_ngw.F
+    cires_tauamf_data.F
+    cires_ugwpv1_initialize.F
+    cires_ugwpv1_module.F
+    cires_ugwpv1_solv2.F
+    cires_ugwpv1_triggers.F
+)
+
+list(TRANSFORM ATMOSPHERE_CORE_PHYSICS_NOAA_SOURCES PREPEND physics/physics_noaa/UGWP/)
+
 set(ATMOSPHERE_CORE_PHYSICS_NOAMP_UTILITY_SOURCES
     CheckNanMod.F90
-    ErrorHandleMod.F90
     Machine.F90
 )
 list(TRANSFORM ATMOSPHERE_CORE_PHYSICS_NOAMP_UTILITY_SOURCES PREPEND physics/physics_noahmp/utility/)
@@ -344,6 +373,7 @@ add_library(core_atmosphere ${ATMOSPHERE_CORE_SOURCES}
             ${ATMOSPHERE_CORE_PHYSICS_SOURCES}
             ${ATMOSPHERE_CORE_PHYSICS_MMM_SOURCES}
             ${ATMOSPHERE_CORE_PHYSICS_WRF_SOURCES}
+            ${ATMOSPHERE_CORE_PHYSICS_NOAA_SOURCES}
             ${ATMOSPHERE_CORE_DIAGNOSTIC_SOURCES}
             ${ATMOSPHERE_CORE_DYNAMICS_SOURCES})
 

src/core_atmosphere/Externals.cfg

@@ -3,7 +3,13 @@ local_path = ./physics_mmm
 protocol = git
 repo_url = https://github.com/NCAR/MMM-physics.git
 tag = 20240626-MPASv8.2
+required = True
 
+[GSL_UGWP]
+local_path = ./physics_noaa/UGWP
+protocol = git
+repo_url = https://github.com/NOAA-GSL/UGWP.git
+tag = MPAS_20241223
 required = True
 
 [externals_description]

src/core_atmosphere/Makefile

@@ -4,8 +4,11 @@
 # To build a dycore-only MPAS-Atmosphere model, comment-out or delete
 # the definition of PHYSICS, below
 #
-PHYSICS=-DDO_PHYSICS
-
+# If MPAS_CAM_DYCORE is found in CPPFLAGS, PHYSICS will become undefined automatically
+#
+ifeq ($(findstring MPAS_CAM_DYCORE,$(CPPFLAGS)),)
+    PHYSICS = -DDO_PHYSICS
+endif
 
 ifdef PHYSICS
     PHYSCORE = physcore
@@ -81,7 +84,7 @@ clean:
 	$(RM) $@ $*.mod
 ifeq "$(GEN_F90)" "true"
 	$(CPP) $(CPPFLAGS) $(PHYSICS) $(CPPINCLUDES) -I./inc $< > $*.f90
-	$(FC) $(FFLAGS) -c $*.f90 $(FCINCLUDES) -I../framework -I../operators -I./physics -I./dynamics -I./diagnostics -I./physics/physics_wrf -I./physics/physics_mmm -I../external/esmf_time_f90
+	$(FC) $(FFLAGS) -c $*.f90 $(FCINCLUDES) -I../framework -I../operators -I./physics -I./dynamics -I./diagnostics -I./physics/physics_wrf -I./physics/physics_mmm -I./physics/physics_noaa/UGWP -I../external/esmf_time_f90
 else
-	$(FC) $(CPPFLAGS) $(PHYSICS) $(FFLAGS) -c $*.F $(CPPINCLUDES) $(FCINCLUDES) -I./inc -I../framework -I../operators -I./physics -I./dynamics -I./diagnostics -I./physics/physics_wrf -I./physics/physics_mmm -I../external/esmf_time_f90
+	$(FC) $(CPPFLAGS) $(PHYSICS) $(FFLAGS) -c $*.F $(CPPINCLUDES) $(FCINCLUDES) -I./inc -I../framework -I../operators -I./physics -I./dynamics -I./diagnostics -I./physics/physics_wrf -I./physics/physics_mmm -I./physics/physics_noaa/UGWP -I../external/esmf_time_f90
 endif