YAKL and SAMXX CIME and CMake Integration Notes

The E3SM CMake files are located in E3SM/components and E3SM/components/cmake.

Adding the USE_CUDA and CUDA_FLAGS CMake variables

The USE_CUDA variable needs to be visible at the highest level CMakeLists.txt, so it has to be specified through CIME using the Macros.cmake file in E3SM/cime_config/machines/config_compilers.xml. Similarly, the CUDA_FLAGS variable is compiler dependent and therefore will have to come from the same source through the same route into CMake.

To make these variables available to add to that file, however, they had to first be added to the XML schema in cime/config/xml_schemas/config_compilers_v2.xsd. It was added in the <xs:group name="compilerVars"> group as a "choice" as follows:

      <xs:element name="CUDA_FLAGS" type="flagsVar"/>
      <xs:element name="USE_CUDA" type="upperBoolean"/>

Then they are added to appropriate machine, compiler configurations in config_compilers.xml. An example is below for Summit with GNU + CUDA:

  <CUDA_FLAGS>
    <append> -O3 -arch sm_70 --use_fast_math </append>
  </CUDA_FLAGS>
  <USE_CUDA>TRUE</USE_CUDA>

Now, when CMake starts its top-level CMakeLists.txt for E3SM, it sees these things set in the local Macros.cmake. This still isn't quite enough, though because the main CMakeLists.txt only looks at the Macros.cmake file in a local setting inside a CMake function called set_compilers_e3sm. To make it visible to the overall project, you have to set it in parent scope inside that scope as follows:

  # USE_CUDA is set through Macros.cmake / config_compilers.xml
  # If it exists, then set parent's scope to true; otherwise to false
  if (USE_CUDA)
    set(USE_CUDA TRUE PARENT_SCOPE)
  else()
    set(USE_CUDA FALSE PARENT_SCOPE)
  endif()

Finally, we use this at the top level to turn on the CUDA language. CMake doesn't like doing this at a lower scope, so it needs to be done at the beginning of the project declaration:

project(E3SM C CXX Fortran)

if(USE_CUDA)
  enable_language(CUDA)
endif()

Creating USE_SAMXX and USE_YAKL CMake variables

Go to E3SM/components/cmake/common_setup.cmake, and add new variables there. Use existing code to get the feel for where to test for adding a given variable. CAM often hooks into CIME through the ${CAM_CONFIG_OPTS} variable that comes from the CAM configure command. USE_YAKL and USE_SAMXX are created with the code below in cmake/build_model.cmake

# Look for -crm samxx in the CAM_CONFIG_OPTS CIME variable
# If it's found, then enable USE_SAMXX
string(FIND "${CAM_CONFIG_OPTS}" "-crm samxx" HAS_SAMXX)
if (NOT HAS_SAMXX EQUAL -1)
  # The following is for the SAMXX code:
  set(USE_SAMXX TRUE)
endif()

# If samxx is being used, then YAKL must be used as well
set(USE_YAKL ${USE_SAMXX})

# If YAKL is being used, then we need to enable USE_CXX
if (${USE_YAKL})
  set(USE_CXX TRUE)
endif()

We then add the YAKL and SAMXX builds into the cmake/build_model.cmake file inside if (COMP_NAME STREQUAL "eam") with the following code:

    # If YAKL is needed, then set YAKL CMake vars
    if (USE_YAKL)
      # YAKL_ARCH can be CUDA, HIP, SYCL, OPENMP45, or empty
      # USE_CUDA is set through Macros.cmake / config_compilers.xml
      if (USE_CUDA)
        set(YAKL_ARCH "CUDA")
        # CUDA_FLAGS is set through Macros.cmake / config_compilers.xml
        set(YAKL_CUDA_FLAGS "${CPPDEFS} ${CUDA_FLAGS}")
      else()
        # For CPU C++ compilers duplicate flags are fine, the last ones win typically
        set(YAKL_CXX_FLAGS "${CPPDEFS} ${CXXFLAGS}")
        set(YAKL_ARCH "")
      endif()
      message(STATUS "Building YAKL")
      # Build YAKL as a static library
      # YAKL_HOME is YAKL's source directlry
      set(YAKL_HOME ${CMAKE_CURRENT_SOURCE_DIR}/../../../externals/YAKL)
      # YAKL_BIN is where we're placing the YAKL library
      set(YAKL_BIN  ${CMAKE_CURRENT_BINARY_DIR}/yakl)
      # Build the YAKL static library
      add_subdirectory(${YAKL_HOME} ${YAKL_BIN})
      # Add the YAKL bin directory, mainly due to a Fortran module if it's needed
      include_directories(${YAKL_BIN})
    endif()

    # if samxx is needed, build samxx as a static library
    if (USE_SAMXX)
      message(STATUS "Building SAMXX")
      # SAMXX_HOME is where the samxx source code lives
      set(SAMXX_HOME ${CMAKE_CURRENT_SOURCE_DIR}/../../eam/src/physics/crm/samxx)
      # SAMXX_BIN is where the samxx library will live
      set(SAMXX_BIN  ${CMAKE_CURRENT_BINARY_DIR}/samxx)
      # Build the static samxx library
      add_subdirectory(${SAMXX_HOME} ${SAMXX_BIN})
      # Add samxx F90 files to the main E3SM build
      set(SOURCES ${SOURCES} cmake/atm/../../eam/src/physics/crm/samxx/cpp_interface_mod.F90
                             cmake/atm/../../eam/src/physics/crm/samxx/params.F90
                             cmake/atm/../../eam/src/physics/crm/crm_ecpp_output_module.F90 )
    endif()

From here, each component (YAKL and SAMXX) can fend for itself with its own CMakeLists.txt, which will build a library. Finally, we link the library into the eam component's library with the following code near the end of cmake/build_model.cmake:

    if (COMP_NAME STREQUAL "eam")
      if (USE_YAKL)
        target_link_libraries(${TARGET_NAME} PRIVATE yakl)
      endif()
      if (USE_SAMXX)
        target_link_libraries(${TARGET_NAME} PRIVATE samxx)
      endif()
    endif()