diff --git a/docs/tools/make.rst b/docs/tools/make.rst index c9aa496dc..6c31cff4a 100644 --- a/docs/tools/make.rst +++ b/docs/tools/make.rst @@ -7,9 +7,10 @@ - ``-y, --yamlfile [experiment yaml] (required)`` - ``-p, --platform [platform] (required)`` - ``-t, --target [target] (required)`` - - ``-j, --njobs [number of jobs to run simultaneously]`` + - ``-gj, --gitjobs [number of submodules to clone in parallel]`` - ``-npc, --no-parallel-checkout (for container build)`` - ``-e, --execute`` + - ``force-checkout`` ``makefile`` ------------- @@ -30,7 +31,7 @@ - ``-y, --yamlfile [experiment yaml] (required)`` - ``-p, --platform [platform] (required)`` - ``-t, --target [target] (required)`` - - ``-j, --njobs [number of jobs to run simultaneously]`` + - ``-mj, --makejobs [number of recipes from the Makefile to run in parallel]`` - ``-n, --parallel [number of concurrent module compiles]`` - ``-e, --execute`` @@ -44,6 +45,8 @@ - ``-y, --yamlfile [experiment yaml] (required)`` - ``-p, --platform [platform] (required)`` - ``-t, --target [target] (required)`` + - ``-nft, --no-format-transfer`` + - ``-e, --execute`` ``all`` ------- @@ -54,6 +57,9 @@ - ``-y, --yamlfile [experiment yaml] (required)`` - ``-p, --platform [platform] (required)`` - ``-t, --target [target] (required)`` - - ``-npc, --no-parallel-checkout (for container build)`` - - ``-j, --njobs [number of jobs to run simultaneously]`` - ``-n, --parallel [number of concurrent module compiles]`` + - ``-mj --makejobs [number of recipes from the Makefile to run in parallel]`` + - ``-gj, --gitjobs [number of submodules to clone in parallel]`` + - ``-npc, --no-paralel-checkout`` + - ``-nft, --no-format-transfer`` + - ``-e, --execute`` diff --git a/docs/usage.rst b/docs/usage.rst index 84a25b1fb..f8133772e 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -30,7 +30,7 @@ Post-processing Yamls Build FMS model =============== .. include:: usage/build_fms_model.rst -.. include:: usage/guides/fre_make_guide.rst +.. include:: usage/fre_make_quickstart.rst Run FMS model ============= diff --git a/docs/usage/build_fms_model.rst b/docs/usage/build_fms_model.rst index 74fe7bae4..eb5b12281 100644 --- a/docs/usage/build_fms_model.rst +++ b/docs/usage/build_fms_model.rst @@ -7,6 +7,7 @@ Capabilities ----------------- Fremake Canopy Supports: - multiple target use; ``-t`` flag to define each target (for multiple platform-target combinations) + - multiple platform use; ``-p`` flag to define each platform (for multiple platform-target combinations) - bare-metal build - container creation - parallel checkouts for bare-metal build @@ -14,7 +15,7 @@ Fremake Canopy Supports: - one yaml format - additional library support if needed -**Note: Users will not be able to create containers without access to podman. To get access, submit a helpdesk ticket.** +**Note: Users will not be able to create containers without access to podman (on GAEA). To get access, submit a helpdesk ticket.** Required configuration files: @@ -22,7 +23,7 @@ Required configuration files: - Compile Yaml - Platforms yaml -These yamls are combined and further parsed through the ``fre make`` tools (see the "Guide" section for the step by step process). +These yamls are combined and further parsed through the ``fre make`` tools. The final combined yaml includes the name of the compile experiment, the platform and target passed in the command line subtool, as well as compile and platform yaml information. The platform that was passed corresponds to the one defined in the platforms YAML file. This file details essential configuration info such as setting up the runtime environment, listing what compiler to use, and providing which container applications to use. These configurations vary based on the specific site where the user is building the model executable or container. Additionally the platform and target passed are used to fill in the build directory in which the compile script is created and run. diff --git a/docs/usage/fre_make_quickstart.rst b/docs/usage/fre_make_quickstart.rst new file mode 100644 index 000000000..4c80e1ad5 --- /dev/null +++ b/docs/usage/fre_make_quickstart.rst @@ -0,0 +1,66 @@ +Quickstart +----------------- +The quickstart instructions build the null model by loading the FRE module and using yaml files located in the fre-cli repository. + +To access the null_model configuration, clone the fre-cli repository: + +.. code-block:: + + git clone --recursive https://github.com/NOAA-GFDL/fre-cli.git + + cd fre/make/tests/null_example + +1. All-in-one fre make subtool: + +`all` kicks off the compilation automatically + +.. code-block:: + + # Bare-metal: create and run checkout script, create makefile, create and RUN compile script to generate a model executable + fre make all -y null_model.yaml -p ncrc5.intel23 -t prod --execute + + # Container: create checkout script, makefile, create dockerfile, and create and RUN the container build script to generate a model container + fre make all -y null_model.yaml -p hpcme.2023 -t prod --execute + +2. Bare-metal Build (Single target) + +For the bare-metal build, the parallel checkout feature is the default behavior. + +.. code-block:: + + # Create and run checkout script + fre make checkout-script -y null_model.yaml -p ncrc5.intel23 -t prod --execute + + # Create the Makefile + fre make makefile -y null_model.yaml -p ncrc5.intel23 -t prod + + # Create and run the compile script + fre make compile-script -y null_model.yaml -p ncrc5.intel23 -t prod --execute + +3. Bare-metal Build (Multiple targets): + +.. code-block:: + + # Create and run checkout script + fre make checkout-script -y null_model.yaml -p ncrc5.intel23 -t prod -t debug --execute + + # Create the Makefile + fre make makefile -y null_model.yaml -p ncrc5.intel23 -t prod -t debug + + # Create and run a compile script for each target specified + fre make compile-script -y null_model.yaml -p ncrc5.intel23 -t prod -t debug --execute + +4. Container Build: + +In order for the container to build successfully, the parallel checkout feature is disabled. + +.. code-block:: + + # Create checkout script + fre make checkout-script -y null_model.yaml -p hpcme.2023 -t prod + + # Create the Makefile + fre make makefile -y null_model.yaml -p hpcme.2023 -t prod + + # Create the Dockerfile and container build script + fre make dockerfile -y null_model.yaml -p hpcme.2023 -t prod --execute diff --git a/docs/usage/guides/fre_make_guide.rst b/docs/usage/guides/fre_make_guide.rst deleted file mode 100644 index abae4d272..000000000 --- a/docs/usage/guides/fre_make_guide.rst +++ /dev/null @@ -1,117 +0,0 @@ -Guide ----------- -1. Bare-metal Build: - -.. code-block:: - - # Create checkout script - fre make checkout-script -y [model yaml file] -p [platform] -t [target] - - # Or create and RUN checkout script - fre make checkout-script -y [model yaml file] -p [platform] -t [target] --execute - - # Create Makefile - fre make makefile -y [model yaml file] -p [platform] -t [target] - - # Create the compile script - fre make compile-script -y [model yaml file] -p [platform] -t [target] - - # Or create and RUN the compile script - fre make compile-script -y [model yaml file] -p [platform] -t [target] --execute - -Users can also run all fre make commands in one subtool: - -.. code-block:: - - # Run all of fremake: creates checkout script, makefile, compile script, and model executable - fre make all -y [model yaml file] -p [platform] -t [target] [other options...] --execute - -2. Container Build: - -For the container build, parallel checkouts are not supported. In addition the platform must be a container platform. - -Gaea users will not be able to create containers unless they have requested and been given podman access. - -.. code-block:: - - # Create checkout script - fre make checkout-script -y [model yaml file] -p [CONTAINER PLATFORM] -t [target] - - # Create Makefile - fre make makefile -y [model yaml file] -p [CONTAINER PLATFORM] -t [target] - - # Create a Dockerfile - fre make dockerfile -y [model yaml file] -p [CONTAINER PLATFORM] -t [target] - - # Or create and RUN the Dockerfile - fre make dockerfile -y [model yaml file] -p [CONTAINER PLATFORM] -t [target] --execute - -To run all of fre make subtools: - -.. code-block:: - - # Run all of fremake: create and checkout script, makefile, dockerfile, container - # creation script, and model container - fre make all -y [model yaml file] -p [CONTAINER PLATFORM] -t [target] --execute - -Quickstart ----------- -The quickstart instructions can be used with the null model example located in the fre-cli repository: https://github.com/NOAA-GFDL/fre-cli/tree/main/fre/make/tests/null_example - -1. Bare-metal Build: - -.. code-block:: - - # Create and run checkout script: checkout script will check out source code as - defined in the compile.yaml - fre make checkout-script -y null_model.yaml -p ncrc5.intel23 -t prod --execute - - # Create Makefile - fre make makefile -y null_model.yaml -p ncrc5.intel23 -t prod - - # Create and run the compile script to generate a model executable - fre make compile-script -y null_model.yaml -p ncrc5.intel23 -t prod --execute - -2. Bare-metal Build Multi-target: - -.. code-block:: - - # Create and run checkout script: checkout script will check out source code as - defined in the compile.yaml - fre make checkout-script -y null_model.yaml -p ncrc5.intel23 -t prod -t debug --execute - - # Create Makefile - fre make makefile -y null_model.yaml -p ncrc5.intel23 -t prod -t debug - - # Create and run a compile script for each target specified; generates model executables - fre make compile-script -y null_model.yaml -p ncrc5.intel23 -t prod -t debug --execute - -3. Container Build: - -In order for the container to build successfully, the parallel checkout feature is disabled. - -.. code-block:: - - # Create checkout script - fre make checkout-script -y null_model.yaml -p hpcme.2023 -t prod - - # Create Makefile - fre make makefile -y null_model.yaml -p hpcme.2023 -t prod - - # Create the Dockerfile and container build script: the container build script (createContainer.sh) - uses the Dockerfile to build a model container - fre make dockerfile -y null_model.yaml -p hpcme.2023 -t prod --execute - -4. Run all of fremake: - -`all` kicks off the compilation automatically - -.. code-block:: - - # Bare-metal: create and run checkout script, create makefile, create and RUN compile script to - generate a model executable - fre make all -y null_model.yaml -p ncrc5.intel23 -t prod --execute - - # Container: create checkout script, makefile, create dockerfile, and create and RUN the container - build script to generate a model container - fre make all -y null_model.yaml -p hpcme.2023 -t prod --execute diff --git a/fre/make/README.md b/fre/make/README.md index 82efe85c6..de20c6ae6 100644 --- a/fre/make/README.md +++ b/fre/make/README.md @@ -3,55 +3,72 @@ Through the fre-cli, `fre make` can be used to create and run a checkout script, * Fre make Supports: - multiple targets; use `-t` flag to define each target + - multiple platforms; use `-p` flag to define each platform - bare-metal build - container creation - - parallel checkouts for bare-metal build** - -** **Note: Users will not be able to create containers without access to podman** - -## Guide - -### **Bare-metal Build:** - -Use fre make subtools to checkout code, create a Makefile, and compile code: - -```bash -# Create and run checkout script -fre make checkout-script -y [model yaml file] -p [platform] -t [target] --execute - -# Create Makefile -fre make makefile -y [model yaml file] -p [platform] -t [target] - -# Create and run the compile script -fre make compile-script -y [model yaml file] -p [platform] -t [target] --execute -``` - -Or use `fre make all` to do the job of all 3 subtools in one step: - -```bash -# Run fre make checkout-script, fre make makefile, and fre make compile-script in order -fre make all -y [model yaml file] -p [platform] -t [target] [other options...] -``` - -### **Container Build:** -***To reiterate, users will not be able to create containers unless they have podman access on gaea.*** - -Use fre make subtools to checkout code, create a Makefile, and build a container: - -```bash -# Create and run checkout script -fre make checkout-script -y [model yaml file] -p [CONTAINER PLATFORM] -t [target] --execute - -# Create Makefile -fre make makefile -y [model yaml file] -p [CONTAINER PLATFORM] -t [target] - -# Create and run the Dockerfile -fre make dockerfile -y [model yaml file] -p [CONTAINER PLATFORM] -t [target] --execute -``` - -Use `fre make all` to do the job of all 3 subtools in one step: - -```bash -# Run fre make checkout-script, fre make makefile, and fre make dockerfile in order -fre make all -y [model yaml file] -p [CONTAINER PLATFORM] -t [target] --execute -``` + - parallel checkouts for bare-metal build + +**Note: The container engine used to create the container (such as podman or docker) is specified in the `platforms.yaml` with the `containerBuild` key. Please ensure the container engine is acccesible before running fre make.** + +## Getting Started + +The quickstart instructions [here](https://noaa-gfdl.readthedocs.io/projects/fre-cli/en/latest/usage.html#quickstart), will build the null model using YAML configurations located in the fre-cli repository. These configurations are combined to create a resolved dictionary that will then be parsed for information to: + +1. Create and run a checkout script (using source code for the `FMS`, `ice_param`, and `coupler` components defined in the `compile.yaml`) +2. Create a Makefile +3. Create and run either a compile.sh (for bare-metal builds) or a Dockerfile and createContainer.sh (for container builds) + +## Subtools +- `fre make checkout-script [options]` + - Purpose: Create and run a checkout script. + - Options: + - `-y, --yamlfile [model yaml] (required)` + - `-p, --platform [platform] (required)` + - `-t, --target [target] (required)` + - `-gj, --gitjobs` + - `-npc, --no-parallel-checkout` + - `--execute` + - `--force-checkout` + +- `fre make makefile [options]` + - Purpose: Create a Makefile. + - Options: + - `-y, --yamlfile [model yaml] (required)` + - `-p, --platform [platform] (required)` + - `-t, --target [target] (required)` + +- `fre make compile-script [options]` + - Purpose: Create and run a compile script to generate a model executable. + - Options: + - `-y, --yamlfile [model yaml] (required)` + - `-p, --platform [platform] (required)` + - `-t, --target [target] (required)` + - `-n --nparallel` + - `-mj --makejobs` + - `-e, --execute` + - `-v, --verbose` + +- `fre make dockerfile [options]` + - Purpose: Create and run a Dockerfile to generate a model container. + - Options: + - `-y, --yamlfile [model yaml] (required)` + - `-p, --platform [platform] (required)` + - `-t, --target [target] (required)` + - `-nft, --no-format-transfer` + - `-e, --execute` + +- `fre make all [options]` + - Purpose: + - For a bare-metal build: Create a checkout script, Makefile, and compile script to generate a model executable + - For a container build: Create a checkout script, Makefile, and Dockerfile to generate a model container. + - Options: + - `-y, --yamlfile [model yaml] (required)` + - `-p, --platform [platform] (required)` + - `-t, --target [target] (required)` + - `-n --nparallel` + - `-mj --makejobs` + - `gj, --gitjobs` + - `-npc, --no-parallel-checkout` + - `-nft, --no-format-transfer` + - `-e, --execute` + - `-v, --verbose`