Merge pull request #341 from GoogleCloudPlatform/develop

Version 0.7.3

Author: heyealex

.pre-commit-config.yaml

@@ -1,4 +1,4 @@
-# Copyright 2021 Google LLC
+# Copyright 2022 Google LLC
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

.tfdocs-markdown.yaml

@@ -1,4 +1,4 @@
-# Copyright 2021 Google LLC
+# Copyright 2022 Google LLC
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.

.tflint.hcl

@@ -1,4 +1,4 @@
-// Copyright 2021 Google LLC
+// Copyright 2022 Google LLC
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.

Makefile

@@ -11,8 +11,8 @@ MIN_GOLANG_VERSION=1.16 # for building ghpc
         check-tflint check-pre-commit
 
 ENG = ./cmd/... ./pkg/...
-TERRAFORM_FOLDERS=$(shell find ./modules ./community/modules ./tools -type f -name "*.tf" -not -path '*/\.*' -printf '%h\n' | sort -u)
-PACKER_FOLDERS=$(shell find ./modules ./community/modules ./tools -type f -name "*.pkr.hcl" -not -path '*/\.*' -printf '%h\n' | sort -u)
+TERRAFORM_FOLDERS=$(shell find ./modules ./community/modules ./tools -type f -name "*.tf" -not -path '*/\.*' -exec dirname "{}" \; | sort -u)
+PACKER_FOLDERS=$(shell find ./modules ./community/modules ./tools -type f -name "*.pkr.hcl" -not -path '*/\.*' -exec dirname "{}" \; | sort -u)
 
 # RULES MEANT TO BE USED DIRECTLY
 
@@ -23,12 +23,12 @@ ghpc: warn-go-version warn-terraform-version warn-packer-version $(shell find ./
 install-user:
 	$(info ******** installing ghpc in ~/bin *********************)
 	mkdir -p ~/bin
-	install -t ~/bin ./ghpc
+	install ./ghpc ~/bin
 
 ifeq ($(shell id -u), 0)
 install:
 	$(info ***** installing ghpc in /usr/local/bin ***************)
-	install -t /usr/local/bin ./ghpc
+	install ./ghpc /usr/local/bin
 
 else
 install: install-user

README.md

@@ -6,7 +6,7 @@ HPC Toolkit is an open-source software offered by Google Cloud which makes it
 easy for customers to deploy HPC environments on Google Cloud.
 
 HPC Toolkit allows customers to deploy turnkey HPC environments (compute,
-networking, storage, etc) following Google Cloud best-practices, in a repeatable
+networking, storage, etc.) following Google Cloud best-practices, in a repeatable
 manner. The HPC Toolkit is designed to be highly customizable and extensible,
 and intends to address the HPC deployment needs of a broad range of customers.
 
@@ -140,14 +140,14 @@ packer build .
 
 ## HPC Toolkit Components
 
-The HPC Toolkit has been designed to simplify the process of deploying a
-familiar HPC cluster on Google Cloud. The block diagram below describes the
-individual components of the HPC toolkit.
+The HPC Toolkit has been designed to simplify the process of deploying an HPC
+cluster on Google Cloud. The block diagram below describes the individual
+components of the HPC toolkit.
 
 ```mermaid
 graph LR
     subgraph HPC Environment Configuration
-    A(1. GCP-provided Blueprint Examples) --> B(2. HPC Blueprint)
+    A(1. Provided Blueprint Examples) --> B(2. HPC Blueprint)
     end
     B --> D
     subgraph Creating an HPC Deployment
@@ -159,19 +159,19 @@ graph LR
     end
 ```
 
-1. **GCP-provided Blueprint Examples** – A set of vetted reference blueprints
-   can be found in the examples directory. These can be used to create a
-   predefined deployment for a cluster or as a starting point for creating a
-   custom deployment.
+1. **Provided Blueprint Examples** – A set of vetted reference blueprints can be
+   found in the ./examples and ./community/examples directories. These can be
+   used to create a predefined deployment for a cluster or as a starting point
+   for creating a custom deployment.
 2. **HPC Blueprint** – The primary interface to the HPC Toolkit is an HPC
    Blueprint file. This is a YAML file that defines which modules to use and how
    to customize them.
-3. **gHPC Engine** – The gHPC engine converts the blueprint file into a
-   self-contained deployment directory.
-4. **HPC Modules** – The building blocks of a deployment directory are the
+3. **HPC Modules** – The building blocks of a deployment directory are the
    modules. Modules can be found in the ./modules and community/modules
    directories. They are composed of terraform, packer and/or script files that
    meet the expectations of the gHPC engine.
+4. **gHPC Engine** – The gHPC engine converts the blueprint file into a
+   self-contained deployment directory.
 5. **Deployment Directory** – A self-contained directory that can be used to
    deploy a cluster onto Google Cloud. This is the output of the gHPC engine.
 6. **HPC environment on GCP** – After deployment, an HPC environment will be
@@ -239,15 +239,14 @@ to the Google Cloud Console.
 Many of the above examples are easily executed within a Cloud Shell environment.
 Be aware that Cloud Shell has [several limitations][cloud-shell-limitations],
 in particular an inactivity timeout that will close running shells after 20
-minutes. Please consider it only for small blueprints that are quickly
-deployed.
+minutes. Please consider it only for blueprints that are quickly deployed.
 
 ## Blueprint Warnings and Errors
 
 By default, each blueprint is configured with a number of "validator" functions
-which perform basic tests of your global variables. If `project_id`, `region`,
-and `zone` are defined as global variables, then the following validators are
-enabled:
+which perform basic tests of your deployment variables. If `project_id`,
+`region`, and `zone` are defined as deployment variables, then the following
+validators are enabled:
 
 ```yaml
 validators:
@@ -344,7 +343,7 @@ To view the Cloud Billing reports for your Cloud Billing account:
    [`Billing`](https://console.cloud.google.com/billing/overview).
 2. At the prompt, choose the Cloud Billing account for which you'd like to view
    reports. The Billing Overview page opens for the selected billing account.
-3. In the Billing navigation menu, select Reports.
+3. In the Billing navigation menu, select `Reports`.
 
 In the right side, expand the Filters view and then filter by label, specifying the key `ghpc_deployment` (or `ghpc_blueprint`) and the desired value.
 
@@ -468,7 +467,7 @@ can be found in the [Slurm on Google Cloud User Guide][slurm-on-gcp-ug],
 specifically the section titled "Create Service Accounts".
 
 After creating the service account, it can be set via the
-"compute_node_service_account" and "controller_service_account" settings on the
+`compute_node_service_account` and `controller_service_account` settings on the
 [slurm-on-gcp controller module][slurm-on-gcp-con] and the
 "login_service_account" setting on the
 [slurm-on-gcp login module][slurm-on-gcp-login].
@@ -493,7 +492,7 @@ message. Here are some common reasons for the deployment to fail:
 * **Filestore resource limit:** When regularly deploying filestore instances
   with a new vpc you may see an error during deployment such as:
   `System limit for internal resources has been reached`. See
-  [this doc](https://cloud.google.com/filestore/docs/troubleshooting#api_cannot_be_disabled)
+  [this doc](https://cloud.google.com/filestore/docs/troubleshooting#system_limit_for_internal_resources_has_been_reached_error_when_creating_an_instance)
   for the solution.
 * **Required permission not found:**
   * Example: `Required 'compute.projects.get' permission for 'projects/... forbidden`
@@ -536,13 +535,34 @@ drop-down menu at the top-left.
 
 ## Inspecting the Deployment
 
-The deployment is created in the directory matching the provided
-`deployment_name` variable in the blueprint. Within this directory are all the
-modules needed to deploy your cluster. The deployment directory will contain
-subdirectories representing the deployment groups defined in the blueprint file.
-Most example configurations contain a single deployment group.
+The deployment will be created with the following directory structure:
+
+```text
+<<OUTPUT_PATH>>/<<DEPLOYMENT_NAME>>/{<<DEPLOYMENT_GROUPS>>}/
+```
+
+If an output directory is provided with the `--output/-o` flag, the deployment
+directory will be created in the output directory, represented as
+`<<OUTPUT_PATH>>` here. If not provided, `<<OUTPUT_PATH>>` will default to the
+current working directory.
+
+The deployment directory is created in `<<OUTPUT_PATH>>` as a directory matching
+the provided `deployment_name` deployment variable (`vars`) in the blueprint.
+
+Within the deployment directory are directories representing each deployment
+group in the blueprint named the same as the `group` field for each element
+in `deployment_groups`.
 
-From the [example above](#quick-start) we get the following deployment directory:
+In each deployment group directory, are all of the configuration scripts and
+modules needed to deploy. The modules are in a directory named `modules` named
+the same as the source module, for example the
+[vpc module](./modules/network/vpc/README.md) is in a directory named `vpc`.
+
+A hidden directory containing meta information and backups is also created and
+named `.ghpc`.
+
+From the [hpc-cluster-small.yaml example](./examples/hpc-cluster-small.yaml), we
+get the following deployment directory:
 
 ```text
 hpc-small/
@@ -556,54 +576,9 @@ hpc-small/
       SchedMD-slurm-on-gcp-login-node/
       SchedMD-slurm-on-gcp-partition/
       vpc/
+    .ghpc/
 ```
 
-## `ghpc` Commands
-
-### Create
-
-``` shell
-./ghpc create <blueprint.yaml>
-```
-
-The create command is the primary interface for the HPC Toolkit. This command
-takes the path to a blueprint file as an input and creates a deployment based on
-it. Further information on creating this blueprint file, see
-[Writing an HPC Blueprint](examples/README.md#writing-an-hpc-blueprint).
-
-By default, the deployment directory will be created in the same directory as
-the `ghpc` binary and will have the name specified by the `deployment_name`
-field from the blueprint. Optionally, the output directory can be specified with
-the `-o` flag as shown in the following example.
-
-```shell
-./ghpc create examples/hpc-cluster-small.yaml -o deployments/
-```
-
-### Expand
-
-```shell
-./ghpc expand <blueprint.yaml> –out <expanded-blueprint.yaml>
-```
-
-The expand command creates an expanded blueprint file with all settings
-explicitly listed and variables expanded. This can be a useful tool for creating
-explicit, detailed examples and for debugging purposes. The expanded blueprint
-is still valid as input to [`ghpc create`](#create) to create the deployment.
-
-### Completion
-
-```shell
-./ghpc completion [bash|zsh|fish|powershell]
-```
-
-The completion command creates a shell completion config file for the specified
-shell. To apply the configuration file created by the command, it is required to
-set up for each shell. For example, loading the completion config by .bashrc is
-required for Bash.
-
-Call `ghpc completion --help` for shell specific setup instructions.
-
 ## Dependencies
 
 Much of the HPC Toolkit deployment is built using Terraform and Packer, and
@@ -620,33 +595,59 @@ List of dependencies:
 * make
 * git
 
-## MacOS Details
+### MacOS Additional Dependencies
+
+On macOS, `make` is packaged with the Xcode command line developer tools. To
+install, run the following command:
 
-* Install GNU `findutils` with Homebrew or Conda
-  * `brew install findutils` (and follow instructions for modifying `PATH`)
-  * `conda install findutils`
-* If using `conda`, it's easier to use conda-forge Golang without CGO
-  * `conda install go go-nocgo go-nocgo_osx-64`
+```shell
+xcode-select --install
+```
+
+Alternatively you can build `ghpc` directly using `go build ghpc.go`.
+
+### Notes on Packer
+
+The Toolkit supports Packer templates in the contemporary [HCL2 file
+format][pkrhcl2] and not in the legacy JSON file format. We require the use of
+Packer 1.7 or above, and recommend using the latest release.
+
+The Toolkit's [Packer template module documentation][pkrmodreadme] describes
+input variables and their behavior. An [image-building example][pkrexample]
+and [usage instructions][pkrexamplereadme] are provided. The example integrates
+Packer, Terraform and
+[startup-script](./modules/scripts/startup-script/README.md) runners to
+demonstrate the power of customizing images using the same scripts that can be
+applied at boot-time.
+
+[pkrhcl2]: https://www.packer.io/guides/hcl
+[pkrmodreadme]: modules/packer/custom-image/README.md
+[pkrexamplereadme]: examples/README.md#image-builderyaml
+[pkrexample]: examples/image-builder.yaml
 
 ## Development
 
 The following setup is in addition to the [dependencies](#dependencies) needed
 to build and run HPC-Toolkit.
 
 Please use the `pre-commit` hooks [configured](./.pre-commit-config.yaml) in
-this repository to ensure that all Terraform and golang modules are validated
-and properly documented before pushing code changes. The pre-commits configured
+this repository to ensure that all changes are validated, tested and properly
+documented before pushing code changes. The pre-commits configured
 in the HPC Toolkit have a set of dependencies that need to be installed before
 successfully passing.
 
+Follow these steps to install and setup pre-commit in your cloned repository:
+
 1. Install pre-commit using the instructions from [the pre-commit website](https://pre-commit.com/).
 1. Install TFLint using the instructions from
    [the TFLint documentation](https://github.com/terraform-linters/tflint#installation).
-   * Note: The version of TFLint must be compatible with the Google plugin
-     version identified in [tflint.hcl](.tflint.hcl). Versions of the plugin
-     `>=0.16.0` should use `tflint>=0.35.0` and versions of the plugin
-     `<=0.15.0` should preferably use `tflint==0.34.1`. These versions are
-     readily available via GitHub or package managers.
+
+   > **_NOTE:_** The version of TFLint must be compatible with the Google plugin
+   > version identified in [tflint.hcl](.tflint.hcl). Versions of the plugin
+   > `>=0.16.0` should use `tflint>=0.35.0` and versions of the plugin
+   > `<=0.15.0` should preferably use `tflint==0.34.1`. These versions are
+   > readily available via GitHub or package managers.
+
 1. Install ShellCheck using the instructions from
    [the ShellCheck documentation](https://github.com/koalaman/shellcheck#installing)
 1. The other dev dependencies can be installed by running the following command
@@ -665,26 +666,16 @@ successfully passing.
 
 Now pre-commit is configured to automatically run before you commit.
 
-### Packer
+### Development on macOS
 
-The Toolkit supports Packer templates in the contemporary [HCL2 file
-format][pkrhcl2] and not in the legacy JSON file format. We require the use of
-Packer 1.7 or above, and recommend using the latest release.
+While macOS is a supported environment for building and executing the Toolkit,
+it is not supported for Toolkit development due to GNU specific shell scripts.
 
-The Toolkit's [Packer template module documentation][pkrmodreadme] describes
-input variables and their behavior. An [image-building example][pkrexample]
-and [usage instructions][pkrexamplereadme] are provided. The example integrates
-Packer, Terraform and Toolkit Runners to demonstrate the power of customizing
-images using the same scripts that can be applied at boot-time.
-
-[pkrhcl2]: https://www.packer.io/guides/hcl
-[pkrmodreadme]: modules/packer/custom-image/README.md
-[pkrexamplereadme]: examples/README.md#image-builderyaml
-[pkrexample]: examples/image-builder.yaml
+If developing on a mac, a workaround is to install GNU tooling by installing
+`coreutils` and `findutils` from a package manager such as homebrew or conda.
 
 ### Contributing
 
 Please refer to the [contributing file](CONTRIBUTING.md) in our github repo, or
 to
 [Google’s Open Source documentation](https://opensource.google/docs/releasing/template/CONTRIBUTING/#).
-Before submitting, we recommend contributors run pre-commit tests (more below).