v1.18.0: ghpc deploy, new examples, better examples names, slurm-gcp 5.7.2

[cboneti]

Key New Features

• ghpc deploy is now the recommended way of deploying your environments
• multigroup blueprints may now use module outputs from one group to another
  • e.g., a. network may be dynamically created in group 1 and its name will be available directly in group 2
• New hpc-enterprise blueprint with various high performance options
• New ML blueprints: ml-slurm.yaml and ml-gke.yaml
• Blueprints renamed for more clarity
• Ability to communicate variables across deployment groups with ghpc deploy or ghpc export-outputs and ghpc import-inputs
• Slurm on GCP V4.x is now deprecated, all core examples are moved to V5.7.2

Examples

• htc-slurm.yaml: shows how to provision a cluster with configuration tuned for many short-duration, loosely coupled jobs.
• client-google-cloud-storage.yaml: demonstrates different ways to use Google Cloud Storage (GCS) buckets in the HPC Toolkit.

New Modules

• gke-job-template: Creates a Kubernetes job templated file that can be used to submit jobs.
• kubernetes-operations: Performs pre-defined operations on Kubernetes resources that would otherwise be executed using kubectl.

Module Improvements

• gke-cluster: Added GPU support and automated installation of Nvidia drivers.

Deprecations

• Slurm V4.x modules: partition, controller and login-node.

Version updates

• schedmd-slurm-gcp-v5-controller: update SchedMD modules to 5.7.2
• Min required Terraform version bumped 1.0 -> 1.2
• Min required Packer version bumped 1.6 -> 1.7.9

What's Changed

• Include group kind in deployment metadata by @tpdownes in Include group kind in deployment metadata #1213
• Bump google.golang.org/api from 0.118.0 to 0.119.0 by @dependabot in Bump google.golang.org/api from 0.118.0 to 0.119.0 #1209
• Bump github.com/otiai10/copy from 1.10.0 to 1.11.0 by @dependabot in Bump github.com/otiai10/copy from 1.10.0 to 1.11.0 #1210
• Increase get URL timeout for CRD module by @tpdownes in Increase get URL timeout for CRD module #1211
• Use optimize utilization autoscaling profile by @nick-stroud in Use optimize utilization autoscaling profile #1214
• Retry project cleanup up to 4 times each night by @tpdownes in Retry project cleanup up to 4 times each night #1217
• Use deadline instead of retries in wait-for-startup by @mr0re1 in Use deadline instead of retries in wait-for-startup #1216
• Silence daily cleanup notifications and enable retries for other builds by @tpdownes in Silence daily cleanup notifications and enable retries for other builds #1218
• Bump minimal Terraform version 1.0 -> 1.2 by @mr0re1 in Bump minimal Terraform version 1.0 -> 1.2 #1178
• Implement stub export-outputs command by @tpdownes in Implement stub export-outputs command #1219
• Bump minimum Terraform in golden copy deployments by @tpdownes in Bump minimum Terraform in golden copy deployments #1222
• Use Dict for Module.Settings, derive connectivity from it by @mr0re1 in Use Dict for Module.Settings, derive connectivity from it #1205
• Initial implementation of export-outputs command by @tpdownes in Initial implementation of export-outputs command #1225
• Minor refactoring config.go by @mr0re1 in Minor refactoring config.go #1223
• Implement stub import-inputs command by @tpdownes in Implement stub import-inputs command #1226
• Add better version comparator for Makefile by @mr0re1 in Add better version comparator for Makefile #1215
• Fix whitespace in deployment directories by @tpdownes in Fix whitespace in deployment directories #1227
• Update git clone instruction to use HTTPS instead of SSH by @mr0re1 in Update git clone instruction to use HTTPS instead of SSH #1233
• Add ghpc version to expanded blueprint by @mr0re1 in Add ghpc version to expanded blueprint #1224
• Update GKE settings to match recommendations from GKE team by @nick-stroud in Update GKE settings to match recommendations from GKE team #1231
• Bump min packer version to 1.7.9 by @mr0re1 in Bump min packer version to 1.7.9 #1232
• Fail wait-for-startup fast if log can not be fetched by @mr0re1 in Fail wait-for-startup fast if log can not be fetched #1220
• Remove typo in README heading by @nick-stroud in Remove typo in README heading #1237
• Fix missing command to print out by @mr0re1 in Fix missing command to print out #1238
• Handle "wrong-type-of-packer" in make warn-packer-missing by @mr0re1 in Handle "wrong-type-of-packer" in make warn-packer-missing #1239
• Fix Chrome Remote Desktop NVIDIA Grid installation by @tpdownes in Fix Chrome Remote Desktop NVIDIA Grid installation #1240
• Address shellcheck -o all wait-for-startup-status.sh by @mr0re1 in Address shellcheck -o all wait-for-startup-status.sh #1242
• Fix retry configuration for daily integration tests by @tpdownes in Fix retry configuration for daily integration tests #1236
• Do not store ModuleInfo in DeploymentConfig by @mr0re1 in Do not store ModuleInfo in DeploymentConfig #1230
• Create a gke-job-template module, which creates a Kubernetes job file by @nick-stroud in Create a gke-job-template module, which creates a Kubernetes job file #1234
• Ensure that terraform cleanup always runs by @tpdownes in Ensure that terraform cleanup always runs #1235
• Remove unused method HasKind by @mr0re1 in Remove unused method HasKind #1246
• Add option to select zones for gke-node-pool by @nick-stroud in Add option to select zones for gke-node-pool #1245
• Add the gke-job-template module to the list of modules by @nick-stroud in Add the gke-job-template module to the list of modules #1243
• Initial implementation of import-inputs command by @tpdownes in Initial implementation of import-inputs command #1228
• Remove ansible-lint to unblock PRs by @mr0re1 in Remove ansible-lint to unblock PRs #1257
• Skip TestFindTerraform if no terraform is installed by @mr0re1 in Skip TestFindTerraform if no terraform is installed #1255
• Unify shared code of create and expand commands by @mr0re1 in Unify shared code of create and expand commands #1244
• Bump google.golang.org/api from 0.119.0 to 0.120.0 by @dependabot in Bump google.golang.org/api from 0.119.0 to 0.120.0 #1253
• Add documentation warning about lustre license cost by @nick-stroud in Add documentation warning about lustre license cost #1254
• Remove modReference by @mr0re1 in Remove modReference #1247
• Bump cryptography from 40.0.1 to 40.0.2 in /community/front-end/ofe by @dependabot in Bump cryptography from 40.0.1 to 40.0.2 in /community/front-end/ofe #1252
• Bump protobuf from 4.22.1 to 4.22.3 in /community/front-end/ofe by @dependabot in Bump protobuf from 4.22.1 to 4.22.3 in /community/front-end/ofe #1250
• Bump pyasn1-modules from 0.2.8 to 0.3.0 in /community/front-end/ofe by @dependabot in Bump pyasn1-modules from 0.2.8 to 0.3.0 in /community/front-end/ofe #1251
• Bump pyasn1 from 0.4.8 to 0.5.0 in /community/front-end/ofe by @dependabot in Bump pyasn1 from 0.4.8 to 0.5.0 in /community/front-end/ofe #1248
• Make Expression into interface by @mr0re1 in Make Expression into interface #1260
• Refactor create_deployment.sh by @nick-stroud in Refactor create_deployment.sh #1258
• Address usability suggestions for multi-group deployments by @tpdownes in Address usability suggestions for multi-group deployments #1262
• Eliminate deployment metadata by @tpdownes in Eliminate deployment metadata #1265
• Use dedicated dtype ModuleID and GroupName instead of string by @mr0re1 in Use dedicated dtype ModuleID and GroupName instead of string #1264
• Adds a basic gke test which provisions and destroys a cluster by @nick-stroud in Adds a basic gke test which provisions and destroys a cluster #1259
• Fix link in image builder example by @tpdownes in Fix link in image builder example #1269
• Eliminate warnings by @tpdownes in Eliminate warnings #1277
• Add Terraform state download command to stdout of integration tests by @tpdownes in Add Terraform state download command to stdout of integration tests #1278
• Write Packer intergroup input values by @tpdownes in Write Packer intergroup input values #1268
• Resolve conflicts before merging main into develop by @mr0re1 in Resolve conflicts before merging main into develop #1282
• Rearange golden_copy tests to not mix with othe configs to validate by @mr0re1 in Rearange golden_copy tests to not mix with othe configs to validate #1283
• Filter blueprint autocompletion on YAML files by @tpdownes in Filter blueprint autocompletion on YAML files #1272
• Expose disk_size_gb, disk_type, & guest_accelerator on gke-node-pool by @nick-stroud in Expose disk_size_gb, disk_type, & guest_accelerator on gke-node-pool #1285
• Add "golden copy" test for text escaping by @mr0re1 in Add "golden copy" test for text escaping #1263
• Add add_deployment_name_before_prefix to blueprints used in test by @mr0re1 in Add add_deployment_name_before_prefix to blueprints used in test #1270
• Fix Ansible panic when IP address not found by @tpdownes in Fix Ansible panic when IP address not found #1293
• Update the terraform provider versions by @mr0re1 in Update the terraform provider versions #1287
• Bump golang.org/x/sys from 0.7.0 to 0.8.0 by @dependabot in Bump golang.org/x/sys from 0.7.0 to 0.8.0 #1295
• Bump google.golang.org/api from 0.120.0 to 0.121.0 by @dependabot in Bump google.golang.org/api from 0.120.0 to 0.121.0 #1296
• Add intergroup instructions to output of create by @tpdownes in Add intergroup instructions to output of create #1289
• Add a simple example using Google Cloud Storage buckets by @nick-stroud in Add a simple example using Google Cloud Storage buckets #1261
• Change default gke-job-template.backoff_limit to have shared fate by @nick-stroud in Change default gke-job-template.backoff_limit to have shared fate #1301
• Deprecating Slurm-gcp-V4, removing experimental from V5 by @cboneti in Deprecating Slurm-gcp-V4, removing experimental from V5 #1298
• Make doc section about enable_reconfigure more visible, add it to e… by @mr0re1 in Make doc section about enable_reconfigure more visible, add it to e… #1284
• Add documentation for using GPUs with GKE by @nick-stroud in Add documentation for using GPUs with GKE #1299
• Remove unused zone policy settings from Slurm v5 node group module by @tpdownes in Remove unused zone policy settings from Slurm v5 node group module #1302
• Fix names of VM resources in HTCondor tutorial by @tpdownes in Fix names of VM resources in HTCondor tutorial #1305
• Update to slurm-gcp 5.7.1 by @SkylerMalinowski in Update to slurm-gcp 5.7.1 #1291
• Simplify zone finding in Slurm v5 partition module by @tpdownes in Simplify zone finding in Slurm v5 partition module #1306
• Don't perform file-wide text substitutions by @mr0re1 in Don't perform file-wide text substitutions #1292
• Add ramble-setup module by @douglasjacobsen in Add ramble-setup module #1297
• Initial implementation of deploy command by @tpdownes in Initial implementation of deploy command #1304
• Add module labels for billing filtering by @nick-stroud in Add module labels for billing filtering #1308
• Update to slurm-gcp 5.7.2 by @SkylerMalinowski in Update to slurm-gcp 5.7.2 #1309
• Creates GPU GKE example, waiting for operations module by @nick-stroud in Creates GPU GKE example, waiting for operations module #1300
• Modify image-builder test by @tpdownes in Modify image-builder test #1311
• Update hpc-cluster-small example to SlurmV5 by @mr0re1 in Update hpc-cluster-small example to SlurmV5 #1315
• Rename example daos-cluster -> pfs-daos. by @mr0re1 in Rename example daos-cluster -> pfs-daos. #1317
• Rename example lustre -> pfs-lustre by @mr0re1 in Rename example lustre -> pfs-lustre #1318
• Rename multiple slurm examples by @mr0re1 in Rename multiple slurm examples #1319
• Rename example htcondor-pool -> hpc-htcondor by @mr0re1 in Rename example htcondor-pool -> hpc-htcondor #1320
• Deprecate network_ip variable as it had no effect and static_ips performs same function by @nick-stroud in Deprecate network_ip variable as it had no effect and static_ips performs same function #1316
• Remove legacy tests provisioning by @mr0re1 in Remove legacy tests provisioning #1312
• Update docs on enable_reconfigure now that gcloud bug is fixed by @nick-stroud in Update docs on enable_reconfigure now that gcloud bug is fixed #1322
• Rename example blueprints by @mr0re1 in Rename example blueprints #1323
• Bump cloud.google.com/go/serviceusage from 1.6.0 to 1.6.1 by @dependabot in Bump cloud.google.com/go/serviceusage from 1.6.0 to 1.6.1 #1327
• Bump google.golang.org/api from 0.121.0 to 0.122.0 by @dependabot in Bump google.golang.org/api from 0.121.0 to 0.122.0 #1326
• Bump cloud.google.com/go/compute from 1.19.1 to 1.19.2 by @dependabot in Bump cloud.google.com/go/compute from 1.19.1 to 1.19.2 #1325
• Modify instructions to user by @tpdownes in Modify instructions to user #1314
• Rename slurm examples by @mr0re1 in Rename slurm examples #1324
• Adding hpc-enterprise-slurm example by @cboneti in Adding hpc-enterprise-slurm example #1321
• Follow up from Update docs on enable_reconfigure now that gcloud bug is fixed #1322, remove --user & note about machine_type w/ enable_reconfigure by @nick-stroud in Follow up from #1322, remove --user & note about machine_type w/ enable_reconfigure #1328
• Remove full dynamic partition example in favor of simple snippet by @nick-stroud in Remove full dynamic partition example in favor of simple snippet #1329
• Add example for ML cluster by @tpdownes in Add example for ML cluster #1313
• Updates for high throughput examples by @tpdownes in Updates for high throughput examples #1330
• Add documentation about IAP IAM role and provide example of adding SSH firewall rule by @nick-stroud in Add documentation about IAP IAM role and provide example of adding SSH firewall rule #1333
• Update Slurm images to align with 5.7.2 release of slurm-gcp by @tpdownes in Update Slurm images to align with 5.7.2 release of slurm-gcp #1332
• fix: heading level was out of sync with surrounding headings by @nick-stroud in fix: heading level was out of sync with surrounding headings #1334
• Fix filenames and links for renamed HTCondor example by @tpdownes in Fix filenames and links for renamed HTCondor example #1337
• Fix missing / in high-io-slurm-gcp-v5 test config by @mr0re1 in Fix missing / in high-io-slurm-gcp-v5 test config #1338
• Fix link for ml-gke.yaml by @nick-stroud in Fix link for ml-gke.yaml #1339
• Update Slurm Controller documentation to point to version 5.7.2. by @rohitramu in Update Slurm Controller documentation to point to version 5.7.2. #1340
• Add kubernetes-operations module to list of modules by @nick-stroud in Add kubernetes-operations module to list of modules #1342
• renaming htc-slurm, reordering deprecated examples by @cboneti in renaming htc-slurm, reordering deprecated examples #1341
• Update version to 1.18.0 by @cboneti in Update version to 1.18.0 #1355
• Release v1.18.0 by @cboneti in Release v1.18.0 #1354

Full Changelog: v1.17.0...v1.18.0

---

This discussion was created from the release v1.18.0: ghpc deploy, new examples, better examples names, slurm-gcp 5.7.2 .