V1.20.0 Filestore for GKE, Improved Windows support, & git-hosted Packer Modules

[mr0re1]

Key New Features

• Native GKE support for Filestore: storage-gke example.
• Improved support for Windows - Packer and windows-startup-script;
• Packer "packages" - treat remote (git-hosted) Packer modules as packages when using Terraform's "//" notation;
• Automate DAOS server/client images

New Modules

• gke-persistent-volume: automatically creates persistent volumes and persistent volume claims for shared storage.
• windows-startup-script: a simple module that curates scripts for customizing Windows VMs.

Module Improvements

• vm_instance:

  • do not swap boot disk (and VM) each time a new disk image is available;

• vpc:

  • enabling TCP tunneling to the WinRM port used by PowerShell;
  • add firewall rule for SSH from arbitrary IP ranges;

• gke-node-pool:

  • add option for static node count;
  • add option to enable gcfs;

• gke-cluster:

  • expose the option to not create a system node pool;
  • add option to create and update timeouts;
  • update service account variable to separate email and scopes;

• gke-job-template: add templating for persistent volume claims

• custom_image:

  • add disk_type support;
  • add Powershell script support;
  • improved support for Windows;
  • treat remote (git-hosted) Packer modules as packages when using Terraform's "//" notation

• htcondor-install:

  • add support for fixed version of HTCondor;
  • improve resilience;

• schedmd-slurm-gcp-v5-controller: allow providing short references for image project

• batch-job-template: use Batch HPC CentOS images as default image

Version updates

• Update to slurm-gcp 5.7.4
• Update google-cloud-daos from v0.4.0 to v0.4.1

What's Changed

• Merge v1.18.1 back to develop by @nick-stroud in Merge v1.18.1 back to develop #1421
• Improve vpc module by @tpdownes in Improve vpc module #1422
• Expose the option on gke-cluster to not create a system node pool by @nick-stroud in Expose the option on gke-cluster to not create a system node pool #1425
• Add option to gke-node-pool for static node count by @nick-stroud in Add option to gke-node-pool for static node count #1424
• Add options for create and update timeouts to gke modules by @nick-stroud in Add options for create and update timeouts to gke modules #1426
• Update gke service account variable to separate email and scopes by @nick-stroud in Update gke service account variable to separate email and scopes #1427
• Update HTCondor example to use Rocky Linux 8 by @tpdownes in Update HTCondor example to use Rocky Linux 8 #1432
• Update develop with release-candidate: Fix Ansible installation upon re-run by @rohitramu in Update develop with release-candidate: Fix Ansible installation upon re-run #1439
• Improver Packer support for Windows by @tpdownes in Improver Packer support for Windows #1431
• Update HTCondor execute point module by @tpdownes in Update HTCondor execute point module #1434
• Add option to enable gcfs on gke-node-pool by @nick-stroud in Add option to enable gcfs on gke-node-pool #1428
• resreader.go code clean up by @mr0re1 in resreader.go code clean up #1445
• Improve HTCondor example and integration tests by @tpdownes in Improve HTCondor example and integration tests #1440
• Do not use log.Fatal in pkg/config by @mr0re1 in Do not use log.Fatal in pkg/config #1444
• Remove Module.RequiredApis by @mr0re1 in Remove Module.RequiredApis #1446
• Conditionally exclude nodeSelector when not needed by @nick-stroud in Conditionally exclude nodeSelector when not needed #1441
• Adopt latest release of startup-script modules by @tpdownes in Adopt latest release of startup-script modules #1411
• Add docs/module-guidelines.md by @mr0re1 in Add docs/module-guidelines.md #1423
• Improve Packer experience for Windows by @tpdownes in Improve Packer experience for Windows #1447
• Add inert Module.RequiresApis for backward compatibility by @mr0re1 in Add inert Module.RequiresApis for backward compatibility #1450
• Bump google.golang.org/api from 0.125.0 to 0.126.0 by @dependabot in Bump google.golang.org/api from 0.125.0 to 0.126.0 #1430
• Fix regex for validating GroupName by @mr0re1 in Fix regex for validating GroupName #1449
• Reduce size of expanded blueprint by adding omitempty where applicable by @mr0re1 in Reduce size of expanded blueprint by adding omitempty where applicable #1452
• Remove Module.DeploymentSource, compute it on demand by @mr0re1 in Remove Module.DeploymentSource, compute it on demand #1453
• Merging v1.19.0 from main back into develop by @rohitramu in Merging v1.19.0 from main back into develop #1462
• Fix static_check warnings in cmd/root*.go by @mr0re1 in Fix static_check warnings in cmd/root*.go #1460
• Slurm gcp 5.7.4 by @SkylerMalinowski in Slurm gcp 5.7.4 #1459
• Fix panic while attempting to tokenize Null-value by @mr0re1 in Fix panic while attempting to tokenize Null-value #1468
• Update "google", "google-beta" providers to 4.69.1 by @rohitramu in Update "google", "google-beta" providers to 4.69.1 #1464
• Adds gke-persistent-volume module by @nick-stroud in Adds gke-persistent-volume module #1442
• Add documentation for gke-persistent-volume-module by @nick-stroud in Add documentation for gke-persistent-volume-module #1478
• Merge v1.19.1 hotfix release into develop by @tpdownes in Merge v1.19.1 hotfix release into develop #1480
• Bump golang.org/x/sys from 0.8.0 to 0.9.0 by @dependabot in Bump golang.org/x/sys from 0.8.0 to 0.9.0 #1475
• Bump github.com/otiai10/copy from 1.11.0 to 1.12.0 by @dependabot in Bump github.com/otiai10/copy from 1.11.0 to 1.12.0 #1476
• Bump google.golang.org/api from 0.126.0 to 0.128.0 by @dependabot in Bump google.golang.org/api from 0.126.0 to 0.128.0 #1477
• Address minor warnings and lint issues by @tpdownes in Address minor warnings and lint issues #1481
• Relax TestNetworkStorage to accomodate gke-persistent-volume by @mr0re1 in Relax TestNetworkStorage to accomodate gke-persistent-volume #1483
• Deprecated WrapSettingsWith by @mr0re1 in Deprecated WrapSettingsWith #1466
• Print advanced instructions after ghpc deploy by @mr0re1 in Print advanced instructions after ghpc deploy #1463
• Add terraform_backend_defaults section to some examples by @mr0re1 in Add terraform_backend_defaults section to some examples #1469
• Use consistent order in "product of module use" mark. by @mr0re1 in Use consistent order in "product of module use" mark. #1484
• Add support for Packer packages by @tpdownes in Add support for Packer packages #1467
• Add community example of how to use filestore with gke by @nick-stroud in Add community example of how to use filestore with gke #1443
• Remove excessive error messages by @mr0re1 in Remove excessive error messages #1485
• Add rich error messages with position and snippet by @mr0re1 in Add rich error messages with position and snippet #1448
• Update "google" provider in OFE from 3.x to 4.x by @rohitramu in Update "google" provider in OFE from 3.x to 4.x #1470
• Use strict Path builder to reduce human error. by @mr0re1 in Use strict Path builder to reduce human error. #1489
• Remove settingsToIgnore from useModule by @mr0re1 in Remove settingsToIgnore from useModule #1486
• Allow providing short names for image project by @rohitramu in Allow providing short names for image project #1472
• Remove debug output from create command by @tpdownes in Remove debug output from create command #1492
• Add custom unmarshaler for Module.Use for better error messaging by @mr0re1 in Add custom unmarshaler for Module.Use for better error messaging #1473
• Use regexall instead of strcontains to stay compatible with terraform 1.2 by @rohitramu in Use regexall instead of strcontains to stay compatible with terraform 1.2 #1493
• Don't swap VM boot disk (and VM) each time a new disk image is available by @issacg in Don't swap VM boot disk (and VM) each time a new disk image is available #1474
• Module documentation update and improved DAOS examples by @tpdownes in Module documentation update and improved DAOS examples #1488
• Add automatic_restart to vm-instance by @mr0re1 in Add automatic_restart to vm-instance #1288
• Add pipefail to Makefile to prevent swallowing failed tests by @mr0re1 in Add pipefail to Makefile to prevent swallowing failed tests #1498
• Vm instance boot disk lifecycle changes by @cboneti in Vm instance boot disk lifecycle changes #1494
• Remove "failed tests" check from enforce_coverage since it doesn't … by @mr0re1 in Remove "failed tests" check from enforce_coverage since it doesn't … #1499
• Drop coverage requirement for pkg/shell by @tpdownes in Drop coverage requirement for pkg/shell #1507
• Update google-cloud-daos version from v0.4.0 to v0.4.1 by @mark-olson in Update google-cloud-daos version from v0.4.0 to v0.4.1 #1503
• Remove getAbsSourcePath as it re-implements filepath.Abs by @mr0re1 in Remove getAbsSourcePath as it re-implements filepath.Abs #1505
• Update OFE requirements by @tpdownes in Update OFE requirements #1508
• Fix broken link by @mr0re1 in Fix broken link #1511
• Document scope of Windows support in HPC Toolkit by @tpdownes in Document scope of Windows support in HPC Toolkit #1512
• Enable flatten and merge functions in Eval. by @mr0re1 in Enable flatten and merge functions in Eval. #1513
• Use Batch HPC CentOS images as default image by @wenyhu-google in Use Batch HPC CentOS images as default image #1482
• Stop using deprecated variables in vpc and batch-login-node by @mr0re1 in Stop using deprecated variables in vpc and batch-login-node #1514
• Reduce size of Spack builder as max default is to use 16 cores by @nick-stroud in Reduce size of Spack builder as max default is to use 16 cores #1515
• Make Windows VM building Toolkit-native by @tpdownes in Make Windows VM building Toolkit-native #1510
• Update OFE Terraform module by @tpdownes in Update OFE Terraform module #1520
• Document known incompatibility of Slurm module with local-ssds by @tpdownes in Document known incompatibility of Slurm module with local-ssds #1529
• Update HTCondor tutorial by @tpdownes in Update HTCondor tutorial #1549
• Version update 1.19.1 -> 1.20.0 by @mr0re1 in Version update 1.19.1 -> 1.20.0 #1550
• Update Django 4.2.2 -> 4.2.3 by @mr0re1 in Update Django 4.2.2 -> 4.2.3 #1552
• Release v1.20.0 by @mr0re1 in Release v1.20.0 #1551

New Contributors

• @wenyhu-google made their first contribution in Use Batch HPC CentOS images as default image #1482

Full Changelog: v1.19.1...v1.20.0

---

This discussion was created from the release V1.20.0 Filestore for GKE, Improved Windows support, & git-hosted Packer Modules.