v1.15.0: Improvements to Slurm and HTCondor solutions

[tpdownes]

Key New Features

• Support for HTCondor pools with both On-demand and Spot VMs
• Slurm solution updated to 5.6.0
  • Support for custom machine types
  • Label exclusive nodes with job ID for cost-tracking
  • New zone_target_shape parameter corresponding to bulkInsert targetShape parameter
  • FIX: lustre mounting regression introduced in 5.5.0

Improvements

• [filestore] module added supported for Shared VPCs via use keyword and pre-existing-vpc module
• HTCondor modules now use minimally-scoped authentication for each daemon
• HTCondor execute points disable benchmarks to decrease time to join pool
• Improved type alignment across modules. e.g. var.labels aligned to map(string)

What's Changed

• Rename filestore network_name to network_id to enable shared VPC via use by @nick-stroud in Rename filestore network_name to network_id to enable shared VPC via use #962
• Improve attribute tracking in HTCondor scheduler by @tpdownes in Improve attribute tracking in HTCondor scheduler #965
• Update fluent tutorial to use pre-existing-vpc module and other minor syntax updates by @nick-stroud in Update fluent tutorial to use pre-existing-vpc module and other minor syntax updates #963
• Revert "Rename filestore network_name to network_id to enable shared VPC via use" by @nick-stroud in Revert "Rename filestore network_name to network_id to enable shared VPC via use" #967
• Mask sleep/suspend targets on chrome-remote-desktop to prevent shutdown by @nick-stroud in Mask sleep/suspend targets on chrome-remote-desktop to prevent shutdown #968
• Update image building example to use Slurm V5 by @mr0re1 in Update image building example to use Slurm V5 #964
• Improve HTCondor job matchmaking speed by @tpdownes in Improve HTCondor job matchmaking speed #971
• Roll-forward:"Rename filestore network_name to network_id to enable shared VPC via use" by @nick-stroud in Roll-forward:"Rename filestore network_name to network_id to enable shared VPC via use" #969
• Increase reliability of blueprints using DDN Exascaler by @tpdownes in Increase reliability of blueprints using DDN Exascaler #972
• Further increase speed at which HTCondor daemons update their ClassAds by @tpdownes in Further increase speed at which HTCondor daemons update their ClassAds #974
• Initial support for Spot VMs within HTCondor pools by @tpdownes in Initial support for Spot VMs within HTCondor pools #973
• Convert HTCondor autoscaler to SystemD timer by @tpdownes in Convert HTCondor autoscaler to SystemD timer #975
• Add validation to prevent usage of variables in backend block. by @mr0re1 in Add validation to prevent usage of variables in backend block. #970
• Making OFE deploy.sh MacOS compatible. Fixes OFE deployment fails when deploying from macOS. #978 by @ek-nag in Making OFE deploy.sh MacOS compatible. Fixes #978 #979
• Improve Slurm log capturing by @tpdownes in Improve Slurm log capturing #980
• Support Spot VMs in HTCondor pools by @tpdownes in Support Spot VMs in HTCondor pools #981
• Add utils for parising and normalizing HCL dtype by @mr0re1 in Add utils for parising and normalizing HCL dtype #977
• Enable depth-first filling of HTCondor pools by @tpdownes in Enable depth-first filling of HTCondor pools #982
• Escalate to root priveleges to fetch Slurm logs by @mr0re1 in Escalate to root priveleges to fetch Slurm logs #987
• Bump google.golang.org/api from 0.110.0 to 0.111.0 by @dependabot in Bump google.golang.org/api from 0.110.0 to 0.111.0 #984
• Bump github.com/spf13/afero from 1.9.4 to 1.9.5 by @dependabot in Bump github.com/spf13/afero from 1.9.4 to 1.9.5 #985
• Bump github.com/go-git/go-git/v5 from 5.4.2 to 5.6.0 by @dependabot in Bump github.com/go-git/go-git/v5 from 5.4.2 to 5.6.0 #986
• Bump dill from 0.3.4 to 0.3.6 in /community/front-end/ofe by @dependabot in Bump dill from 0.3.4 to 0.3.6 in /community/front-end/ofe #990
• Bump google-cloud-core from 2.2.2 to 2.3.2 in /community/front-end/ofe by @dependabot in Bump google-cloud-core from 2.2.2 to 2.3.2 in /community/front-end/ofe #991
• Bump astroid from 2.9.3 to 2.15.0 in /community/front-end/ofe by @dependabot in Bump astroid from 2.9.3 to 2.15.0 in /community/front-end/ofe #992
• Bump proto-plus from 1.20.1 to 1.22.2 in /community/front-end/ofe by @dependabot in Bump proto-plus from 1.20.1 to 1.22.2 in /community/front-end/ofe #993
• Bump isort from 5.10.1 to 5.12.0 in /community/front-end/ofe by @dependabot in Bump isort from 5.10.1 to 5.12.0 in /community/front-end/ofe #994
• Merge main into develop after release v1.14.0 by @mr0re1 in Merge main into develop after release v1.14.0 #997
• Bump terraform providers version 4.53.1 -> 4.56.0 by @mr0re1 in Bump terraform providers version 4.53.1 -> 4.56.0 #998
• Clean up Filestore regardless of instances presence by @mr0re1 in Clean up Filestore regardless of instances presence #999
• Upgrade to slurm-gcp 5.6.0 by @SkylerMalinowski in Upgrade to slurm-gcp 5.6.0 #995
• Fix nfs-server example to use local_mounts instead of local_mount by @nick-stroud in Fix nfs-server example to use local_mounts instead of local_mount #1001
• Add missing description for gcs_bucket_path by @nick-stroud in Add missing description for gcs_bucket_path #1002
• Doc fix by @issacg in Doc fix #1010
• Add mounting of cloud-storage-bucket to Slurm v5 test by @nick-stroud in Add mounting of cloud-storage-bucket to Slurm v5 test #1007
• Use DeploymentName getter instead of looking up Vars by @mr0re1 in Use DeploymentName getter instead of looking up Vars #1005
• Specify strict type for labels = map(string) by @mr0re1 in Specify strict type for labels = map(string) #1000
• Pass empty string instead of null to avoid mounting failure in Slurm by @nick-stroud in Pass empty string instead of null to avoid mounting failure in Slurm #1003
• Remove ghpc_role setting from nfs-server example by @nick-stroud in Remove ghpc_role setting from nfs-server example #1008
• Actually check mount instead of just checking dir exists by @nick-stroud in Actually check mount instead of just checking dir exists #1004
• Remove hostname test as it is not providing incremental value by @nick-stroud in Remove hostname test as it is not providing incremental value #1006
• Double length of time for HTCondor integration test to detect job queue by @tpdownes in Double length of time for HTCondor integration test to detect job queue #1020
• Bump github.com/googleapis/gax-go/v2 from 2.7.0 to 2.7.1 by @dependabot in Bump github.com/googleapis/gax-go/v2 from 2.7.0 to 2.7.1 #1011
• Bump github.com/hashicorp/hcl/v2 from 2.16.1 to 2.16.2 by @dependabot in Bump github.com/hashicorp/hcl/v2 from 2.16.1 to 2.16.2 #1012
• Update slurm v5 readme about local-exec dependencies by @mr0re1 in Update slurm v5 readme about local-exec dependencies #1023
• Bump google.golang.org/api from 0.111.0 to 0.112.0 by @dependabot in Bump google.golang.org/api from 0.111.0 to 0.112.0 #1013
• Update OFE Dependabot configuration by @tpdownes in Update OFE Dependabot configuration #1055
• Release v1.15.0 by @tpdownes in Release v1.15.0 #1065

New Contributors

• @issacg made their first contribution in Doc fix #1010

Full Changelog: v1.14.1...v1.15.0

---

This discussion was created from the release v1.15.0: Improvements to Slurm and HTCondor solutions.