v1.14.0: HTCondor highly available, HCLS blueprint

[mr0re1]

Key New Features

• HCLS blueprint supports running GROMACS on GPUs and has added several tutorials.
• Support for highly available HTCondor pools
  • Job queue (SchedD) high availability remains experimental see README

Module Improvements

• vpc: new option to enable firewall rule that allows tunneling of Windows Remote Desktop connections
• schedmd-slurm-gcp-v5-partition: all deprecated variables have been removed; these have migrated to schedmd-slurm-gcp-v5-node-group
• htcondor-configure:
  • job history will now include VM instance ID, zone and machine type
  • VMs are now provisioned with minimally-permissioned IDTOKENs for their respective daemons (e.g. ADVERTISE_STARTD)
• startup-script: installation script for Cloud Ops Agent on Debian platforms will retry when other processes are blocking apt operations
• htcondor-execute-point: add a simple health check of port 9618 on any machine within the execute point
• vm-instance: vm-instance can be named using both a prefix and the deployment name

Improvements

• Improved error message when YAML blueprint has syntax errors preventing it from being loaded
• Regular updates to Go and Python dependencies to address potential security vulnerabilities
• Fixed Open Front End (OFE) issue with static content (icons) not displaying properly

What's Changed

• Add Windows Remote Desktop IAP firewall rule by @tpdownes in Add Windows Remote Desktop IAP firewall rule #885
• Bump oauthlib from 3.2.1 to 3.2.2 in /community/front-end/ofe by @dependabot in Bump oauthlib from 3.2.1 to 3.2.2 in /community/front-end/ofe #886
• Fix: having the same share name and local mount caused slurm failure by @nick-stroud in Fix: having the same share name and local mount caused slurm failure #887
• Implicitly add outputs to modules when they are being used across deployment groups by @tpdownes in Implicitly add outputs to modules when they are being used across deployment groups #878
• Reorder validator list to test blueprint correctness first by @heyealex in Reorder validator list to test blueprint correctness first #889
• Bump cryptography from 36.0.1 to 39.0.1 in /community/front-end/ofe by @dependabot in Bump cryptography from 36.0.1 to 39.0.1 in /community/front-end/ofe #890
• Adding example sbatch and config for Factor Xa protein by @nick-stroud in Adding example sbatch and config for Factor Xa protein #888
• Update HCLS blueprint examples to run Gromacs w/ GPUs by @nick-stroud in Update HCLS blueprint examples to run Gromacs w/ GPUs #891
• Add support for highly available HTCondor Central Managers by @tpdownes in Add support for highly available HTCondor Central Managers #892
• Remove deprecations from slurm-gcp v5 partition by @heyealex in Remove deprecations from slurm-gcp v5 partition #893
• Bump cryptography from 36.0.1 to 39.0.1 in /community/front-end/ofe by @tpdownes in Bump cryptography from 36.0.1 to 39.0.1 in /community/front-end/ofe #896
• Bump cryptography from 36.0.1 to 39.0.1 in /community/front-end/ofe by @dependabot in Bump cryptography from 36.0.1 to 39.0.1 in /community/front-end/ofe #894
• Ensure Open Front End dependabot updates target develop by @tpdownes in Ensure Open Front End dependabot updates target develop #897
• Fix: Always generate Batch instance template to avoid known at apply time error by @nick-stroud in Fix: Always generate Batch instance template to avoid known at apply time error #898
• Update sbatch to copy results to output bucket & minor tweaks by @nick-stroud in Update sbatch to copy results to output bucket & minor tweaks #895
• Add troubleshooting documentation for filestore share name exportfs bug by @nick-stroud in Add troubleshooting documentation for filestore share name exportfs bug #899
• Bump github.com/hashicorp/hcl/v2 from 2.16.0 to 2.16.1 by @dependabot in Bump github.com/hashicorp/hcl/v2 from 2.16.0 to 2.16.1 #900
• Fix addlicense check in weekly image building by @tpdownes in Fix addlicense check in weekly image building #901
• Update HTCondor modules by @tpdownes in Update HTCondor modules #902
• Bump django from 3.2.16 to 3.2.17 in /community/front-end/ofe by @dependabot in Bump django from 3.2.16 to 3.2.17 in /community/front-end/ofe #905
• Bump cryptography from 36.0.1 to 39.0.1 in /community/front-end/ofe by @dependabot in Bump cryptography from 36.0.1 to 39.0.1 in /community/front-end/ofe #906
• Bump oauthlib from 3.2.1 to 3.2.2 in /community/front-end/ofe by @dependabot in Bump oauthlib from 3.2.1 to 3.2.2 in /community/front-end/ofe #907
• Bump TFLint version in image to latest by @tpdownes in Bump TFLint version in image to latest #903
• Add instructions to ssh to VM created by vm-instance by @nick-stroud in Add instructions to ssh to VM created by vm-instance #880
• Address terraform_unused_required_providers errors by @tpdownes in Address terraform_unused_required_providers errors #908
• Manage HTCondor yum repo configuration directly by @tpdownes in Manage HTCondor yum repo configuration directly #904
• Bump github.com/aws/aws-sdk-go from 1.33.0 to 1.34.0 by @dependabot in Bump github.com/aws/aws-sdk-go from 1.33.0 to 1.34.0 #911
• Fail integration tests on validation warnings. by @mr0re1 in Fail integration tests on validation warnings. #910
• Set minimal scopes for HTCondor IDTOKENs by @tpdownes in Set minimal scopes for HTCondor IDTOKENs #919
• Fix cmd/root_test.go test runs from linked Git worktrees. by @mr0re1 in Fix cmd/root_test.go test runs from linked Git worktrees. #918
• OFE update 14/02/2023. by @ek-nag in OFE update 14/02/2023. #913
• Improve error message for yaml parsing failures by @heyealex in Improve error message for yaml parsing failures #923
• HTCondor job track machine information by @tpdownes in HTCondor job track machine information #924
• Update develop with release v1.13.0 by @nick-stroud in Update develop with release v1.13.0 #928
• Add explicit output dependencies to HTCondor by @tpdownes in Add explicit output dependencies to HTCondor #925
• Pin terraform google provider to v4.53.1 by @nick-stroud in Pin terraform google provider to v4.53.1 #929
• Fix root_test failure on MacOS by @mr0re1 in Fix root_test failure on MacOS #932
• Update htcondor-configure README example snippet by @tpdownes in Update htcondor-configure README example snippet #935
• Bump github.com/hashicorp/go-getter from 1.6.2 to 1.7.0 by @dependabot in Bump github.com/hashicorp/go-getter from 1.6.2 to 1.7.0 #930
• Add retries to cloud ops install by @heyealex in Add retries to cloud ops install #933
• Bump django from 3.2.17 to 3.2.18 in /community/front-end/ofe by @dependabot in Bump django from 3.2.17 to 3.2.18 in /community/front-end/ofe #922
• Remove unused error message by @tpdownes in Remove unused error message #939
• Bump google.golang.org/api from 0.109.0 to 0.110.0 by @dependabot in Bump google.golang.org/api from 0.109.0 to 0.110.0 #937
• Fix typo in hcls instructions command by @nick-stroud in Fix typo in hcls instructions command #940
• Update hcls example to use lysozyme protein instead of factor xa by @nick-stroud in Update hcls example to use lysozyme protein instead of factor xa #942
• HTCondor Job Queue High Availability by @tpdownes in HTCondor Job Queue High Availability #934
• Refactor useModule by @tpdownes in Refactor useModule #941
• Bugfix to HTCondor autoscaler script by @tpdownes in Bugfix to HTCondor autoscaler script #945
• Update hcls spack builder to use c2 machine by @nick-stroud in Update hcls spack builder to use c2 machine #948
• Enable OS Login by default in HTCondor execute points by @tpdownes in Enable OS Login by default in HTCondor execute points #944
• Remove dependency on unused module by @mr0re1 in Remove dependency on unused module #947
• Add health check for HTCondor VMs by @tpdownes in Add health check for HTCondor VMs #946
• HCLS tutorial update by @nick-stroud in HCLS tutorial update #950
• Bump github.com/spf13/afero from 1.9.3 to 1.9.4 by @dependabot in Bump github.com/spf13/afero from 1.9.3 to 1.9.4 #955
• Bump github.com/zclconf/go-cty from 1.12.1 to 1.13.0 by @dependabot in Bump github.com/zclconf/go-cty from 1.12.1 to 1.13.0 #954
• Allow VM instance name to include prefix and deployment name by @nick-stroud in Allow VM instance name to include prefix and deployment name #949
• Use HTCondor Python bindings in autoscaler by @tpdownes in Use HTCondor Python bindings in autoscaler #951
• Update hcls Lysozyme example to include visualization instructions by @nick-stroud in Update hcls Lysozyme example to include visualization instructions #958
• Unify validatorConfig check logic, reduce code duplication by @mr0re1 in Unify validatorConfig check logic, reduce code duplication #952
• Add warnings about tutorial cost and left behind resources by @nick-stroud in Add warnings about tutorial cost and left behind resources #957
• Lysozyme example update by @vjb-collab in Lysozyme example update #956
• Do not populate subnetwork_project as it is populated from subnetwork_self_link by @nick-stroud in Do not populate subnetwork_project as it is populated from subnetwork_self_link #959
• Remove warning about schema changes by @mr0re1 in Remove warning about schema changes #960
• Re-order tasks in HTCondor configuration runner by @tpdownes in Re-order tasks in HTCondor configuration runner #961
• Mask sleep/suspend targets on chrome-remote-desktop to prevent shutdown by @nick-stroud in Mask sleep/suspend targets on chrome-remote-desktop to prevent shutdown #966
• Version update by @mr0re1 in Version update #988
• Release v1.14.0 by @mr0re1 in Release v1.14.0 #989

New Contributors

• @vjb-collab made their first contribution in Lysozyme example update #956

Full Changelog: v1.13.0...v1.14.0

---

This discussion was created from the release v1.14.0: HTCondor highly available, HCLS blueprint .