v1.19.0: ghpc destroy command, automatic ssh configuration, and Ramble integration

[rohitramu]

Key New Features

• New destroy command that automates deletion of all infrastructure from a deployment
• New ramble-execute module. Example blueprint: ramble.yaml.
• Automated SSH configuration using startup-script module with configure_ssh_host_patterns setting.

Module Improvements

• ramble-setup Made the module idempotent.
• Blueprint labels are now added to all resources in these modules:
  • gke-job-template
  • pbspro-preinstall
  • monitoring/dashboard
• packer/custom-image: Remove temporary users from the final image.
• project/service-account: Simplified Service Account usage.
• startup-script: Enable custom service accounts with startup-script
• gke-cluster: Exposed Container Storage Interface drivers addons for several different GKE storage types.
• Eliminated the need to activate a Python virtual environment to run Ansible.

Improvements

• Add support for indexing to "simple blueprint expressions"
• Added community wrapper blueprint for LLNL flux-framework example

Version updates

• Intel DAOS from 0.3.0 to 0.4.0:
  • hpc-slurm-daos.yaml: Server update
  • pfs-daos.yaml: Client update
• Upgraded Terraform provider from 4.63.1 to 4.65.2
• Upgraded Spack default version from 0.19.0 to 0.20.0
• Update to slurm-gcp 5.7.3
  • Allow metadata key slurmd_feature to initiate dynamic node setup.
  • Disable TreeWidth when dynamic nodes are configured.
  • Fix NVIDIA driver install after kernel upgrade for rocky-linux-8.

What's Changed

• Add integration tests runs for release-candidate brunch by @mr0re1 in Add integration tests runs for release-candidate brunch #1335
• Non-exclusive debug partition for hpc-slurm by @cboneti in Non-exclusive debug partition for hpc-slurm #1345
• Reword module descriptions so that they fit on single line by @nick-stroud in Reword module descriptions so that they fit on single line #1343
• Improve instance ID printout in tests by @tpdownes in Improve instance ID printout in tests #1346
• Add hpc-enterprise-slurm integration test by @mr0re1 in Add hpc-enterprise-slurm integration test #1331
• Add pre-commit to check for ghpc_module label by @nick-stroud in Add pre-commit to check for ghpc_module label #1344
• Add example for the lustre file system. by @rohitramu in Add example for the lustre file system. #1348
• Fix label value validation. by @rohitramu in Fix label value validation. #1349
• making hpc-slurm-ubuntu debug partition non-exclusive by @cboneti in making hpc-slurm-ubuntu debug partition non-exclusive #1350
• Merge main into develop by @cboneti in Merge main into develop #1358
• Change GCP API packages cloud.google.com/go>google.golang.org/api by @mr0re1 in Change GCP API packages cloud.google.com/go>google.golang.org/api #1356
• Upgrading terraform provider to 4.65.2 by @cboneti in Upgrading terraform provider to 4.65.2 #1359
• Add a ramble execute module by @douglasjacobsen in Add a ramble execute module #1310
• Disable release tests by @mr0re1 in Disable release tests #1361
• DAOSGCP-175 Updates for google-cloud-daos v0.4.0 by @mark-olson in DAOSGCP-175 Updates for google-cloud-daos v0.4.0 #1351
• Bump google.golang.org/api from 0.122.0 to 0.123.0 by @dependabot in Bump google.golang.org/api from 0.122.0 to 0.123.0 #1364
• Simplify adoption of Spack build caches in Google Cloud Storage by @tpdownes in Simplify adoption of Spack build caches in Google Cloud Storage #1352
• Remove modulereader.ModuleFS use sourcereader.ModuleFS instead. by @mr0re1 in Remove modulereader.ModuleFS use sourcereader.ModuleFS instead. #1365
• Improve test coverage by @tpdownes in Improve test coverage #1368
• Bump github.com/cloudflare/circl from 1.1.0 to 1.3.3 by @dependabot in Bump github.com/cloudflare/circl from 1.1.0 to 1.3.3 #1372
• Update Django to 4.1.9 to address CVE-2023-31047 by @tpdownes in Update Django to 4.1.9 to address CVE-2023-31047 #1370
• Address CVE-2023-32681 by upgrading requests by @tpdownes in Address CVE-2023-32681 by upgrading requests #1371
• Add test that all file-system mods output network_storage by @mr0re1 in Add test that all file-system mods output network_storage #1373
• Expose csi driver addons in gke-cluster by @nick-stroud in Expose csi driver addons in gke-cluster #1374
• Eliminate need to activate virtual environment to run Ansible by @tpdownes in Eliminate need to activate virtual environment to run Ansible #1353
• Update spack default version to v0.20.0 by @saltysoup in Update spack default version to v0.20.0 #1367
• Add all Ansible binaries to default PATH by @tpdownes in Add all Ansible binaries to default PATH #1379
• Fix batch mpi example by @tpdownes in Fix batch mpi example #1380
• Update ramble-setup module to be idempotent by @douglasjacobsen in Update ramble-setup module to be idempotent #1375
• Ensure that all modules take labels if they create resources by @rohitramu in Ensure that all modules take labels if they create resources #1362
• Add support for indexing to "simple blueprint expressions" by @mr0re1 in Add support for indexing to "simple blueprint expressions" #1377
• Remove omnia dependencies from GHPC virtual environment by @tpdownes in Remove omnia dependencies from GHPC virtual environment #1381
• Add validation for deprecated input variables by @rohitramu in Add validation for deprecated input variables #1390
• Add destroy command for deployments by @tpdownes in Add destroy command for deployments #1382
• Bump google.golang.org/api from 0.123.0 to 0.124.0 by @dependabot in Bump google.golang.org/api from 0.123.0 to 0.124.0 #1385
• Disable auto_activate_base in conda examples by @tpdownes in Disable auto_activate_base in conda examples #1392
• Ensure local users are not present in final image by @tpdownes in Ensure local users are not present in final image #1393
• add config-ssh as a startup script option by @cboneti in add config-ssh as a startup script option #1378
• Bump github.com/zclconf/go-cty from 1.13.1 to 1.13.2 by @dependabot in Bump github.com/zclconf/go-cty from 1.13.1 to 1.13.2 #1384
• Bump tomlkit from 0.11.7 to 0.11.8 in /community/front-end/ofe by @dependabot in Bump tomlkit from 0.11.7 to 0.11.8 in /community/front-end/ofe #1395
• Bump google-cloud-storage from 2.8.0 to 2.9.0 in /community/front-end/ofe by @dependabot in Bump google-cloud-storage from 2.8.0 to 2.9.0 in /community/front-end/ofe #1396
• Bump github.com/go-git/go-git/v5 from 5.6.1 to 5.7.0 by @dependabot in Bump github.com/go-git/go-git/v5 from 5.6.1 to 5.7.0 #1383
• Bump cachetools from 5.3.0 to 5.3.1 in /community/front-end/ofe by @dependabot in Bump cachetools from 5.3.0 to 5.3.1 in /community/front-end/ofe #1397
• Bump typing-inspect from 0.8.0 to 0.9.0 in /community/front-end/ofe by @dependabot in Bump typing-inspect from 0.8.0 to 0.9.0 in /community/front-end/ofe #1398
• Fix typos by @tpdownes in Fix typos #1402
• Bump urllib3 from 1.26.15 to 2.0.2 in /community/front-end/ofe by @dependabot in Bump urllib3 from 1.26.15 to 2.0.2 in /community/front-end/ofe #1399
• Add community wrapper blueprint for LLNL flux-framework by @wkharold in Add community wrapper blueprint for LLNL flux-framework #1369
• Enable remote git Packer by @tpdownes in Enable remote git Packer #1401
• Update READMEs about "literal variables" by @mr0re1 in Update READMEs about "literal variables" #1391
• Bump google.golang.org/api from 0.124.0 to 0.125.0 by @dependabot in Bump google.golang.org/api from 0.124.0 to 0.125.0 #1409
• Simplify service-account module by @tpdownes in Simplify service-account module #1400
• Bump github.com/hashicorp/hcl/v2 from 2.16.2 to 2.17.0 by @dependabot in Bump github.com/hashicorp/hcl/v2 from 2.16.2 to 2.17.0 #1408
• Enable custom service accounts with startup-script by @tpdownes in Enable custom service accounts with startup-script #1404
• Update to slurm-gcp 5.7.3 by @SkylerMalinowski in Update to slurm-gcp 5.7.3 #1410
• Validate a blueprint's top-level "labels" variable by @rohitramu in Validate a blueprint's top-level "labels" variable #1394
• Update vm-instance instructions to handle the case that there are zero instances by @nick-stroud in Update vm-instance instructions to handle the case that there are zero instances #1413
• Identify G2 family as having accelerators by @tpdownes in Identify G2 family as having accelerators #1415
• Fix CRD Slurm example by @nick-stroud in Fix CRD Slurm example #1414
• updating examples and documentation to point to newer SchedMD images by @cboneti in updating examples and documentation to point to newer SchedMD images #1412
• Move hpc-enterprise-slurm test to avoid stockouts by @nick-stroud in Move hpc-enterprise-slurm test to avoid stockouts #1416
• Fix nfs-server attached disk mounting. by @mr0re1 in Fix nfs-server attached disk mounting. #1406
• Do not swallow error during expanded.yaml write by @mr0re1 in Do not swallow error during expanded.yaml write #1417
• Update Open Front End usage of service-account module by @tpdownes in Update Open Front End usage of service-account module #1420
• Add flux example to examples/README by @mr0re1 in Add flux example to examples/README #1407
• Update the Batch example to use the HPC variant of the Rocky 8 image by @nick-stroud in Update the Batch example to use the HPC variant of the Rocky 8 image #1419
• Fix Ansible installation upon re-run by @tpdownes in Fix Ansible installation upon re-run #1433
• Merge v1.18.1 back to release-candidate by @nick-stroud in Merge v1.18.1 back to release-candidate #1437
• Remove OFE double-usage of deployment name in service account name by @tpdownes in Remove OFE double-usage of deployment name in service account name #1451
• Update all references to golang base image by @rohitramu in Update all references to golang base image #1455
• Update version to 1.19.0 by @rohitramu in Update version to 1.19.0 #1456
• Release v1.19.0 by @rohitramu in Release v1.19.0 #1457

New Contributors

• @wkharold made their first contribution in Add community wrapper blueprint for LLNL flux-framework #1369

Full Changelog: v1.18.1...v1.19.0

---

This discussion was created from the release v1.19.0.