Create a new community scheduler module for Slinky (Slurm on Kubernetes)

[ndebuhr]

Added a community Slinky (Slurm-on-Kubernetes) module and example.

This PR introduces a new community module and an accompanying example blueprint to enable the deployment of Slinky, a Slurm workload manager implementation on Kubernetes.

The new module handles the installation of necessary components (Cert Manager, Slinky Operator, and Slurm Cluster) via their respective Helm charts onto a target GKE cluster. It allows for customization through Helm values overrides and supports best practice node affinities for Slinky system components and nodepools.

The example blueprint provides a practical example demonstrating how to deploy a Slinky cluster with both a debug node pool and an H3 HPC node pool. The example also includes configuration for monitoring the Slurm cluster using Google Managed Prometheus (GMP) via a PodMonitoring resource.

Design notes:

1. The Helm-based approach closely follows the documented "quickstart" Slinky installation.
2. A lightweight, GCP-native metrics/monitoring system is adopted by default, rather than the Slinky-documented cluster-local Kube Prometheus Stack.
3. As of 2025/04/03, the quickstart documentation suggests a v0.1.0 installation, but v0.2.0 is implemented as the module default given its improved stability.
4. The example adopts a zonal implementation, which is common to avoid inter-zonal networking charges and to minimize latency. This includes skipping the GKE Cluster "system node pool", to avoid potential zone-based volume node affinity conflicts (or the need to use custom regional storage classes).
5. The module is unopinionated on nodeset node affinities, given the need for compute fabric flexibility. However, a module dependency is used for the Slurm system component node affinities, given less need for flexibility and to ensure a node pool remains up during Helm uninstalls (gcluster destroys).
6. Terraform's concat() is used for Helm values, given nested object type safety constraints.

The new module and example have been manually tested, pretty extensively, using Terraform v1.11.3 and Packer v1.12.0.

CC @samskillman as FYI, given additional context

Submission Checklist

NOTE: Community submissions can take up to 2 weeks to be reviewed.

Please take the following actions before submitting this pull request.

• ✅ Fork your PR branch from the Toolkit "develop" branch (not main)
• ✅ Test all changes with pre-commit in a local branch #
• ✅ Confirm that "make tests" passes all tests
• ✅ Add or modify unit tests to cover code changes
• ✅ Ensure that unit test coverage remains above 80%
• ✅ Update all applicable documentation
• ✅ Follow Cluster Toolkit Contribution guidelines #

[ndebuhr]

@ighosh98 This is ready for review. Keep me posted here (or via internal chat messages) on any additional questions/concerns/steps.

[ighosh98]

Could you please add an integration test for this module in the PR?

[ighosh98]

The integration test won't run in the current definition. Have added explanation on how to set it up. We can connect if needed.

Rest of the changes look good to me.

[samskillman]

Overall looks good. I have a few minor comments below. However, when testing I'm running into several issues / questions:

1. Why aren't we creating a login service like the quickstart (https://github.com/SlinkyProject/slurm-operator/blob/main/docs/quickstart.md).

2. I'm unable to launch a job on the h3 partition that spins up even though I see an h3 node in the gke nodepool.

3. When I run sbatch --wrap "hostname", I don't see output logfiles in the expected location in /tmp/slurm-{job id}.out/err.

[ndebuhr]

Thanks @samskillman! Will rework and retest a number of things here, based on Friday's Slinky v0.2.1 release (which FYI is also related to "Why aren't we creating a login service like the quickstart" and "Consider adding a note on how to connect to the cluster.", as the quickstart didn't have a login service in v0.2.0, and the documented path to "connect" was kubectl exec. Glad to see some improvements in v0.2.1 and happy to integrate. Will ping you when this is ready for another round of review

[ndebuhr]

@samskillman Reworked, retested, and ready for your review. Some high-level notes:

• v0.2.1 did indeed fix the v0.2.0 Slurm Exporter issue - removed the workaround.
• v0.2.1 does not include the new login node and connection system. I'll keep an eye out and submit a PR as soon as v0.2.2 or v0.3.0 is released. Even if we wanted to (which we don't 🙂), building on top of Slinky main or some commit hash would be very messy, as we'd have to circumvent the official publishings in the OCI chart repo.
• The blueprint previously used a default h3 nodeset of 0 replicas - assuming folks would use the debug nodeset for development and then scale out the h3 nodeset as needed. However, as your issue with the h3 partition highlights, this was not intuitive or well documented. Both the debug nodeset and h3 nodeset have been updated to a default of 2 replicas (both can be overridden with blueprint variables), and the documentation is clarified/expanded.

Tested your sbatch --wrap "hostname" command on debug and h3 partitions. With the aforementioned 0→2 h3 nodeset replica change, they both work fine - just need to check /tmp on the nodeset replica that ran the job (e.g., via kubectl exec, since there is no RWX shared volume between the nodeset replicas and the controller pod.

[ndebuhr]

After some discussions on this, I'll wait until Slinky v0.3.0 is released, integrate the latest, and re-ping for review. There's some good stuff in v0.3.0 (especially a login node and RWX mount support) that will significantly improve usability, and it seems like the release will come soon.