FOI: a solution for a "cluster in the cloud"

[yarikoptic]

https://github.com/ComputeCanada/magic_castle

The Digital Research Alliance of Canada provides HPC infrastructure and support to every academic research institution in Canada. The Alliance uses CVMFS, a software distribution system developed at CERN, to make its research software stack available on its HPC clusters, and anywhere else with internet access. This enables replication of the user experience outside of The Alliance physical infrastructure.

From these new possibilities emerged an open-source software project named Magic Castle, which aims to recreate the HPC user experience in public clouds. Magic Castle uses the open-source software Terraform and HashiCorp Language (HCL) to define the virtual machines, volumes, and networks that are required to replicate a virtual HPC infrastructure. The infrastructure definition is packaged as a Terraform module that users can customize as they require. After deployment, the user is provided with a complete HPC cluster software environment including a Slurm scheduler, a Globus Endpoint, JupyterHub, LDAP, DNS, and over 3000 research software applications compiled by experts with EasyBuild. Magic Castle is compatible with AWS, Microsoft Azure, Google Cloud, OpenStack, and OVH.