Training on Compute canada

[plbenveniste]

This issue is opened to detail the steps to use Compute Canada to train model.

[plbenveniste]

Here are my notes from trying to use Compute canada

Before starting, here are some useful links:

• Compute canada FAQ: https://docs.alliancecan.ca/wiki/Frequently_Asked_Questions_about_the_CCDB/fr
• Getting started documentation: https://docs.alliancecan.ca/wiki/Getting_started

First steps:

1. Check to have an account on Compute Canada which is still active: https://ccdb.alliancecan.ca/
   What it should look like:

You can also check which machines are available for you:

2. You can try connecting to one of the clusters: ssh USERNAME@cedar.alliancecan.ca
   You will get the following message: Access is not allowed because you are not enrolled in Duo.

3. To set-up multi-factor authentification: use this tutorial: https://docs.alliancecan.ca/wiki/Multifactor_authentication#Registering_multiple_factors
   I used the mobile Duo app, as it is recommended.

4. For TamIA, an additional step is required: To gain access to TamIA please go to the link below and select "I request access": https://ccdb.alliancecan.ca/me/access_services. More details can be found in: https://docs.alliancecan.ca/wiki/TamIA/en.

5. You can now try to ssh again and it should work.
   NB: You cannot log into Niagara this way (look at this if you want to use it: https://docs.alliancecan.ca/wiki/Getting_started/fr#Quels_sont_les_syst%C3%A8mes_disponibles?:~:text=Niagara%20est%20une%20grappe%20con%C3%A7ue%20pour%20les%20t%C3%A2ches%20parall%C3%A8les%20intensives%20(plus%20de%201000%20c%C5%93urs).%20Pour%20savoir%20comment%20y%20acc%C3%A9der%2C%20voyez%20les%20services%20disponibles. )

How to transfer data to compute canada:

• using scp or rsync: https://docs.alliancecan.ca/wiki/Transferring_data/fr

How to run a job in compute canada:

• useful documentation here: Add template scripts for training on compute Canada utilities#55
• Another example: https://github.com/SomeoneInParticular/dcm-classic-ml/blob/master/slurm_scripts/_utils/kost_slurm_header.sl

Other stuff:

• Currently SCT doesn't work on Compute Canada because of conda
• Compute Canada doesn’t like conda -> use pip instead
• To use Wandb: https://docs.alliancecan.ca/wiki/Weights_%26_Biases_(wandb). It is only available on certain nodes.
• Calgary’s SLURM documentation is also pretty great: https://rcs.ucalgary.ca/index.php/ARC_Cluster_Guide#Prepare_job_scripts
• salloc: for quick jobs like making a python environment: it requires stuff to be done on the spot. (DO NOT ABUSE IT)
  Example: salloc --mem=1G -t 01:00:00 -p gpu-v100 --gres=gpu:1

[jcohenadad]

Currently SCT doesn't work on Compute Canada because of conda

Couldn't that be fixed by using a Singularity container?

[plbenveniste]

I have no experience with Singularity containers. I don't require the use of SCT on Compute Canada right now. I will look into it later on. Maybe @SomeoneInParticular has tried this?

[SomeoneInParticular]

I've not used containerization on any SLURM cluster before, unfortunately; I've always found it to be more trouble than its worth (namely, the "it works on my machine, lets ship my machine" approach is a good way to ensure your results are nigh impossible for other scientists to reproduce on their own).

I'll give it a spin, though, as this would be a much more straightforward means of getting SCT to run on Compute Canada. If it works, it'd also be pretty easy to add a GitHub Action to automatically generate SCT containers on release to streamline usage in this context; thoughts?

[jcohenadad]

Tagging @joshuacwnewton @mguaypaq

[joshuacwnewton]

Currently SCT doesn't work on Compute Canada because of conda

I remember that @sandrinebedard had gotten SCT installed on Compute Canada here:

• Clarify steps for installing SCT onto ComputeCanada Graham (requirements.txt conflicts) spinalcordtoolbox/spinalcordtoolbox#4593

Regarding Singularity containers, there was some previous discussion in the above issue (spinalcordtoolbox/spinalcordtoolbox#4593 (comment)).

@rohanbanerjee has also experimented with the "pip-based SCT installation" to install SCT on CC here:

• Installation on Compute Canada fails due to missing steps in experimental pip-based installation method spinalcordtoolbox/spinalcordtoolbox#4116 (comment)

[plbenveniste]

Discussion on using SCT in Compute Canada was transferred to this issue: spinalcordtoolbox/spinalcordtoolbox#4859

[plbenveniste]

Edit: conversation transferred to issue spinalcordtoolbox/spinalcordtoolbox#4593

[plbenveniste]

Data storing on Compute Canada:
In the documentation here they mention that data which uis not frequently shoudl be stored in the project folder while the data which is frequently modified should be stored in the scratch.
I used scp to transfer the data directly from moneta:

• scp dataset_2025-04-15_seed42.json USERNAME@tamia.alliancecan.ca:~/links/projects/aip-jcohen/plb/msd_data/
• scp -r Dataset902_msLesionAgnostic USERNAME@tamia.alliancecan.ca:~/links/projects/aip-jcohen/plb/nnUNet_experiments/nnUNet_raw

I had to change the conversion_dict.json files so that the path of the nnUNet_raw files corresponds to the path of the files on Compute Canada.

[plbenveniste]

I tried launching a training on TamIA, I had the following error message:

sbatch: error: This job exceeds the maximum walltime of 1 days on tamia.
sbatch: error: Please resubmit with a shorter walltime, or on a different cluster that allows longer walltimes.

This means trainings can't last longer than 1 day ?!
If that is the case, then I cannot use TamIA for training my models.

I tried launching a training using a max time of 1 day to see how far 1 day gets me: the script is here

[SomeoneInParticular]

This means trainings can't last longer than 1 day ?!

Yes, that seems to be the case:

Unfortunately, due to the high demand of GPUs, this isn't all that uncommon in compute clusters. You might want to look into "check-pointing" to get around this if you end up needing more than that. This website gives an excellent overview of how to do so: https://machinelearningmastery.com/managing-a-pytorch-training-process-with-checkpoints-and-early-stopping/.

Alternatively, you might be able to get away with splitting you sub-tasks (data pre-processing, inference etc.) into SLURM sub-scripts and chain them with --dependency to ensure they run in order (see an example of how this is done here). Would give you 24 hours for each step that way.

Either way, definitely not ideal...

[plbenveniste]

Okay! Thanks for the thorough response! I'll look into it.

[jcohenadad]

You could also contact compute canada and ask if you could have some exceptions-- they do that in some cases