Basics

Every experiment has its own directory under exps. Each experiment will have its source code under a subdir called exp. For example, the experiment for replicating the MLP from the eLife paper is called e2_mlp. Its source code will be located in exps/e2_mlp/exp. Results generated by running the experiment will be located at subdir called results.ign. For example, results for the e2_mlp experiment will be located under exps/e2_mlp/results.ign. Some experiments will have a third subdir under their directory called script. This usually contains post-scripts, i.e scripts for post-processing the results.

Singularity

Dependencies of the experiments for both Python and R are managed by a singularity container. The definition of the container is deeplearn.def. The singularity container can be built using the script build_singularity.sh. This script will produce a singularity container in the root dir with the name deeplearn.sif. Running any experiment depends on this image.

Running Experiments

For running an experiment without using SLURM (i.e on a workstation, not on a cluster), the singularity_python_run.sh script is used. The first command-line argument for the script is the entry point of the experiment. This is usually a file called main.py in the exp subdir. For example, to run the e2_mlp experiment

./singularity_python_run.sh exps/e2_mlp/exp/main.py

Any arguments specified after the first argument are passed to the experiment. For example, each experiment requires a mandatory argument called tag, which specifies the directory name to save the results under results.ign. For example, running:

./singularity_python_run.sh exps/e2_mlp/exp/main.py ++tag=my_exp

will run the e2_mlp experiment using the default hyperparameters and store the results at exps/e2_mlp/results.ign/my_exp.

For submitting jobs on the cluster using SLURM, the singularity_python_run.sh has to be replaced by sub_job.sh. The usage of sub_job.sh is almost identical to using singularity_python_run.sh.

Config and Hyperparameters

The configurations for each experiment is in exps/<exp_name>/exp/config.yaml. This file contains the default values for all the experimental hyperparameter. These configurations are managed by hydra. The hydra documentation provides information on the syntax to change/override any configuration when running an experiment. For example, to change the dropout value in the e2_mlp experiment to 0.5

./singularity_python_run.sh exps/e2_mlp/exp/main.py ++tag=my_exp model.dropout=0.5

Git Ignored Files/Dirs

Any directory that ends with .ign is ignored by git. This is mainly used to exclude data and result files and directories from the source control. For example, the results.ign directory used to store experimental results is ignored by git. Also, any file name containing .ign. will be excluded from version control. This is mainly used for temporary test scripts that do not need to be tracked by the source control. For example, a bash script with the name test.ign.sh will not be tracked by git.

Running R Scripts

Similar to python, R scripts can be run using singularity_r_run.sh. This follows a similar syntax to singularity_python_run.sh. The first argument is the R script to be run. Further arguments are passed to the R script itself.

singularity_r_run.sh ./path/to/script.R <arguments to R script>

Example

To run experiment e2_mlp on an HPC using SLURM with the following configurations:

• 2 hidden layers, each 512 neurons
• Nonlinearity LeakyReLU
• Instance Normalization layer
• Dropout 0.5

the follwoing command is used

./sub_job.sh exps/e2_mlp/exp/main.py ++tag=my_exp \
    model.hidden_layers=[512,512] \
    model.activation.name=LeakyReLU \
    model.norm_layer.name=instance \
    model.dropout=0.5

After the experiment is finished, the results will be stored at e2_mlp/results.ign/my_exp.

License:

This project is licensed under the Creative Commons Attribution 4.0 International License. See the LICENSE file for details.