Skip to content

Commit 274e3fc

Browse files
committed
Fix Information_storage.md, remove anchor
Signed-off-by: Fabrice Normandin <[email protected]>
1 parent 3dd3126 commit 274e3fc

File tree

1 file changed

+18
-20
lines changed

1 file changed

+18
-20
lines changed

docs/Information_storage.md

Lines changed: 18 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,19 @@
1-
<a id="milacluster_storage"></a>
2-
3-
### Storage
4-
5-
| Path | Performance | Usage | Quota (Space/Files) | Backup | Auto-cleanup |
6-
|------|-------------|-------|---------------------|--------|--------------|
7-
| `/network/datasets/` | High | Curated raw datasets (read only) ||||
8-
| `/network/weights/` | High | Curated models weights (read only) ||||
9-
| `$HOME` or `/home/mila/<u>/<username>/` | Low | Personal user space; specific libraries, code, binaries | 100GB/1000K | Daily | no |
10-
| `$SCRATCH` or `/network/scratch/<u>/<username>/` | High | Temporary job results; processed datasets; optimized for small files | 5TB/no | no | 90 days |
11-
| `$SLURM_TMPDIR` | Highest | High speed disk for temporary job results | no/no | no | at job end |
12-
| `/network/projects/<groupname>/` | Fair | Shared space for collaboration; long-term project storage | 1TB/1000K | Daily | no |
13-
| `$ARCHIVE` or `/network/archive/<u>/<username>/` | Low | Long-term personal storage | 5TB | no | no |
1+
# Storage
2+
3+
| Path | Performance | Usage | Quota (Space/Files) | Backup | Auto-cleanup |
4+
| ------------------------------------------------ | ----------- | -------------------------------------------------------------------- | ------------------- | ------ | ------------ |
5+
| `/network/datasets/` | High | Curated raw datasets (read only) ||||
6+
| `/network/weights/` | High | Curated models weights (read only) ||||
7+
| `$HOME` or `/home/mila/<u>/<username>/` | Low | Personal user space; specific libraries, code, binaries | 100GB/1000K | Daily | no |
8+
| `$SCRATCH` or `/network/scratch/<u>/<username>/` | High | Temporary job results; processed datasets; optimized for small files | 5TB/no | no | 90 days |
9+
| `$SLURM_TMPDIR` | Highest | High speed disk for temporary job results | no/no | no | at job end |
10+
| `/network/projects/<groupname>/` | Fair | Shared space for collaboration; long-term project storage | 1TB/1000K | Daily | no |
11+
| `$ARCHIVE` or `/network/archive/<u>/<username>/` | Low | Long-term personal storage | 5TB | no | no |
1412

1513
!!! note
1614
The `$HOME` file system is backed up once a day. For any file restoration request, file a request to [Mila's IT support](https://it-support.mila.quebec) with the path to the file or directory to restore, with the required date.
1715

18-
#### $HOME
16+
## $HOME
1917

2018
`$HOME` is appropriate for codes and libraries which are small and read once,
2119
as well as the experimental results that would be needed at a later time (e.g.
@@ -29,7 +27,7 @@ million per user. The command to check the quota usage from a login node is:
2927
disk-quota
3028
```
3129

32-
#### $SCRATCH
30+
## $SCRATCH
3331

3432

3533
`$SCRATCH` can be used to store processed datasets, work in progress datasets
@@ -47,14 +45,14 @@ the quota usage from a login node is:
4745
disk-quota
4846
```
4947

50-
#### $SLURM_TMPDIR
48+
## $SLURM_TMPDIR
5149

5250

5351
`$SLURM_TMPDIR` points to the local disk of the node on which a job is
5452
running. It should be used to copy the data on the node at the beginning of the
5553
job and write intermediate checkpoints. This folder is cleared after each job.
5654

57-
#### projects
55+
## projects
5856

5957

6058
`projects` can be used for collaborative projects. It aims to ease the
@@ -67,7 +65,7 @@ of files (inodes). The limits for blocks and inodes are respectively 1TiB and
6765
!!! note
6866
It is possible to request higher quota limits if the project requires it. File a request to [Mila's IT support](https://it-support.mila.quebec).
6967

70-
#### $ARCHIVE
68+
## $ARCHIVE
7169

7270

7371
`$ARCHIVE` purpose is to store data other than datasets that has to be kept
@@ -96,7 +94,7 @@ df -h $ARCHIVE
9694
!!! note
9795
There is **NO** backup of this file system.
9896

99-
#### datasets
97+
## datasets
10098

10199

102100
`datasets` contains curated datasets to the benefit of the Mila community. To
@@ -120,7 +118,7 @@ command:
120118
ssh [CLUSTER_LOGIN] -C "projects/rrg-bengioy-ad/data/curated/list_datasets_cc.sh"
121119
```
122120

123-
#### weights
121+
## weights
124122

125123

126124
`weights` contains curated models weights to the benefit of the Mila

0 commit comments

Comments
 (0)