Storage docs #121
Storage docs #121
Conversation
|
preview available: https://docs.tds.cscs.ch/121 |
1 similar comment
|
preview available: https://docs.tds.cscs.ch/121 |
|
preview available: https://docs.tds.cscs.ch/121 |
|
preview available: https://docs.tds.cscs.ch/121 |
|
preview available: https://docs.tds.cscs.ch/121 |
|
preview available: https://docs.tds.cscs.ch/121 |
|
preview available: https://docs.tds.cscs.ch/121 |
|
preview available: https://docs.tds.cscs.ch/121 |
|
preview available: https://docs.tds.cscs.ch/121 |
… storage-refactor
|
preview available: https://docs.tds.cscs.ch/121 |
There was a problem hiding this comment.
Superficial nitpicks. I hope someone who knows a bit more about filesystems can do a deeper review.
Potential restructuring idea already mentioned to @bcumming offline: Add separate sections about filesystem types, i.e. Lustre and VAST, and link to those from each of the filesystem sections. This means that one can avoid repeating e.g. warnings about performance degradation when the fileystem is close to full. It may become more complicated to navigate though, but worth considering.
| | | [backups][ref-storage-backups] | [snapshot][ref-storage-snapshots] | [cleanup][ref-storage-cleanup] | access | | ||
| | --------- | ---------- | ---------- | ----------- | --------- | | ||
| | [Home][ref-storage-home] | yes | yes | no | user | | ||
| | [Scratch][ref-storage-scratch]| no | no | yes | user | |
There was a problem hiding this comment.
Scratch default access is as far as I know user and primary group, so access should be primary group?
There was a problem hiding this comment.
The point here is that every user gets a scratch path in /<scratch>/$USER.
/capstor/scratch contains a single path cscs, which contains all users... from what I can tell.
| ```console | ||
| $ ls $HOME/.snapshot | ||
| big_catalog_2025-05-21_08_49_34_UTC | ||
| big_catalog_2025-05-21_09_19_34_UTC |
There was a problem hiding this comment.
What is big_catalog_*? It is unclear why it is there, and what to expect from it.
There was a problem hiding this comment.
No idea. Maybe the Storage guys can chip in with an explanation.
There was a problem hiding this comment.
We are investigating - it looks like it is an "implementation artifact" of the VAST file system.
We have reached out to them for clarification.
I will add it to the docs when they reply (and if the information is useful)
| ## Common Questions | ||
|
|
||
| ??? question "My files are gone, but the directories are still there" | ||
| When the [cleanup policy][ref-storage-cleanup] is applied on LUSTRE file systems, the files are removed, but the directories remain. |
There was a problem hiding this comment.
Yes, you describe the observation, but you do not provide the reason for this behaviour. Why is it a good idea to keep the directories, and not the files?
There was a problem hiding this comment.
I have no idea why we keep the directories - and I don't think explaining the why changes the observation (which is strange the first time you see it)
If I had to guess, it looks like the following is run:
find /capstor/scratch/cscs/$username -type f -atime +30 -delete
Co-authored-by: Mikael Simberg <[email protected]>
|
preview available: https://docs.tds.cscs.ch/121 |
Co-authored-by: Mikael Simberg <[email protected]>
|
preview available: https://docs.tds.cscs.ch/121 |
Co-authored-by: Mikael Simberg <[email protected]>
Co-authored-by: Mikael Simberg <[email protected]>
|
preview available: https://docs.tds.cscs.ch/121 |
2 similar comments
|
preview available: https://docs.tds.cscs.ch/121 |
|
preview available: https://docs.tds.cscs.ch/121 |
|
preview available: https://docs.tds.cscs.ch/121 |
|
preview available: https://docs.tds.cscs.ch/121 |
|
preview available: https://docs.tds.cscs.ch/121 |
|
preview available: https://docs.tds.cscs.ch/121 |
|
preview available: https://docs.tds.cscs.ch/121 |
|
preview available: https://docs.tds.cscs.ch/121 |
This is a draft of the new storage documentation.
do not merge
https://docs.tds.cscs.ch/121/storage/filesystems/