Bioinf pathway #1189
Bioinf pathway #1189
Conversation
|
@tefirman - can you take a look at this and see if there's anything else we should cover? |
There was a problem hiding this comment.
@laderast -- This is a great step by step intro to job submission, thanks for putting this together! No additional section suggestions, but added a few more hyperlinks that might help route people to more information.
Also tried building the site locally and the mermaid diagram isn't compiling, but maybe it's my machine, just wanted to flag just in case.
| The next thing to do is to move your files to either: | ||
|
|
||
| - `/temp` | ||
| - `/working` |
There was a problem hiding this comment.
I'm assuming these should be /hpc/temp and /fh/working?
|
|
||
| To move files, you can use [Motuz](https://sciwiki.fredhutch.org/compdemos/motuz/), which is the web app, to move files back and forth from | ||
| - your computer | ||
| - from other Fred Hutch storage (such as `/fast/`) |
There was a problem hiding this comment.
Same thing here, probably more clear as /fh/fast/
|
|
||
| [HutchGO is also available for transferring files](https://sciwiki.fredhutch.org/scicomputing/hutchgo_overview/), which uses an App called Globus Connect. | ||
|
|
||
| You can also use an SFTP (secure FTP) client such as Mountain Duck to transfer files as well. |
There was a problem hiding this comment.
Maybe link to the MountainDuck docs here: https://sciwiki.fredhutch.org/compdemos/Mountain-CyberDuck/
| To move files, you can use [Motuz](https://sciwiki.fredhutch.org/compdemos/motuz/), which is the web app, to move files back and forth from | ||
| - your computer | ||
| - from other Fred Hutch storage (such as `/fast/`) | ||
| - or from Fred Hutch's S3 bucket |
There was a problem hiding this comment.
| - or from Fred Hutch's S3 bucket | |
| - or from an AWS S3 bucket |
| At this point, we have two main paths to run the software: | ||
|
|
||
| 1. We can use the built in job scheduler, SLURM, to batch process our files. | ||
| 2. We can automate our batching using a WDL (Workflow Description Language) file and PROOF |
There was a problem hiding this comment.
Maybe add a hyperlink to the "Developing WDL Workflows" course and the PROOF intro documentation:
https://hutchdatascience.org/Developing_WDL_Workflows/
https://sciwiki.fredhutch.org/datademos/proof-how-to/
| - `/temp` | ||
| - `/working` | ||
|
|
||
| These file systems have the advantage of being directly connected to the cluster and are faster. They are meant to be working storage, so when you are done processing your files, you will need to transfer them back to either `fast/` or your own system. |
There was a problem hiding this comment.
Might be perfect to link to the Data Storage overview explaining the difference between all of the mounted storage options: https://sciwiki.fredhutch.org/scicomputing/store_overview/
No description provided.