Merge pull request #33 from pepkit/dev

0.6.0

Author: nsheff

.gitignore

@@ -84,4 +84,6 @@ peppy.egg-info/
 
 #ipynm checkpoints
 *ipynb_checkpoints*
-*.egg-info*
+*.egg-info*
+
+

docs/README.md

@@ -2,26 +2,25 @@
 
 [![PEP compatible](http://pepkit.github.io/img/PEP-compatible-green.svg)](http://pepkit.github.io)
 
-`geofetch` is a command-line tool that does two things when given one or more GEO/SRA accessions:
+`geofetch` is a command-line tool that downloads and organizes data and metadata from GEO and SRA. When given one or more GEO/SRA accessions, `geofetch` will:
 
-  - Downloads either raw or processed data from either GEO or SRA
+  - Download either raw or processed data from either GEO or SRA
   - Produces a standardized [PEP](http://pepkit.github.io) sample annotation sheet of public metadata. This makes it really easy to run [looper](https://pepkit.github.io/docs/looper/)-compatible pipelines on public datasets by handling data acquisition and metadata formatting and standardization for you.
 
 You can use it with the [sra_convert](http://github.com/pepkit/sra_convert) pipeline, a [pypiper](http://pypiper.readthedocs.io) pipeline that converts SRA files into BAM files.
 
+## Quick demo
 
-## Installing
+`geofetch` runs on the command line. This command will download the metadata for the given GSE number.
 
-```bash
-pip install geofetch
+```console
+geofetch -i GSE95654
 ```
 
-## Quick start
-
-Now, run it on the command line:
+You can add `--just-metadata` if you don't want to download the raw SRA files.
 
 ```console
-geofetch --help
+geofetch -i GSE95654 --just-metadata
 ```
 
-Next, check out the [usage](usage) reference, or for a detailed walkthrough, head on over to the [tutorial](tutorial).
+For more details, check out the [usage](usage.md) reference, [installation instructions](install.md), or head on over to the [tutorial](tutorial.md) for a detailed walkthrough.

docs/changelog.md

@@ -1,5 +1,12 @@
 # Changelog
 
+## [Unreleased] 
+
+## [0.6.0] -- 2019-06-20
+- Fixed a bug with specifying a processed data output folder
+- Added a pre-check and warning message for `prefetch` command 
+
+
 ## [0.5.0] -- 2019-05-09
 
 - `geofetch` will now re-try a failed prefetch 3 times and warn if unsuccessful.

docs/file-specification.md

@@ -0,0 +1,20 @@
+#  How to specify samples to download
+
+The command-line interface provides a way to give GSE or SRA accession IDs. By default, `geofetch` will download all the samples it can find in the accession you give it. What if you want to restrict the download to just a few samples? Or what if you want to combine samples from multiple accessions? If you want more control, either because you have multiple accessions or you want to specify a subset of samples, then you can use the *file-based sample specification*, in which you provide `geofetch` with a file listing your GSE/GSM accessions.
+
+## The file-based sample specification
+
+
+Create a file with 3 columns that correspond to `GSE`, `GSM`, and `Sample_name. You may mix 1, 2, and 3 column lines in the file. An example input file could look like this:
+
+```console
+GSE123  GSM#### Sample1
+GSE123  GSM#### Sample2
+GSE123  GSM####
+GSE456
+```
+
+By default, `geofetch` will download all the samples in every included accession, but you can limit this by adding a second column with **GSM accessions** (which specify individual samples with a **GSE dataset**). If the second column is included, a third column may also be included and will be used
+as the sample_name; otherwise, the sample will be named according to the GEO Sample_title field. Any columns after the third will be ignored.
+
+This will download 3 particular GSM experiments from GSE123, and everything from GSE456. It will name the first two samples Sample1 and Sample2, and the third, plus any from GSE456, will have names according to GEO metadata.

docs/howto-limit.md

@@ -0,0 +1,37 @@
+# Installing geofetch
+
+## Prerequisites
+
+You must have the [sratoolkit from NCBI](https://www.ncbi.nlm.nih.gov/books/NBK158900/) installed, with the tools in your PATH. Once it's installed, you should check to make sure you can run `prefetch`. Also, make sure it's configured to store SRA files where you want them. For more information, see how to change sratools download location.
+
+## Setting data download location for `sratools`
+
+`geofetch` is using the [sratoolkit](https://trace.ncbi.nlm.nih.gov/Traces/sra/?view=toolkit_doc&f=std) to download raw data from SRA -- which means it's stuck with the [default path for downloading SRA data](http://databio.org/posts/downloading_sra_data.html), which is in your home directory. So before you run `geofetch`, make sure you have set up your download location to the correct place. In our group, we use a shared group environment variable called `${SRARAW}`, which points to a shared folder (`${DATA}/sra`) where the whole group has access to downloaded SRA data. You can point the `sratoolkit` (and therefore `geofetch`) to use that location with this one-time configuration code:
+
+```
+echo "/repository/user/main/public/root = \"$DATA\"" > ${HOME}/.ncbi/user-settings.mkfg
+```
+
+Now `sratoolkit` will download data into an `/sra` folder in `${DATA}`, which is what `${SRARAW}` points to.
+
+If you are getting an error that the `.ncbi` folder does not exist in your home directory, you can just make a folder `.ncbi` with an empty file `user-settings.mkfg` and follow the same command above.
+
+## Installing geofetch
+
+Releases are posted as [GitHub releases](https://github.com/pepkit/geofetch/releases), or you can install from PyPI using `pip`:
+
+```bash
+pip install geofetch
+```
+
+Confirm it was successful by running it on the command line:
+
+```console
+geofetch --help
+```
+
+If the executable in not in your $PATH, append this to your `.bashrc` or `.profile` (or `.bash_profile` on macOS):
+
+```
+export PATH=~/.local/bin:$PATH
+```

docs/howto.md

@@ -0,0 +1,19 @@
+# Metadata output
+
+For each GSE input accession (ACC), `geofetch` produces:
+
+- GSE_ACC####.soft a SOFT file (annotating the experiment itself)
+- GSM_ACC####.soft a SOFT file (annotating the samples within the experiment)
+- SRA_ACC####.soft a CSV file (annotating each SRA Run, retrieved from GSE->GSM->SRA)
+
+In addition, a single combined metadata file ("annoComb") for the whole input,
+including SRA and GSM annotations for each sample. Here, "combined" means that it will have
+rows for every sample in every GSE included in your input. So if you just gave a single GSE,
+then the combined file is the same as the GSE file. If any "merged" samples exist
+(samples for which there are multiple SRR Runs for a single SRX `Experiment`), the
+script will also produce a merge table CSV file with the relationships between
+SRX and SRR.
+
+The way this works: Starting from a GSE, select a subset of samples (GSM Accessions) provided, 
+and then obtain the SRX identifier for each of these from GEO. Now, query SRA for these SRX 
+accessions and get any associated SRR accessions. Finally, download all of these SRR data files.

docs/install.md

@@ -1,12 +1,8 @@
 # <img src="../img/geofetch_logo.svg" class="img-header">  tutorial
 
-## Prerequisites
-
-You must have the `sratoolkit` from NCBI installed, with tools in your `PATH` (check to make sure you can run `prefetch`). Make sure it's configured to store `sra` files where you want them. For more information, see [how to change sratools download location](howto-location.md).
-
 ## Download SRA data using `geofetch`
 
-To see full options, see the help menu with:
+Before starting, make sure you've followed the [installation instructions](install.md). To see your options, display the help menu:
 
 ```console
 geofetch -h
@@ -24,11 +20,17 @@ This will do 3 things:
 2. produce a sample annotation sheet, `PROJECT_NAME_annotation.csv`, in your metadata folder
 3. produce a project configuration file, `PROJECT_NAME_config.yaml`, in your metadata folder.
 
+Complete details about geofetch outputs is cataloged in the [metadata outputs reference](metadata_output.md).
 
 ## Finalize the project config and sample annotation
 
 That's basically it! `Geofetch` will have produced a general-purpose PEP for you, but you'll need to modify it for whatever purpose you have. For example, one common thing is to link to the pipeline you want to use by adding a `pipeline_interface` to the project config file. You may also need to adjust the `sample_annotation` file to make sure you have the right column names and attributes needed by the pipeline you're using. GEO submitters are notoriously bad at getting the metadata correct.
 
+## Selecting samples to download.
+
+By default, `geofetch` downloads all the data for one accession of interest. If you need more fine-grained control, either because you have multiple accessions or you need a subset of samples within them, you can use the [file-based sample specification](file-specification.md).
+
+
 ## A few real-world examples
 
 ```console

docs/metadata_output.md

@@ -1,2 +1 @@
-__version__ = "0.5.0"
-
+__version__ = "0.6.0"