Merge pull request #63 from ScilifelabDataCentre/docs-on-queries

Add drafts for the user guides on queries

Author: brinkdp

docs/development/deployment.md

@@ -21,6 +21,10 @@ We have accounts on both [PyPI](https://pypi.org/) and [TestPyPI](https://test.p
 
 (Login credentials are in bitwarden, but if you use the GH actions below you don't need to use them.)
 
+## Considerations for Kubernetes deployment
+
+TODO: add some lessons learned from the k8s deployment, e.g. on resources needed for the celery worker.
+
 ### Good to know
 
 1. You cannot upload the same version of a package to PyPI or TestPyPI if that version already exists (even if you delete the existing package). So in such cases you need to bump the version number in the packages before uploading new versions.

docs/user-guides/how-to-create-efficient-divbase-queries.md

@@ -0,0 +1,7 @@
+# How to create efficient DivBase queries
+
+Split large VCF files into smaller files. For instance by chromosome. If the assembly does not have chromosome level contiguity, we suggest to store a range of scaffolds per file.
+
+divbase bcftools "pipes"
+
+If subsets based on variant range are included, run then first before any sample subsets. subsetting on samples requres inspecting and potentially updating every row in the VCF (N+1 problem); subsetting on samples will be faster on smaller datasets and therefore better to run after all row-based subsets have been done.

docs/user-guides/query-syntax.md

@@ -0,0 +1,3 @@
+# DivBase Query Syntax for VCF data
+
+TODO

docs/user-guides/quick-start.md

@@ -53,7 +53,7 @@ Add your project(s) to the configuration file:
 divbase-cli config add PROJECT_NAME --default
 ```
 
-By setting this as the default project, you won't need to specify the project name in future commands. To select a different project in any future command, you can add the `--project` flag to any command.
+By setting this as the default project, you won't need to specify the project name in future commands. To select a different project in any future command, you can add the `--project` flag to any command. Note that the rest of the example commands in this quick start guide will omit the `--project` flag for brevity.
 
 !!! note
     On the divbase website you can see your project's name(s) under the "Projects" tab after logging in.
@@ -89,26 +89,37 @@ divbase-cli files list
 
 ## Step 7: Upload sample metadata
 
+TODO It might make more sense to have run the dimensions update job before this if we are to use a pre-populated template file
+
 Sample metadata must be uploaded as follows:
 
 - In TSV format and be named "sample_metadata.tsv"
 - Must contain a column named "sample_id" which matches the sample IDs in your VCF files
 - The names and values of all other columns are optional.
 
-TODO - do we support numerical columns, including queries on them, i.e in range 31 - 50. etc...
+TODO update this after `sidecar-metadata.md` docs are done, there are changes planned for some details.
 
 ```bash
 divbase-cli files upload path/to/your/sample_metadata.tsv
 ```
 
 ## Step 8: Dimensions update
 
-Update the project dimensions after uploading your files:
+For DivBase to be able to efficiently handle the VCF files in the the project, some key information about each VCF files is fetched from the files. In DivBase, this is refered to as "VCF dimensions". These include for instance which samples and scaffolds that a VCF file contains.
+
+Update the project dimensions after uploading your files.
 
 ```bash
 divbase-cli dimensions update
 ```
 
+This submits a task to the DivBase task management system. The task will wait in a queue until the system is ready to work on it. Depending on the size of the VCF files, this make take a couple of minutes.
+
+!!! notes
+    1. Please note that it is not possible to run VCF queries in DivBase until the dimensions update task has finished. The reason for this is that the VCF queries use the dimensions data ensure that the queries are feasible and to know which VCF files from the project to process.
+
+    2. Please also note that the `divbase-cli dimensions update` command needs to be done every time a new VCF or a new version of a VCF file is uploaded.
+
 ## Step 9: Confirm dimensions update job completion
 
 Check the task history to confirm the dimensions update job has completed:
@@ -119,16 +130,52 @@ divbase-cli task-history user
 
 Once complete, you can run any queries on the uploaded data.
 
+It is possible to inspect the cached VCF dimensions data for the project at any time with the command:
+
+```bash
+divbase-cli dimensions show
+```
+
 ## Step 10: Run your queries
 
-Query your VCF data:
+There are three types of queries in DivBase:
+
+- Sample metadata query
+- VCF data query
+- Combined sample metadata and VCF data query
+
+!!! note
+    Queries are one of the more complex aspects of DivBase and therefore the user is encouraged to read the section on [Running Queries](running-queries.md) after reading this quick start.
+
+### Running sample metadata queries
+
+As an example, let's assume that user-defined sidecar sample metadata TSV file contains a custom column name `Area` and that `Northern Portugal` is one of the values in the column. To filter on all samples on that column and value:
+
+```bash
+divbase-cli query tsv "Area:Northern Portugal"
+```
+
+!!! note
+    Please see [Sidecar Metadata TSV files: creating and querying sample metadata files](sidecar-metadata.md) for more details on the syntax for writing sample metadata queries.
+
+### Running VCF data queries
+
+DivBase uses [`bcftools`](https://github.com/samtools/bcftools) to subset VCF data and therefore the query syntax is based on `bcftools view` syntax (as described in the [bcftools manual](https://samtools.github.io/bcftools/bcftools.html#view)).
+
+For instance, to subset all VCF files in the project on a chromosomal region in a scaffold named `21`:
+
+```bash
+divbase-cli query bcftools-pipe --command "view -r 21:15000000-25000000"
+```
+
+The VCF queries can be combined with sidecar sample metadata queries with `--tsv-filter` and the fixed expression `view -s SAMPLES` (where `SAMPLES` tells DivBase to use the results from the sidecar filtering as input for `bcftools view -s`). In this way, only the VCF files that fulfil the sample metadata query will be used in the `bcftools` subset commands. An example:
 
 ```bash
-# TODO - some simple examples to show what kind of queries can be run
+divbase-cli query bcftools-pipe --tsv-filter "Area:Northern Portugal" --command "view -s SAMPLES; view -r 21:15000000-25000000"
 ```
 
 !!! note
-    For more detailed information on running queries, refer to the [Running Queries Guide](running-queries.md).
+    DivBase only allows `bcftools view` in its query syntax and no other `bcftools` commands. The `merge`, `concat`, and `annotate` commands are used when processing a query, but should not be defined by the user.
 
 ## Step 11: Download any results files
 

docs/user-guides/running-queries.md

@@ -1,3 +1,53 @@
-# Running Queries
+# Running Queries: Overview
+
+DivBase allows users to submit queries to checkout sample metadata and VCF data from the DivBase project data store. There types of queries in DivBase are:
+
+- Sample metadata query
+- VCF data query
+- Combined sample metadata and VCF data query
+
+The sample metadata is a user defined, sidecar TSV (tab separated variables) file where users can add any custom metadata as columns. The VCF queries use `bcftools` to subset data from the files in the data store based on user defined filters. The VCF queries can be combined with a sample metadata query to allow users to filter both on data in the sidecar metadata and VCF files in a single query.
 
 TODO
+
+- [TO BE IMPLEMENTED] think about the `divbase-cli query` commands. it would make sense to use `sample-metadata` (perhaps with `tsv` as a short form) and `vcf`
+
+The system will use the latest version the the files for all queries.
+
+TODO COPIED OVER FROM QUICKSTART REIMPLEMENT
+
+DivBase only allows `bcftools view` in its query syntax and no other `bcftools` commands. The `merge`, `concat`, and `annotate` commands are used when processing a query, but should not be defined by the user.
+
+## Side car sample metadata queries
+
+For more details, see [Sidecar Metadata TSV files: creating and querying sample metadata files](sidecar-metadata.md).
+
+how to write the sample metadata TSV (+template)
+
+how to use dimensions show to get all samples in the project
+
+how to write query
+
+## Before any VCF queries: run the VCF dimensions command
+
+For more details see [VCF Dimensions caching](vcf-dimensions.md)
+
+For performance reasons and to ensure query feasibility, key metadata from the VCF files must first be cached in DivBase.
+For each VCF in the DivBase project, the system extracts file name, number and name of all samples, number and name of all scaffolds, number of variants,
+
+DivBase will use this whenever a user submits a query to the project. For instance, the user might make a sample metadata filtering that results in only certain samples. The system knows which file names each requested sample are located in, and will ensure that only those files will be transferred to the worker.
+
+example
+dimensions show
+
+## VCF queries
+
+See also [DivBase Query Syntax for VCF data](query-syntax.md), [How to create efficient DivBase queries](how-to-create-efficient-divbase-queries.md), and [Tutorial: Running a query on a public dataset](tutorial-query-on-public-data.md).
+
+can be run with or without sample metadata filtering
+
+for sample metadata linked VCF queries, it can be good to do a dry run first [TO BE IMPLEMENTED, at the moment it needs to be run seperatelly]
+
+how to write a query
+
+how to optimize a query (see separate markdown)

docs/user-guides/sidecar-metadata.md

@@ -0,0 +1,48 @@
+# Sidecar Metadata TSV files: creating and querying sample metadata files
+
+TODO
+
+- rationale: what is this and how can it be used
+
+## Creating a sidecar TSV for a DivBase project
+
+Note! there can be multiple TSVs in the same project and it is possible to call them for the queries with the `--metadata-tsv-name` flag.
+
+TODOs:
+
+- [TO BE IMPLEMENTED] consider changing the mandatory column name from `Sample_ID` to `Sample`
+- [TO BE IMPLEMENTED] CLI command to generate template (empty template and template with the samples from the DivBase project pre-filled). Pre-filling the template will require that dimensions update has been run
+- [TO BE IMPLEMENTED] what happens if a TSV does not contain all the samples in the DivBase project? There should probably be a warning, but not an error?
+- [TO BE IMPLEMENTED] what happens if a sample name is misspelled in the TSV? a warning? can this be checked against the dimensions show?
+
+- [TO BE IMPLEMENTED] what happens if a sample is duplicated in the file. what happens if the sample name is duplicated but not the values (diverging duplicate)?
+
+1 mandatory column: sample name.
+
+any other columns are optional and user defined
+
+it is possible to have more than one sidecar sample metadata file in each DivBase project.
+
+example
+
+```
+TODO add example here
+```
+
+## Query Syntax for sidecar metadata
+
+from `divbase-cli query tsv -h` docstring:
+String consisting of keys:values in the tsv file to filter on. The syntax is 'Key1:Value1,Value2;Key2:Value3,Value4', where the key are the column header names in the tsv, and values are the column values. Multiple values for a key are separated by commas, and multiple keys are separated by semicolons. When multple keys are provided, an intersect query will be performed. E.g. 'Area:West of Ireland,Northern Portugal;Sex:F'.
+
+- [TO BE IMPLEMENTED] filtering based on ranges (ints and maybe floats) and not just on strings, e.g. in range 31 - 50. etc...
+- [TO BE IMPLEMENTED] add more Set Operations: union, intersection, difference, symmetric difference. Be clear on the default behaviour
+
+## Trying out a query
+
+```bash
+divbase-cli query tsv "Area:Northern Portugal"
+```
+
+- [TO BE IMPLEMENTED] what to do if a query references a column that does not exist. E.g. `divbase-cli query tsv "Area:Northern Portugal"` when Area does not exist? This should probably give a warning and not just return nothing
+
+- [TO BE IMPLEMENTED] what to do if a query references a column value. E.g. `divbase-cli query tsv "Area:Northern Portugal"` when Northern Portugal does not exist in the column? This should probably also give a warning and not just return nothing, but nothing is a result here and not a syntax problem...

docs/user-guides/tutorial-query-on-public-data.md

@@ -0,0 +1,137 @@
+# Tutorial: Running a DivBase query on a public dataset
+
+This tutorial assumes that you have an account on DivBase and have a membership in a DivBase project with at least an EDIT role (i.e. can upload files and run queries).
+
+We will use a mouse (_Mus musculus_) data set availalbe on the European Nucleotide Archive: <https://www.ebi.ac.uk/ena/browser/view/ERZ022025>. It a 5.5 Gb VCF.gz file that contains 18 samples and 66,007,044 variants.
+
+## 1. Obtain the data and upload it to your DivBase project
+
+First we need download the files from EVA to our local computer. We can do that from the ENA homepage with a web browser, or from the terminal with a tool like `curl` or `wget`:
+
+```bash
+curl -o mgp.v3.snps.rsIDdbSNPv137.vcf.gz ftp://ftp.sra.ebi.ac.uk/vol1/analysis/ERZ022/ERZ022025/mgp.v3.snps.rsIDdbSNPv137.vcf.gz
+```
+
+On the terminal, log in to DivBase:
+
+```bash
+divbase-cli auth login <USER_EMAIL>
+```
+
+We then need to upload the file to the DivBase project. If the project you want to upload to is your default project, you can skip `--project <YOUR_DIVBASE_PROJECT_NAME>`, but otherwise you will need to specify that.
+
+```bash
+divbase-cli files upload mgp.v3.snps.rsIDdbSNPv137.vcf.gz --project <YOUR_DIVBASE_PROJECT_NAME>
+```
+
+For the sake of demonstration, we can create a sidecar metadata file for the 18 samples in this dataset so that we can use that for the query. In fact, a file prepared for this demo can be downloaded from the DivBase repo with:
+
+```bash
+curl -o tutorial_mock_metadata_mgpv3snps.tsv https://raw.githubusercontent.com/ScilifelabDataCentre/divbase/refs/heads/docs-on-queries/tests/fixtures/tutorial_mock_metadata_mgpv3snps.tsv
+```
+
+TODO update the TSV url when it has been merged to main.
+
+And then upload it to the DivBase project with:
+
+```bash
+divbase-cli files upload tutorial_mock_metadata_mgpv3snps.tsv --project <YOUR_DIVBASE_PROJECT_NAME>
+```
+
+With this, we are set in terms of the files we need in the DivBase project to be able to run the query.
+
+### A comment on the mock sidecar metadata TSV
+
+If we look at the first few lines of this TSV, we see that the first column has the SAMPLE IDs from the VCF. There is a column for population grouping, and a column for the geographical area the sample comes from. All of this is made up for the tutorial, just to illustrate what a sidecar metadata file can look like. We will use the Area column later on to subset on samples.
+
+```bash
+head tests/fixtures/tutorial_mock_metadata_mgpv3snps.tsv
+
+#Sample_ID      Population      Area
+129P2   1       North
+129S1   2       East
+129S5   3       South
+AJ      4       West
+```
+
+For comparison, we can inspect all the samples in the VCF with:
+
+```bash
+# On MacOS, use gzcat instead of zcat
+zcat mgp.v3.snps.rsIDdbSNPv137.vcf.gz | grep "#CHROM"
+
+#CHROM  POS     ID      REF     ALT     QUAL    FILTER  INFO    FORMAT  129P2   129S1   129S5   AJ      AKRJ    BALBcJ  C3HHeJ  C57BL6NJ        CASTEiJ CBAJ    DBA2J   FVBNJ   LPJ     NODShiLtJ       NZOHlLtJ   PWKPhJ  SPRETEiJ        WSBEiJ
+```
+
+## 2. Submit a job to cache the VCF dimensions in the DivBase backend
+
+DivBase needs this to make calculations. It can be seen as a pre-processing step that needs to be run every time a new version of a VCF is uploaded to DivBase.
+
+```bash
+divbase-cli dimensions update --project <YOUR_DIVBASE_PROJECT_NAME>
+```
+
+The task should now have been submitted. The terminal prints a DivBase Task ID with a message like this:
+
+```
+# Example with Task ID 102
+Job submitted successfully with task id: 102
+```
+
+Make note of the Task ID integer for for now; we will use it later in the tutorial for a variable named `<THE_JOB_ID_OF_THE_QUERY>`.
+
+We need to wait until this has finished before we can send the actual DivBase query.
+
+```bash
+divbase-cli task-history user
+```
+
+## 3. Submit a query job
+
+```bash
+divbase-cli query bcftools-pipe --tsv-filter 'Area:North,East' --command 'view -s SAMPLES; view -r 1:15000000-25000000' --metadata-tsv-name tutorial_mock_metadata_mgpv3snps.tsv --project <YOUR_DIVBASE_PROJECT_NAME>
+```
+
+note that we are specifically using the `tutorial_mock_metadata_mgpv3snps.tsv` sample metadata. If this is not specified, the query will default to `sample_metadata.tsv`.
+
+Depending on queue length, this job will take some time to run. Check that it has started, and then leave it running for half an hour.
+
+```bash
+divbase-cli task-history user
+```
+
+## 4. Download the results file
+
+```bash
+divbase-cli file download result_of_job_<THE_JOB_ID_OF_THE_QUERY>.vcf.gz--project <YOUR_DIVBASE_PROJECT_NAME>
+```
+
+We can now run some quick sanity-checks on the result file.
+
+```bash
+# On MacOS, use gzcat instead of zcat
+zcat result_of_job_<THE_JOB_ID_OF_THE_QUERY>.vcf.gz | grep -v "^#" |wc -l
+
+# Expected terminal output:
+297415
+```
+
+If you want, you can compare this to the original file:
+
+```bash
+# Note! This will take a little time since this file has many rows
+zzcat mgp.v3.snps.rsIDdbSNPv137.vcf.gz | grep -v "^#" |wc -l
+
+# Expected terminal output:
+66007044
+```
+
+As for the samples, the expected result is the following:
+
+```bash
+zcat mgp.v3.snps.rsIDdbSNPv137.vcf.gz | grep "#CHROM"
+#CHROM  POS     ID      REF     ALT     QUAL    FILTER  INFO    FORMAT  129P2   129S1   129S5   AJ      AKRJ    BALBcJ  C3HHeJ  C57BL6NJ        CASTEiJ CBAJ    DBA2J   FVBNJ   LPJ     NODShiLtJ       NZOHlLtJ   PWKPhJ  SPRETEiJ        WSBEiJ
+
+zcat result_of_job_<THE_JOB_ID_OF_THE_QUERY>.vcf.gz | grep "#CHROM"
+#CHROM  POS     ID      REF     ALT     QUAL    FILTER  INFO    FORMAT  129P2   129S1   AKRJ    BALBcJ  CASTEiJ CBAJ    LPJ     NODShiLtJ       SPRETEiJ        WSBEiJ
+```

docs/user-guides/vcf-dimensions.md

@@ -0,0 +1,22 @@
+# VCF Dimensions caching
+
+TODO
+
+This a prerequisite step before running any queries
+
+The dimensions update job will:
+
+- For each VCF files in the project data store:
+  - store filename and data store file version ID
+  - store timestamp of last update [TODO IS THIS LAST TIME DIMENSIONS WAS RUN OR LAST TIME FILE WAS UPDATED?]
+  - Fetche and store all sample names from the VCF file
+  - Fetch and store all  chromosome/scaffold/contig names
+  - count the number of samples and variants in the VCF file
+- not consider results files produced by DivBase
+  - by filename (`result_of_job_` prefix)
+  - as an additional check, looks for bcftools annotate stamp in header (all DivBase results files are branded like this)
+
+Run `divbase-cli dimensions update` whenever:
+
+- A new VCF file is uploaded to the project
+- A new version of an existing VCF file is uploaded to the project