SQ-837: Refine the sample metadata TSV handling and write user guides

.gitignore

@@ -13,14 +13,15 @@ __pycache__/
 
 # project specific files
 /sample_metadata.tsv
-sample_metadata_*.tsv
+/sample_metadata_*.tsv
 *.vcf
 *.vcf.gz
 *.vcf.gz.csi
 *.vcf.gz.tbi
 !tests/fixtures/*.vcf.gz
 tests/fixtures/temp*
 tests/fixtures/merged*
+divbase_metadata_template*.tsv
 
 # query job config files
 bcftools_divbase_job_config.json
@@ -37,6 +38,4 @@ scripts/benchmarking/results
 .DS_Store
 
 # mkdocs build cache
-.cache/
-# pypi 
-dist/
+.cache/

docs/cli/_auto_generated/dimensions.md

@@ -18,6 +18,8 @@ $ divbase-cli dimensions [OPTIONS] COMMAND [ARGS]...
 
 * `update`: Calculate and add the dimensions of a VCF...
 * `show`: Show the dimensions index file for a project.
+* `create-metadata-template`: Use the samples index in a projects...
+* `validate-metadata-file`: Validate a sidecar metadata TSV file...
 
 ## `divbase-cli dimensions update`
 
@@ -49,5 +51,60 @@ $ divbase-cli dimensions show [OPTIONS]
 
 * `--filename TEXT`: If set, will show only the entry for this VCF filename.
 * `--unique-scaffolds`: If set, will show all unique scaffold names found across all the VCF files in the project.
+* `--unique-samples`: If set, will show all unique sample names found across all the VCF files in the project.
+* `--sample-names-limit INTEGER RANGE`: Maximum number of sample names to display per list in terminal output.  [default: 20; x>=1]
+* `--sample-names-output TEXT`: Write full sample names to file instead of truncating in terminal output. Mutually exclusive with --sample-names-stdout.
+* `--sample-names-stdout`: Print full sample names to stdout (useful for piping). Mutually exclusive with --sample-names-output.
+* `--project TEXT`: Name of the DivBase project, if not provided uses the default in your DivBase config file
+* `--help`: Show this message and exit.
+
+## `divbase-cli dimensions create-metadata-template`
+
+Use the samples index in a projects dimensions cache to create a TSV metadata template file
+that has the sample names as pre-filled as the first column.
+
+**Usage**:
+
+```console
+$ divbase-cli dimensions create-metadata-template [OPTIONS]
+```
+
+**Options**:
+
+* `-o, --output TEXT`: Name of the output TSV file to create. Defaults to sample_metadata_<project_name>.tsv. If a file with the same name already exists in the current directory, you will be prompted to confirm if you want to overwrite it.
+* `--project TEXT`: Name of the DivBase project, if not provided uses the default in your DivBase config file
+* `--help`: Show this message and exit.
+
+## `divbase-cli dimensions validate-metadata-file`
+
+Validate a sidecar metadata TSV file against DivBase formatting requirements and project dimensions.
+
+Validation is run client-side to keep sensitive metadata local during validation.
+
+Validation checks:
+- File is properly tab-delimited
+- First column is named '#Sample_ID'
+- No commas in cells
+- Sample_ID has only one value per row (no semicolons)
+- No duplicate sample IDs
+- Invalid characters
+- Basic type consistency in user-defined columns. But not Pandas type inference,
+  as we want to avoid having the user install Pandas just for validation. So just check that numeric columns have only numeric values (excluding header).
+- All samples in the TSV exist in the project's dimensions index
+
+Returns errors for critical issues and warnings for non-critical issues.
+
+**Usage**:
+
+```console
+$ divbase-cli dimensions validate-metadata-file [OPTIONS] INPUT_FILENAME
+```
+
+**Arguments**:
+
+* `INPUT_FILENAME`: Name of the input TSV file to validate.  [required]
+
+**Options**:
+
 * `--project TEXT`: Name of the DivBase project, if not provided uses the default in your DivBase config file
 * `--help`: Show this message and exit.

docs/development/metadata_queries.md

@@ -0,0 +1,20 @@
+# Sidecar sample metadata TSV queries
+
+The query engine for filtering on user-defined sidecar TSVs is centralised around the class `SharedMetadataValidator` in the `divbase-lib` package. This class validated that a sidecar TSV is compatible with DivBase, and is called by the client-side validator and by the server-side query engine (`SidecarQueryManager`). Important to understand is that the `SharedMetadataValidator` was designed to not raise errors or reports warnings: it only collects them. Then the CLI and query engine each acts on the information.
+
+The client side uses the results from `SharedMetadataValidator` to print stats, errors, and warnings in the user's terminal.
+
+The server side uses Enums in the `SharedMetadataValidator` results to raise errors and report warnings to the user. This means that the `SidecarQueryManager` only need to be concerned about acting on the Enum and not the error/warning messages themselves, thus avoiding having to rely on (potentially fragile) string-matching.
+
+The syntax for the TSV and the query fileters is described in the [Sidecar Metadata TSV files: creating and querying sample metadata files](../user-guides/sidecar-metadata.md).
+
+## Script to inspect the dataframe
+
+Despite all the tests, there might be cases where the Pandas dataframe does not look like you'd think after ingesting the TSV. Such issues can be difficult to debug. To help with that this script can be used to calls the `SharedMetadataValidator` dataframe logic for TSV ingestion and print the Pandas dataframe to terminal:
+
+```bash
+python scripts/tsv_to_dataframe.py --tsv path/to/metadata.tsv
+
+# Try it with example fixture
+python scripts/tsv_to_dataframe.py --tsv tests/fixtures/sample_metadata.tsv
+```

docs/user-guides/query-syntax.md

@@ -1,3 +1,13 @@
 # DivBase Query Syntax for VCF data
 
 TODO
+
+## combined sample metadata and VCF queries
+
+TODO - there is a link to here from the sample metadata guide, so the combined queries should be described in detail here
+
+It is also possible to run a sidecar sample metadata query as part of a VCF query by adding the query as a sting to the flag `--tsv-filter`:
+
+```bash
+divbase-cli query bcftools-pipe --tsv-filter "Area:North" --command "view -s SAMPLES"
+```

docs/user-guides/quick-start.md

@@ -79,23 +79,7 @@ Check your uploaded files:
 divbase-cli files ls
 ```
 
-## Step 7: Upload sample metadata
-
-TODO It might make more sense to have run the dimensions update job before this if we are to use a pre-populated template file
-
-Sample metadata must be uploaded as follows:
-
-- In TSV format and be named "sample_metadata.tsv"
-- Must contain a column named "sample_id" which matches the sample IDs in your VCF files
-- The names and values of all other columns are optional.
-
-TODO update this after `sidecar-metadata.md` docs are done, there are changes planned for some details.
-
-```bash
-divbase-cli files upload path/to/your/sample_metadata.tsv
-```
-
-## Step 8: Dimensions update
+## Step 7: Dimensions update
 
 For DivBase to be able to efficiently handle the VCF files in the the project, some key information about each VCF files is fetched from the files. In DivBase, this is refered to as "VCF dimensions". These include for instance which samples and scaffolds that a VCF file contains.
 
@@ -112,7 +96,7 @@ This submits a task to the DivBase task management system. The task will wait in
 
     2. Please also note that the `divbase-cli dimensions update` command needs to be done every time a new VCF or a new version of a VCF file is uploaded.
 
-## Step 9: Confirm dimensions update job completion
+## Step 8: Confirm dimensions update job completion
 
 Check the task history to confirm the dimensions update job has completed:
 
@@ -128,6 +112,40 @@ It is possible to inspect the cached VCF dimensions data for the project at any
 divbase-cli dimensions show
 ```
 
+## Step 9: Upload sample metadata
+
+DivBase can checkout data based the VCF files themselves, but can also take an optional sidecar sample metadata file into account. The metadata file must be a TSV (tab-separated variables) file. The metadata contents of the file is defined by the users. If the VCF dimensions command has been run for the project, the cached dimensions data can be used create a template where the samples of the project have been pre-filled:
+
+```bash
+divbase-cli dimensions create-metadata-template
+```
+
+Details on how to write this file are given in [Sidecar Metadata TSV files: creating and querying sample metadata files](sidecar-metadata.md). In short, the first row starts with `#` and contains the headers for different metadata columns. The first column (`Sample_ID`) is mandatory and can be created by the system as just described; if created manually just make sure that each sample name is spelled exactly as in the VCF files. The rest of the columns are free for the user to define.
+
+Example of a sidecar metadata TSV file with the mandatory `Sample_ID` column and two user defined columns.
+
+```
+#Sample_ID Population Area
+129P2 1 North
+129S1 2 East
+129S5 3 South
+```
+
+!!! note
+    Please use a text editor that preserves the tabs when the file is saved. Incorrect tabs can lead to issues with running metadata queries in DivBase.
+
+There is a command to help check that the sidecar metadata TSV is correctly formatted for use with DivBase. Running it is optional:
+
+```bash
+divbase-cli dimensions validate-metadata-file path/to/your/sample_metadata.tsv
+```
+
+When you are happy with the sample metadata file, it should be uploaded to the DivBase project with the following:
+
+```bash
+divbase-cli files upload path/to/your/sample_metadata.tsv
+```
+
 ## Step 10: Run your queries
 
 There are three types of queries in DivBase:

docs/user-guides/running-queries.md

@@ -14,40 +14,60 @@ TODO
 
 The system will use the latest version the the files for all queries.
 
-TODO COPIED OVER FROM QUICKSTART REIMPLEMENT
+## Before running any queries: run the VCF dimensions command
 
-DivBase only allows `bcftools view` in its query syntax and no other `bcftools` commands. The `merge`, `concat`, and `annotate` commands are used when processing a query, but should not be defined by the user.
+TODO - finish writing this section
 
-## Side car sample metadata queries
+For more details see [VCF Dimensions caching](vcf-dimensions.md)
 
-For more details, see [Sidecar Metadata TSV files: creating and querying sample metadata files](sidecar-metadata.md).
+For performance reasons and to ensure query feasibility, key metadata from the VCF files must first be cached in DivBase.
+For each VCF in the DivBase project, the system extracts file name, number and name of all samples, number and name of all scaffolds, number of variants,
 
-how to write the sample metadata TSV (+template)
+DivBase will use this whenever a user submits a query to the project. For instance, the user might make a sample metadata filtering that results in only certain samples. The system knows which file names each requested sample are located in, and will ensure that only those files will be transferred to the worker.
+
+example
+dimensions show
 
 how to use dimensions show to get all samples in the project
 
-how to write query
+## Sidecar sample metadata queries
 
-## Before any VCF queries: run the VCF dimensions command
+DivBase supports that users store extensive sample metadata in a separate TSV file. This metadata can be queried on its own or in combination with VCF data. Users are free to define their own metadata as they see if: column names represent metadata categories and rows represent the samples found in the VCF files in the DivBase project. The TSVs need to follow a few mandatory requirements, but no strict metadata schema is enforced, This allows DivBase to accomodate a variety of research projects with different metadata needs.
 
-For more details see [VCF Dimensions caching](vcf-dimensions.md)
+The metadata can be queried on its own to learn which samples that fulfil a certain metadata query and the VCF files the samples are present in. The same query syntax is used in the combined sample metadata and VCF data queries, and user can use the dedicated sample metadata query command as a dry-run before running full combined query to ensure that the metadata query produces the results the user intended.
 
-For performance reasons and to ensure query feasibility, key metadata from the VCF files must first be cached in DivBase.
-For each VCF in the DivBase project, the system extracts file name, number and name of all samples, number and name of all scaffolds, number of variants,
+For instructions on how to create the sidecar sample metadata TSV files and how to run sample metadata queries, see the guide on [Sidecar Metadata TSV files: creating and querying sample metadata files](sidecar-metadata.md). The guide also describes the CLI commands that specifically relate to the sample metadata TSV files. These are:
 
-DivBase will use this whenever a user submits a query to the project. For instance, the user might make a sample metadata filtering that results in only certain samples. The system knows which file names each requested sample are located in, and will ensure that only those files will be transferred to the worker.
+```bash
+divbase-cli dimensions create-metadata-template
 
-example
-dimensions show
+divbase-cli dimensions validate-metadata-file path/to/your/sample_metadata.tsv
+
+divbase-cli files upload path/to/your/sample_metadata.tsv
+
+divbase-cli query tsv "Area:Northern Portugal"
+```
 
 ## VCF queries
 
+TODO - finish writing this section
+
+Run the query on all VCF files in the DivBase project unless specified. There are two ways to specify the files: either as direct input to the `bcftools` command, or by combining the VCF query with a sample metadata query to determine which VCF files to use.
+
 See also [DivBase Query Syntax for VCF data](query-syntax.md), [How to create efficient DivBase queries](how-to-create-efficient-divbase-queries.md), and [Tutorial: Running a query on a public dataset](tutorial-query-on-public-data.md).
 
-can be run with or without sample metadata filtering
+can be run with or without sample metadata filtering.
 
 for sample metadata linked VCF queries, it can be good to do a dry run first [TO BE IMPLEMENTED, at the moment it needs to be run seperatelly]
 
 how to write a query
 
 how to optimize a query (see separate markdown)
+
+TODO COPIED OVER FROM QUICKSTART REIMPLEMENT
+
+DivBase only allows `bcftools view` in its query syntax and no other `bcftools` commands. The `merge`, `concat`, and `annotate` commands are used when processing a query, but should not be defined by the user.
+
+## Combined sample metadata and VCF data query
+
+Uses a sample metadata query to identify the VCF files in the DivBase project to run the VCF queries on.