ScilifelabDataCentre · brinkdp · Mar 11, 2026 · Mar 2, 2026 · Mar 2, 2026 · Mar 2, 2026
@@ -2,6 +2,11 @@
 
 TODO
 
+TODO this note is from the dimensions page, but it's good to have here too:
+
+!!! Note
+    When you submit a query, DivBase will use the state of the VCF Dimensions and the VCF files at that very point in time to produce the query results. It is therefore fine if you or another project member uploads new VCF files to the project while a query is queued or running.
+
 ## combined sample metadata and VCF queries
 
 TODO - there is a link to here from the sample metadata guide, so the combined queries should be described in detail here

@@ -81,7 +81,7 @@ divbase-cli files ls
 
 ## Step 7: Dimensions update
 
-For DivBase to be able to efficiently handle the VCF files in the the project, some key information about each VCF files is fetched from the files. In DivBase, this is refered to as "VCF dimensions". These include for instance which samples and scaffolds that a VCF file contains.
+For DivBase to be able to efficiently handle the VCF files in the the project, some key information about each VCF files is fetched from the files and cached in the server. In DivBase, this is refered to as "VCF dimensions". These include for instance which samples and scaffolds that a VCF file contains.
 
 Update the project dimensions after uploading your files.
 
@@ -94,7 +94,7 @@ This submits a task to the DivBase task management system. The task will wait in
 !!! notes
     1. Please note that it is not possible to run VCF queries in DivBase until the dimensions update task has finished. The reason for this is that the VCF queries use the dimensions data ensure that the queries are feasible and to know which VCF files from the project to process.
 
-    2. Please also note that the `divbase-cli dimensions update` command needs to be done every time a new VCF or a new version of a VCF file is uploaded.
+    2. Please also note that the `divbase-cli dimensions update` command needs to be run every time a new VCF or a new version of a VCF file is uploaded.
 
 ## Step 8: Confirm dimensions update job completion
 

@@ -1,34 +1,57 @@
 # Running Queries: Overview
 
-DivBase allows users to submit queries to checkout sample metadata and VCF data from the DivBase project data store. There types of queries in DivBase are:
+DivBase allows users to submit queries to checkout sample metadata and VCF data from the DivBase project data store. The three types of queries in DivBase are:
 
 - Sample metadata query
 - VCF data query
 - Combined sample metadata and VCF data query
 
 The sample metadata is a user defined, sidecar TSV (tab separated variables) file where users can add any custom metadata as columns. The VCF queries use `bcftools` to subset data from the files in the data store based on user defined filters. The VCF queries can be combined with a sample metadata query to allow users to filter both on data in the sidecar metadata and VCF files in a single query.
 
-TODO
+A few quick things that are good to know when it comes to running DivBase queries:
 
-- [TO BE IMPLEMENTED] think about the `divbase-cli query` commands. it would make sense to use `sample-metadata` (perhaps with `tsv` as a short form) and `vcf`
+- The queries are run as jobs in the DivBase job queue management system since they can potentially take some time to run, depending on the VCF file size and the query itself.
 
-The system will use the latest version the the files for all queries.
+- The system will use the latest version of the files for all queries.
+
+- The VCF Dimensions for the project needs to be up-to-date before running any queries, as described in [Before running any queries: run the VCF dimensions command](#before-running-any-queries-run-the-vcf-dimensions-command) below.
+
+This page is meant as medium detailed description of these query commands, and therefore sits in-between the [Quick Start Guide](quick-start.md) and the full guides found here in the Running Queries chapter in terms of level-of-detail. The sections below contain links to the full guides.
 
 ## Before running any queries: run the VCF dimensions command
 
-TODO - finish writing this section
+To ensure that users get good performance out of DivBase, technical metadata from all VCF files that are uploaded to a DivBase project will be indexed on the DivBase server. For each VCF file name, the server will extract the number and name of all samples, the number and name of all scaffolds, and the number of variants. This technical VCF metadata is referred to as VCF Dimensions in DivBase. The full details of how to use this are described on the page: [VCF Dimensions](vcf-dimensions.md).
+
+DivBase will use the VCF Dimensions whenever a user submits a query to the project. Instead of transferring VCF files internally and reading them each time a user sends a query to DivBase, the server will instead use the VCF Dimensions cache, which is a much quicker process.
+
+The VCF Dimensions must be updated each time a new VCF file or a new version of a VCF file is uploaded to the project. This needs to be done manually by running the command:
+
+```bash
+divbase-cli dimensions update
+```
+
+This command is sent to the DivBase job queue system, since this process can potentially take some time for larger VCF files. DivBase uses `bcftools` to extract the VCF dimensions metadata, which is a fast software. Nevertheless, the time it will take to extract the VCF dimensions will scale proportionally to the VCF file size. You can see this as a once-per-VCF-file-version investment: it will take a little time here up-front, but all subsequent queries to this file will be much faster than they otherwise would be.
 
-For more details see [VCF Dimensions caching](vcf-dimensions.md)
+Please wait until the VCF dimensions job has finished for your project before you make any queries to DivBase. You can check the status of the job with:
 
-For performance reasons and to ensure query feasibility, key metadata from the VCF files must first be cached in DivBase.
-For each VCF in the DivBase project, the system extracts file name, number and name of all samples, number and name of all scaffolds, number of variants,
+```bash
+divbase-cli task-history user
+```
+
+To see the current VCF dimensions for a project, use:
 
-DivBase will use this whenever a user submits a query to the project. For instance, the user might make a sample metadata filtering that results in only certain samples. The system knows which file names each requested sample are located in, and will ensure that only those files will be transferred to the worker.
+```bash
+divbase-cli dimensions show
+```
 
-example
-dimensions show
+This will print the currently cached VCF Dimensions in DivBase per VCF file name ([see example here](vcf-dimensions.md#dimensions-show))
 
-how to use dimensions show to get all samples in the project
+The VCF Dimensions of the project should now be up-to-date with the VCF files in the projects data store. You are now ready to run queries.
+
+!!! Note
+    Remember that if you or another project member uploads a VCF file to the project, you will need to run `divbase-cli dimensions update` again before running queries.
+
+    When you submit a query, DivBase will use the state of the VCF Dimensions and the VCF files at that very point in time to produce the query results. It is therefore fine if you or another project member uploads new VCF files to the project while a query is queued or running.
 
 ## Sidecar sample metadata queries
 
@@ -70,4 +93,6 @@ DivBase only allows `bcftools view` in its query syntax and no other `bcftools`
 
 ## Combined sample metadata and VCF data query
 
+TODO - finish writing this section
+
 Uses a sample metadata query to identify the VCF files in the DivBase project to run the VCF queries on.
@@ -1,22 +1,128 @@
 # VCF Dimensions caching
 
-TODO
+The DivBase query system is built around the DivBase server having cached key technical metadata from each VCF file in a project. In DivBase, this information as the "VCF dimensions", and for instance includes the names of samples and scaffolds in each VCF file and the version ID of the file in the project data store. The VCF Dimensions is a snapshot of the VCF files in the object store at the time the command to update the VCF Dimensions cache for the project was last run.
 
-This a prerequisite step before running any queries
+This allows the DivBase server to make quick checks against against the project VCF Dimensions cache when users submit a query or validate a sidecar metadata TSV file instead of having to read each VCF file every time a query is submitted. This makes the server-side operations more efficient when it comes to internal transfer and reading of files, and for checking VCF query feasibility against the requirements of `bcftools`. For the user, this will lead to faster feedback, especially when the system needs to send an error or warning message.
 
-The dimensions update job will:
+**An updated VCF Dimensions cache for DivBase project is a prerequisite step before submitting any queries**. Updating it is done with the command `divbase-cli dimensions update`. The command needs to be run every time the VCF files in a DivBase project has changed, that is:
 
-- For each VCF files in the project data store:
-  - store filename and data store file version ID
-  - store timestamp of last update [TODO IS THIS LAST TIME DIMENSIONS WAS RUN OR LAST TIME FILE WAS UPDATED?]
-  - Fetche and store all sample names from the VCF file
-  - Fetch and store all  chromosome/scaffold/contig names
-  - count the number of samples and variants in the VCF file
-- not consider results files produced by DivBase
-  - by filename (`result_of_job_` prefix)
-  - as an additional check, looks for bcftools annotate stamp in header (all DivBase results files are branded like this)
+- When any new VCF file is uploaded to the project.
 
-Run `divbase-cli dimensions update` whenever:
+- When an existing VCF file is replaced with a new version.
 
-- A new VCF file is uploaded to the project
-- A new version of an existing VCF file is uploaded to the project
+!!! Warning
+    DivBase system does not automatically run `divbase-cli dimensions update` when files are uploaded. This needs to be manually done by one project member that has at least an EDIT role in the project.
+
+There are four CLI commands sorted under `divbase-cli dimensions`:
+
+- [update](#dimensions-update)
+
+- [show](#dimensions-show)
+
+- [create-metadata-template](sidecar-metadata.md#creating-a-sidecar-sample-metadata-tsv-for-a-divbase-project)
+
+- [validate-metadata-file](sidecar-metadata.md#validating-a-sidecar-metadata-tsv-with-divbase-cli)
+
+The former two relate generating/updating and viewing the VCF dimensions of a project and will be described on this page. The latter two relate to the user-defined sidecar metadata TSV and are described in the guide on [Sidecar sample metadata TSV queries](sidecar-metadata.md). The commands are sorted under `divbase-cli dimensions` since they rely on calling the VCF dimensions cache of the project. A list of all DivBase CLI commands that require an up-to-date VCF Dimensions cache is found in the section [CLI Commands that rely on that the project's VCF dimensions cache is up to date](#cli-commands-that-rely-on-that-the-projects-vcf-dimensions-cache-is-up-to-date)
+
+## Dimensions update
+
+The `divbase-cli dimensions update` pre-reads the VCF files in the project's data store so that DivBase does not need to fetch and read each VCF every time the user submits commands that require knowledge about the VCF dimensions, such as queries.
+
+The `divbase-cli dimensions update` will look for `.vcf.gz` files in the project's data store. If there is no VCF dimensions cache for the project, it will create it. If not, it will compare the existing record in the cache with the current status of the object store. If any new VCF files have been added or if any VCF file version have been updated, it will update the VCF dimensions cache with that information. DivBase uses `bcftools` to extract the dimensions information from the VCF files.
+
+The update of the VCF Dimensions is scheduled as a job in the DivBase job queue system since it can a potentially take little time to update a VCF dimensions for projects that contain many or large VCF files that has not been previously cached. This is an up-front time investment: the time it takes to run the update command is saved on every subsequent command that needs to check the VCF dimensions. After submitting a job with `divbase-cli dimensions update`, use `divbase-cli task-history user` or `divbase-cli task-history id <JOB_ID_RECIEVED_AT_JOB_SUBMISSION>` to check the status of the VCF Dimensions Update job.
+
+VCF results files that have been produced by DivBase are not indexed in the VCF Dimensions cache. The main reason for this is that they result files contain a subset of the VCF data in the project, and will thus contain duplicate data. The results files are not used for any queries: only the source VCF files uploaded by the users are. DivBase recognizes its results files on two levels: the files names have a `result_of_job_` prefix, and their VCF headers contain a row with `##DivBase_created`.
+
+### What is cached by divbase-cli dimensions update?
+
+The VCF Dimensions cache stores this information for each VCF file in the project's data store:
+
+- filename (`*.vcf.gz`)
+- version ID in the data store (each file name is versioned and has a version history. The VCF Dimensions cache uses the latest version of all files in the data store.)
+- time stamp of when the VCF dimensions for this file was updated
+- sample names contained in this VCF file
+- sample count
+- scaffold names contained in this VCF file
+- scaffold count
+- variant count
+
+## Dimensions show
+
+The `divbase-cli dimensions show` command print **the current state** of the VCF Dimensions cache for the project to the user's terminal. It will display the cached dimensions data for each VCF file, as described in [the previous section](#what-is-cached-by-divbase-cli-dimensions-update). If a project has many VCF files, samples, and/or scaffolds, this display can potentially be long.
+
+```bash
+divbase-cli dimensions show
+```
+
+For each VCF file in the project, it will show the following (example data):
+
+```text
+indexed_files:
+- filename: my_vcf_file1.vcf.gz
+  file_version_ID_in_bucket: <VERSION_ID>
+  last_updated: <TIME_STAMP>
+  dimensions:
+    scaffold_count: 12
+    scaffolds:
+    - 'chr1'
+    - 'chr2'
+    - 'chr3'
+    - 'chr4'
+    - 'chr5'
+    - 'chr6'
+    - 'chr7'
+    - 'chr8'
+    - 'chr9'
+    - 'chr10'
+    - 'chr11'
+    - 'chr12'
+    sample_count: 10
+    sample_names:
+    - 1_ABC
+    - 2_ABC
+    - 3_ABC
+    - 4_ABC
+    - 5_ABC
+    - 6_ABC
+    - 7_ABC
+    - 8_ABC
+    - 8_ABC
+    - 10_ABC
+    variants: 12524
+```
+
+It is also possible to filter the output by adding the following options to the `divbase-cli dimensions show` command:
+
+| Option                   | Description                                                                                                             |
+|--------------------------|-------------------------------------------------------------------------------------------------------------------------|
+| `--filename`             | Show only the entry for this VCF filename                                                                               |
+| `--unique-scaffolds`     | Show all unique scaffold names found across all the VCF files in the project                                            |
+| `--unique-samples`       | Show all unique sample names found across all the VCF files in the project                                              |
+| `--sample-names-limit`   | Maximum number of sample names to display per list in terminal output. [default: 20]                                    |
+| `--sample-names-output`  | Write full sample names to file instead of truncating in terminal output. Mutually exclusive with --sample-names-stdout.|
+| `--sample-names-stdout`  | Print full sample names to stdout (useful for piping). Mutually exclusive with --sample-names-output.                   |
+
+(This information can also be displayed in the terminal with `divbase-cli dimensions show -h`)
+
+!!! Note
+    If the user is running `divbase-cli` in a UNIX environment, it is possible to send the full output to file with UNIX shell output redirection, e.g. `divbase-cli dimensions show > my_dimensions_cache.txt`. This should be equivalent of using the `--sample-names-output` option.
+
+## CLI Commands that rely on that the project's VCF dimensions cache is up to date
+
+Several DivBase CLI commands required that the VCF dimensions cache of the project is up-to-date with the current versions of the VCF files in the project's data store:
+
+`divbase-cli dimensions show`
+
+`divbase-cli dimensions create-metadata-template`
+
+`divbase-cli dimensions validate-metadata-file`
+
+`divbase-cli query tsv`
+
+`divbase-cli query bcftools-pipe`
+
+TODO: describe how these commands use the dimensions and how they behave when the dimensions cache is missing or not up to date with the project's S3 bucket
+
+TODO: add link here to discussion in VCF Query docs on sample set compatibility. E.g.: The VCF dimensions are also used to a-priori estimate whether the sample sets in the VCF files are compatible with the requirements of `bcftools merge` and `bcftools concat`.