Skip to content

Commit 8c68980

Browse files
RunpengLuoRunpengLuo
committed
docs: consolidate guides into README; add CI; cnplot @main
- remove docs/pipeline.md (duplicated the README) and docs/workflow.md; repoint all inbound links to README.md / reference.md - README: self-contained install/configure/run; fix root-relative doc links - reference.md: add Dependencies section; phase_dataset_ids multi/auto - bulk doc: add phase_and_concat.bulk.pdf QC output; per-mode docs point to README - add .github/workflows/ci.yml: pytest DAG dry-run + real-sample (HCC1395) snakemake -n, matrix over Python 3.11 / 3.13 - base.yaml: track cnplot @main
1 parent f03a94d commit 8c68980

11 files changed

Lines changed: 153 additions & 180 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
branches: [main, dev, "feat/**"]
6+
pull_request:
7+
8+
concurrency:
9+
group: ci-${{ github.ref }}
10+
cancel-in-progress: true
11+
12+
jobs:
13+
dryrun:
14+
name: pytest + snakemake dry-run (py${{ matrix.python-version }})
15+
runs-on: ubuntu-latest
16+
strategy:
17+
fail-fast: false
18+
matrix:
19+
python-version: ["3.11", "3.13"]
20+
steps:
21+
- uses: actions/checkout@v4
22+
23+
- uses: actions/setup-python@v5
24+
with:
25+
python-version: ${{ matrix.python-version }}
26+
27+
- name: Install snakemake + DAG-parse deps
28+
run: |
29+
python -m pip install --upgrade pip
30+
pip install "snakemake>=9" snakemake-storage-plugin-http snakemake-storage-plugin-fs \
31+
pandas numpy pyyaml pulp pytest
32+
33+
- name: pytest (DAG dry-run, all modes x JSON/TSV)
34+
run: pytest tests/
35+
36+
- name: snakemake dry-run on a real sample sheet (HCC1395, chr16)
37+
run: |
38+
set -euo pipefail
39+
# Stub only the references not bundled in resources/data/; a dry-run just
40+
# builds the DAG, so empty files are enough.
41+
R=.test-run/HCC1395/reference
42+
mkdir -p "$R/target_positions" "$R/phasing_panel"
43+
: > "$R/hg38.fa"
44+
: > "$R/gencode.hg38.gtf.gz"
45+
: > "$R/genetic_map_hg38_withX.txt.gz"
46+
: > "$R/target_positions/target.chr16.pos.gz"
47+
: > "$R/target_positions/target.chr16.pos.gz.tbi"
48+
: > "$R/phasing_panel/chr16.genotypes.bcf"
49+
: > "$R/phasing_panel/chr16.genotypes.bcf.csi"
50+
snakemake -n -c1 -s workflow/Snakefile --configfile tests/data/HCC1395/config.yaml

‎README.md‎

Lines changed: 75 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,27 +1,94 @@
11
# Universal Genotyping Pipeline
22

3-
[![Snakemake](https://img.shields.io/badge/snakemake->=9.0-brightgreen.svg)](https://snakemake.readthedocs.io)
43
[![Version](https://img.shields.io/badge/version-0.1.0b1-blue.svg)](VERSION)
4+
[![Snakemake](https://img.shields.io/badge/snakemake->=9.0-brightgreen.svg)](https://snakemake.readthedocs.io)
55

66
Universal Genotyping Pipeline is a Snakemake preprocessing pipeline for downstream allele-specific CNA inference softwares including:
7-
- [HATCHet](https://github.com/raphael-group/hatchet) - bulk **short-read** WGS/WES, **long-read** PacBio HiFi/ONT,
8-
- [Copy-typing](https://github.com/raphael-group/Copy-typing) - scRNA-seq, scATAC-seq,
7+
- [HATCHet](https://github.com/raphael-group/hatchet) - bulk **short-read** WGS/WES, **long-read** PacBio HiFi/Oxford Nanopore,
8+
- [Copy-typing](https://github.com/raphael-group/Copy-typing) - scRNA-seq, scATAC-seq, Epi Multiome,
99
- [CalicoST](https://github.com/raphael-group/CalicoST) - Visium ST.
1010

1111
---
1212

13+
## Installation
14+
15+
`Universal-Genotyping-Pipeline` requires a 64-bit Linux system and [conda](https://docs.conda.io/en/latest/) and [Snakemake](https://snakemake.readthedocs.io/) (version 9 or newer). Create the runner environment from [`environment.yaml`](environment.yaml) and activate it:
16+
17+
```sh
18+
conda env create -f environment.yaml
19+
conda activate genotyping-env
20+
```
21+
22+
Build the conda environments for pipeline dependencies **once** for all future runs:
23+
24+
```sh
25+
snakemake --profile profile/ \
26+
--conda-create-envs-only --cores 1 \
27+
-s workflow/Snakefile
28+
```
29+
30+
All pipeline dependencies can be found at [`workflow/envs/`](./workflow/envs/), see [dependencies](./docs/reference.md#dependencies) for details.
31+
32+
---
33+
34+
## Configuration
35+
36+
---
37+
The profile at [`profile/config.yaml`](profile/config.yaml) holds run-wide settings. The main keys to check:
38+
39+
| Key | Usage |
40+
|-----|--------------|
41+
| `cores` | Max CPU cores the whole pipeline may use for job scheduling. |
42+
| `use-conda` | Keep `true`. |
43+
| `conda-prefix` | Where the pipeline dependencies environments live; set to an **absolute path**. |
44+
| `resources: downloads` | How many remote (URL) input files download at once; default `2`. |
45+
| `local-storage-prefix` | Where remote input files are downloaded; set to a large scratch disk. |
46+
47+
> [!IMPORTANT]
48+
> The default `conda-prefix` is the relative path `.snakemake/conda` w.r.t. current working directory. After the environments are built, change it to an absolute path to avoid re-building the environment for every new run.
49+
50+
---
51+
52+
The sample sheet ([template](resources/templates/samples.json)) is a JSON file lists the input datasets & configurations. Copy the template and modify from it according to [schema](docs/sample_sheet.md). Validate the format and check the file existence via:
53+
54+
```sh
55+
python resources/scripts/validate_sample_file.py /path/to/samples.json --check-files
56+
```
57+
58+
---
59+
60+
The config file ([template](resources/templates/config.yaml)) is a YAML file lists the detailed workflow parameters and paths. See [reference.md](docs/reference.md#configuration) for detailed guidelines for each specific workflow: [bulk-genotyping](docs/bulk_genotyping.md), [single-cell-genotyping](docs/single_cell_genotyping.md), [copytyping-preprocess](docs/copytyping_preprocess.md).
61+
62+
---
63+
64+
## Running the pipeline
65+
66+
Run the snakemake pipeline using following commands. `--config` allows user to overwrite config parameters against the config file.
67+
68+
```sh
69+
snakemake --profile profile/ \
70+
-s workflow/Snakefile \
71+
--configfile /path/to/my_config.yaml \
72+
--directory <output_dir> \
73+
--config sample_file=/path/to/samples.json sample_id=<PATIENT_ID>
74+
```
75+
76+
77+
> [!TIP]
78+
> - For the first run, use CMD argument `--dry-run` (`-n`). It lists the jobs Snakemake would run without actual executions, so you can confirm the plan and catch potential configuration errors.
79+
> - If a run failed at intermediate jobs or user changed downstream parameters, use CMD argument
80+
> `--rerun-incomplete` to resume execution.
81+
82+
---
83+
1384
## Documentation
1485

1586
| Document | Description |
1687
|----------|-------------|
17-
| [docs/pipeline.md](docs/pipeline.md) | Install, configure, and run the Snakemake pipeline. |
1888
| [docs/bulk_genotyping.md](docs/bulk_genotyping.md) | Bulk genotyping workflow |
1989
| [docs/single_cell_genotyping.md](docs/single_cell_genotyping.md) | Single-cell/spatial genotyping workflow |
2090
| [docs/copytyping_preprocess.md](docs/copytyping_preprocess.md) | Copy-typing preprocessing workflow |
21-
| [docs/workflow.md](docs/workflow.md) | Detailed description per workflow |
2291
| [docs/sample_sheet.md](docs/sample_sheet.md) | Sample sheet specification |
23-
| [docs/reference.md](docs/reference.md) | Manual for input, output, and (hyper-)parameters |
24-
| [config/config.yaml](config/config.yaml) | Default configuration (auto-loaded by the Snakefile) |
25-
| [resources/templates/](resources/templates/) | User config and sample-file templates |
92+
| [docs/reference.md](docs/reference.md) | Reference manual |
2693
| [resources/README.md](resources/README.md) | External data resources |
2794
| [CHANGELOG.md](CHANGELOG.md) | Release notes and version history |

‎docs/README.md‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,12 +2,10 @@
22

33
| Document | Contents |
44
|----------|----------|
5-
| [workflow.md](workflow.md) | The three workflow modes: stages, rules, and where each writes. |
65
| [sample_sheet.md](sample_sheet.md) | Sample-file schema, remote inputs, legacy TSV, validation. |
76
| [reference.md](reference.md) | Every config key and its default; every output file, column, and QC plot. |
8-
| [pipeline.md](pipeline.md) | Install, configure (cores/memory/downloads/storage), and run the Snakemake pipeline. |
97
| [bulk_genotyping.md](bulk_genotyping.md) | Tutorial: bulk WGS/WES + long-read, from panels to final bins. |
108
| [single_cell_genotyping.md](single_cell_genotyping.md) | Tutorial: single-cell (scRNA/scATAC/VISIUM). |
119
| [copytyping_preprocess.md](copytyping_preprocess.md) | Tutorial: copytyping preprocess. |
1210
| [TODO.md](TODO.md) | Development TODO. |
13-
| [../resources/README.md](../resources/README.md) | External resources: SNP/phasing panels, genetic maps, window BEDs. |
11+
| [../resources/README.md](../resources/README.md) | External resources |

‎docs/bulk_genotyping.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Bulk Genotyping
22

3-
This documentation covers input preparation and result intepretation for preprocessing **short-read** WGS/WES and/or **long-read** (e.g., PacBio HiFi, Oxford Nanopore) sequencing data to run [HATCHet](https://github.com/raphael-group/hatchet). Refer to [Pipeline](pipeline.md) for Snakemake pipeline installation and execution instructions.
3+
This documentation covers input preparation and result intepretation for preprocessing **short-read** WGS/WES and/or **long-read** (e.g., PacBio HiFi, Oxford Nanopore) sequencing data to run [HATCHet](https://github.com/raphael-group/hatchet). Refer to the [README](../README.md) for Snakemake pipeline installation and execution instructions.
44

55
## Table of Contents
66
1. [Overview](#overview) <br>

‎docs/copytyping_preprocess.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Copytyping Preprocess
22

3-
This documentation covers input preparation and result interpretation for the copytyping preprocess mode, which aggregates single-cell / spatial (**scRNA**, **scATAC**, **Visium**) allele and native counts onto a **pre-computed** set of copy-number blocks to run [CalicoST](https://github.com/raphael-group/CalicoST). This mode never genotypes or phases: a pre-computed phased het-SNP VCF (`het_snp_vcf`) and the genomic bin annotations (`bb_file`) are **required** inputs from running genotyping using matched bulk samples. Refer to [Pipeline](pipeline.md) for Snakemake pipeline installation and execution instructions.
3+
This documentation covers input preparation and result interpretation for the copytyping preprocess mode, which aggregates single-cell / spatial (**scRNA**, **scATAC**, **Visium**) allele and native counts onto a **pre-computed** set of copy-number blocks to run [CalicoST](https://github.com/raphael-group/CalicoST). This mode never genotypes or phases: a pre-computed phased het-SNP VCF (`het_snp_vcf`) and the genomic bin annotations (`bb_file`) are **required** inputs from running genotyping using matched bulk samples. Refer to the [README](../README.md) for Snakemake pipeline installation and execution instructions.
44

55
## Table of Contents
66
1. [Overview](#overview) <br>

‎docs/pipeline.md‎

Lines changed: 0 additions & 94 deletions
This file was deleted.

‎docs/reference.md‎

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
# Reference
22

33
## Table of Contents
4+
- [Dependencies](#dependencies)
45
- [Sample File](#sample-file)
56
- [Configuration](#configuration)
67
- [Input Data](#input-data)
@@ -13,6 +14,25 @@
1314
- [Barcodes (single-cell)](#barcodes-single-cell)
1415
- [QC](#qc-qc_dir)
1516

17+
---
18+
19+
## Dependencies
20+
21+
Pipeline dependencies live under `workflow/envs/`:
22+
23+
| Environment | Purpose |
24+
|-------------|---------|
25+
| `base.yaml` | Python scientific stack (used by every mode). |
26+
| `bcftools.yaml` | bcftools/tabix: bulk genotyping + het-SNP pileup. |
27+
| `eagle.yaml` | Eagle2 phasing. |
28+
| `shapeit.yaml` | SHAPEIT5 phasing. |
29+
| `longphase.yaml` | LongPhase (long-read) phasing. |
30+
| `cellsnp.yaml` | cellsnp-lite: single-cell genotyping + pileup. |
31+
| `mosdepth.yaml` | mosdepth read-depth counting (bulk). |
32+
| `ucsc.yaml` | UCSC tools (`bigWigToBedGraph`, `liftOver`) for the Repli-seq track. |
33+
34+
---
35+
1636
## Sample File
1737
Refer to spec **[sample_sheet.md](sample_sheet.md)** and [templates](../resources/templates/).
1838

@@ -187,7 +207,7 @@ Used by all multi-thread rules.
187207

188208
## Outputs
189209

190-
Directories (`snp_dir`, `phase_dir`, `pileup_dir`, `allele_dir`, `bb_dir`, `qc_dir`, `log_dir`, `aux_dir`, `bench_dir`) are set in `config.yaml`, relative to `snakemake --directory`. Which rule writes what, per mode: [workflow.md](workflow.md). Each rule logs to `log_dir/{rule}/...` and writes a Snakemake `benchmark:` TSV (wall time, `max_rss`, `max_vms`, `cpu_time`, ...) to `bench_dir/{rule}/...`.
210+
Directories (`snp_dir`, `phase_dir`, `pileup_dir`, `allele_dir`, `bb_dir`, `qc_dir`, `log_dir`, `aux_dir`, `bench_dir`) are set in `config.yaml`, relative to `snakemake --directory`. Each rule logs to `log_dir/{rule}/...` and writes a Snakemake `benchmark:` TSV (wall time, `max_rss`, `max_vms`, `cpu_time`, ...) to `bench_dir/{rule}/...`.
191211

192212
`.npz` are matrices: rows = SNPs or bins, columns = samples or cells; dense for bulk, scipy sparse CSR for single-cell. BAF is never stored — derive it from `Ballele` / `Tallele`.
193213

‎docs/sample_sheet.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -47,10 +47,10 @@ Sample file is a JSON-format configuration file contains dataset records over mu
4747
| `assay_type` | Yes | `bulkWGS` \| `bulkWGS-lr` \| `bulkWES` \| `scRNA` \| `scATAC` \| `VISIUM` \| `VISIUM3prime`. |
4848
| `sample_type` | Yes | `normal` \| `tumor`. |
4949
| `files` | Yes | Input files; see [Files](#files). |
50-
| `rdr_base_dataset_id` | No | matched-normal sample `dataset_id`; see [RDR normalization](workflow.md#bulk_genotyping). |
50+
| `rdr_base_dataset_id` | No | matched-normal sample `dataset_id`; see [RDR normalization](reference.md#params_combine_counts). |
5151
| `passage_id` | No | e.g. `p23`. |
5252
| `platform` | No | e.g. `Illumina PCR-free`. |
53-
| `reference_version` | No | e.g. `GRCh38-GIABv3`; see [Workflow](workflow.md). |
53+
| `reference_version` | No | e.g. `GRCh38-GIABv3`; see [reference.md](reference.md#input-data). |
5454
| `cancer_type` | No | e.g. `PDAC`. |
5555

5656
## Files

‎docs/single_cell_genotyping.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Single-Cell Genotyping
22

3-
This documentation covers input preparation and result interpretation for single-cell and spatial genotyping using **scRNA**, **scATAC** (incl. 10x Epi Multiome), and **Visium** (`VISIUM`/`VISIUM3prime`) data to run [CalicoST](https://github.com/raphael-group/CalicoST). Germline SNPs are genotyped from a pseudobulk of the sample's own cells (cellsnp-lite), phased, then pileup-counted per cell. Refer to [Pipeline](pipeline.md) for Snakemake pipeline installation and execution instructions.
3+
This documentation covers input preparation and result interpretation for single-cell and spatial genotyping using **scRNA**, **scATAC** (incl. 10x Epi Multiome), and **Visium** (`VISIUM`/`VISIUM3prime`) data to run [CalicoST](https://github.com/raphael-group/CalicoST). Germline SNPs are genotyped from a pseudobulk of the sample's own cells (cellsnp-lite), phased, then pileup-counted per cell. Refer to the [README](../README.md) for Snakemake pipeline installation and execution instructions.
44

55
## Table of Contents
66
1. [Overview](#overview) <br>

0 commit comments

Comments
 (0)