Skip to content

Commit f8351a6

Browse files
committed
Merge remote-tracking branch 'origin/main' into 461-make-updates-to-work-with-storage-plugins
2 parents c57723f + 9c8f1ca commit f8351a6

27 files changed

+234
-262
lines changed

.github/PULL_REQUEST_TEMPLATE.md

-1
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,5 @@
55

66
If this is for a release:
77
* [ ] I have updated documentation
8-
* [ ] I have updated all conda pin files (`snakedeploy pin-conda-envs workflow/envs/*.yml`)
98
* [ ] I have updated the hardcoded version at the top of `install.sh` to match what this release's version will be
109
* [ ] I have created a release archive that will be attached to this release (`bash dev_scripts/generate_archive.sh`)

.github/workflows/dockerhub.yml

+71-6
Original file line numberDiff line numberDiff line change
@@ -2,19 +2,44 @@ name: Push to DockerHub
22

33
on:
44
workflow_call:
5+
inputs:
6+
version:
7+
description: "Sunbeam version (make sure it has the leading 'v')"
8+
required: true
9+
type: string
10+
outputs:
11+
sunbeam_package_versions:
12+
description: "Versions in main Sunbeam image"
13+
value: ${{ jobs.push-dockerhub.outputs.package_versions }}
14+
sunbeam_package_versions_slim:
15+
description: "Versions in slim Sunbeam image"
16+
value: ${{ jobs.push-dockerhub.outputs.package_versions_slim }}
17+
cutadapt_package_versions:
18+
description: "Versions in cutadapt image"
19+
value: ${{ jobs.push-cutadapt-dockerhub.outputs.package_versions }}
20+
komplexity_package_versions:
21+
description: "Versions in komplexity image"
22+
value: ${{ jobs.push-komplexity-dockerhub.outputs.package_versions }}
23+
qc_package_versions:
24+
description: "Versions in qc image"
25+
value: ${{ jobs.push-qc-dockerhub.outputs.package_versions }}
26+
reports_package_versions:
27+
description: "Versions in reports image"
28+
value: ${{ jobs.push-reports-dockerhub.outputs.package_versions }}
529

630
workflow_dispatch:
731

832
jobs:
933
push-dockerhub:
1034
name: Push to Dockerhub
1135
runs-on: ubuntu-latest
36+
37+
outputs:
38+
package_versions: ${{ steps.report.outputs.package_versions }}
39+
package_versions_slim: ${{ steps.report.outputs.package_versions_slim }}
1240

1341
steps:
1442
- uses: actions/checkout@v4
15-
16-
- id: get_version
17-
uses: battila7/get-version-action@v2
1843

1944
- name: Log in to Docker Hub
2045
uses: docker/login-action@v3
@@ -58,21 +83,32 @@ jobs:
5883
context: .
5984
file: ./Dockerfile
6085
push: true
61-
tags: sunbeamlabs/sunbeam:latest, sunbeamlabs/sunbeam:${{ steps.get_version.outputs.version-without-v }}
86+
tags: sunbeamlabs/sunbeam:latest, sunbeamlabs/sunbeam:${{ inputs.version }}
6287
labels: ${{ steps.meta.outputs.labels }}
88+
build-args: "SUNBEAM_VER=${{ inputs.version }}"
6389

6490
- name: Build and push slim Docker image
6591
uses: docker/build-push-action@v5
6692
with:
6793
context: .
6894
file: ./slim.Dockerfile
6995
push: true
70-
tags: sunbeamlabs/sunbeam:${{ steps.get_version.outputs.version-without-v }}-slim
96+
tags: sunbeamlabs/sunbeam:${{ inputs.version }}-slim
7197
labels: ${{ steps.meta.outputs.labels }}
98+
build-args: "SUNBEAM_VER=${{ inputs.version }}"
99+
100+
- name: Generate image report
101+
id: report
102+
run: |
103+
echo "package_versions=$(docker run --rm sunbeamlabs/sunbeam:${{ inputs.version }} cat installed_packages.txt)" >> "$GITHUB_OUTPUT"
104+
echo "package_versions_slim=$(docker run --rm sunbeamlabs/sunbeam:${{ inputs.version }}-slim cat installed_packages.txt)" >> "$GITHUB_OUTPUT"
72105
73106
push-cutadapt-dockerhub:
74107
name: Push cutadapt env to Dockerhub
75108
runs-on: ubuntu-latest
109+
110+
outputs:
111+
package_versions: ${{ steps.report.outputs.package_versions }}
76112

77113
steps:
78114
- uses: actions/checkout@v4
@@ -98,9 +134,17 @@ jobs:
98134
tags: ${{ steps.meta.outputs.tags }}
99135
labels: ${{ steps.meta.outputs.labels }}
100136

137+
- name: Generate image report
138+
id: report
139+
run: |
140+
echo "package_versions=$(docker run --rm sunbeamlabs/cutadapt:${{ inputs.version }} cat installed_packages.txt)" >> "$GITHUB_OUTPUT"
141+
101142
push-komplexity-dockerhub:
102143
name: Push komplexity env to Dockerhub
103144
runs-on: ubuntu-latest
145+
146+
outputs:
147+
package_versions: ${{ steps.report.outputs.package_versions }}
104148

105149
steps:
106150
- uses: actions/checkout@v4
@@ -126,9 +170,17 @@ jobs:
126170
tags: ${{ steps.meta.outputs.tags }}
127171
labels: ${{ steps.meta.outputs.labels }}
128172

173+
- name: Generate image report
174+
id: report
175+
run: |
176+
echo "package_versions=$(docker run --rm sunbeamlabs/komplexity:${{ inputs.version }} cat installed_packages.txt)" >> "$GITHUB_OUTPUT"
177+
129178
push-qc-dockerhub:
130179
name: Push qc env to Dockerhub
131180
runs-on: ubuntu-latest
181+
182+
outputs:
183+
package_versions: ${{ steps.report.outputs.package_versions }}
132184

133185
steps:
134186
- uses: actions/checkout@v4
@@ -154,9 +206,17 @@ jobs:
154206
tags: ${{ steps.meta.outputs.tags }}
155207
labels: ${{ steps.meta.outputs.labels }}
156208

209+
- name: Generate image report
210+
id: report
211+
run: |
212+
echo "package_versions=$(docker run --rm sunbeamlabs/qc:${{ inputs.version }} cat installed_packages.txt)" >> "$GITHUB_OUTPUT"
213+
157214
push-reports-dockerhub:
158215
name: Push reports env to Dockerhub
159216
runs-on: ubuntu-latest
217+
218+
outputs:
219+
package_versions: ${{ steps.report.outputs.package_versions }}
160220

161221
steps:
162222
- uses: actions/checkout@v4
@@ -180,4 +240,9 @@ jobs:
180240
file: workflow/envs/reports.Dockerfile
181241
push: true
182242
tags: ${{ steps.meta.outputs.tags }}
183-
labels: ${{ steps.meta.outputs.labels }}
243+
labels: ${{ steps.meta.outputs.labels }}
244+
245+
- name: Generate image report
246+
id: report
247+
run: |
248+
echo "package_versions=$(docker run --rm sunbeamlabs/reports:${{ inputs.version }} cat installed_packages.txt)" >> "$GITHUB_OUTPUT"

.github/workflows/release.yml

+25-15
Original file line numberDiff line numberDiff line change
@@ -5,14 +5,19 @@ on:
55
types: [published]
66

77
workflow_dispatch:
8-
8+
99
jobs:
1010
run-tests:
1111
uses: ./.github/workflows/tests.yml
1212
secrets: inherit
1313

1414
check-version:
1515
runs-on: ubuntu-latest
16+
17+
outputs:
18+
version-without-v: ${{ steps.get_version.outputs.version-without-v }}
19+
version: ${{ steps.get_version.outputs.version }}
20+
1621
steps:
1722
- name: Checkout
1823
uses: actions/checkout@v4
@@ -24,16 +29,6 @@ jobs:
2429
GIT_VERSION=$(bash install.sh -w)
2530
echo "GIT_VERSION=$GIT_VERSION" >> $GITHUB_ENV
2631
27-
- name: Download and Extract
28-
shell: bash
29-
run: |
30-
wget https://github.com/sunbeam-labs/sunbeam/releases/latest/download/sunbeam.tar.gz
31-
mkdir tar_sunbeam
32-
tar -zxf sunbeam.tar.gz -C tar_sunbeam
33-
cd tar_sunbeam
34-
TAR_VERSION=$(bash install.sh -w)
35-
echo "TAR_VERSION=$TAR_VERSION" >> $GITHUB_ENV
36-
3732
- id: get_version
3833
uses: battila7/get-version-action@v2
3934

@@ -42,9 +37,8 @@ jobs:
4237
RELEASE_VERSION=${{ steps.get_version.outputs.version-without-v }}
4338
echo "Release version: ${RELEASE_VERSION}"
4439
echo "GitHub version: ${{ env.GIT_VERSION }}"
45-
echo "Tarball version: ${{ env.TAR_VERSION }}"
4640
47-
if [[ $RELEASE_VERSION == ${{ env.GIT_VERSION }} ]] && [[ ${{ env.GIT_VERSION }} == ${{ env.TAR_VERSION }} ]]; then
41+
if [[ $RELEASE_VERSION == ${{ env.GIT_VERSION }} ]]; then
4842
echo "Versions match, continuing..."
4943
else
5044
echo "Versions don't match, exiting..."
@@ -60,12 +54,12 @@ jobs:
6054

6155
steps:
6256
- name: setup-conda
63-
uses: s-weigand/[email protected]
57+
uses: s-weigand/[email protected]
6458

6559
- name: Download and Extract
6660
shell: bash
6761
run: |
68-
wget https://github.com/sunbeam-labs/sunbeam/releases/latest/download/sunbeam.tar.gz
62+
wget https://github.com/sunbeam-labs/sunbeam/releases/download/${{ needs.check-version.outputs.version }}/sunbeam.tar.gz
6963
mkdir sunbeam
7064
tar -zxf sunbeam.tar.gz -C sunbeam
7165
cd sunbeam && ./install.sh -e sunbeam
@@ -79,10 +73,26 @@ jobs:
7973
8074
push-dockerhub:
8175
uses: ./.github/workflows/dockerhub.yml
76+
with:
77+
version: ${{ needs.check-version.outputs.version }}
8278
secrets: inherit
8379
needs:
8480
- run-tests
8581
- check-version
82+
83+
udpate-release-notes:
84+
name: Update release notes
85+
runs-on: ubuntu-latest
86+
needs: push-dockerhub
87+
88+
steps:
89+
- name: Add Image Reports to Release
90+
uses: irongut/[email protected]
91+
with:
92+
token: ${{ secrets.GITHUB_TOKEN }}
93+
id: ${{ github.event.release.id }}
94+
body: "**sunbeamlabs/sunbeam**: ${{ needs.push-dockerhub.outputs.sunbeam_package_versions }}\n**sunbeamlabs/sunbeam:slim**: ${{ needs.push-dockerhub.outputs.sunbeam_package_versions_slim }}\n**sunbeamlabs/cutadapt**: ${{ needs.push-dockerhub.outputs.cutadapt_package_versions }}\n**sunbeamlabs/komplexity**: ${{ needs.push-dockerhub.outputs.komplexity_package_versions }}\n**sunbeamlabs/qc**: ${{ needs.push-dockerhub.outputs.qc_package_versions }}\n**sunbeamlabs/reports**: ${{ needs.push-dockerhub.outputs.reports_package_versions }}"
95+
replacebody: false
8696

8797
run-integration-tests:
8898
uses: ./.github/workflows/integration-tests.yml

Dockerfile

+5-3
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ FROM condaforge/mambaforge:latest
22

33
# Setup
44
WORKDIR /home/sunbeam
5+
ARG SUNBEAM_VER
56

67
RUN mkdir -p etc/
78
COPY etc/* etc/
@@ -21,12 +22,12 @@ COPY environment.yml install.sh MANIFEST.in pyproject.toml pytest.ini README.md
2122
# Install sunbeam
2223
RUN apt-get update && \
2324
apt-get upgrade -y && \
24-
apt-get install -y git
25+
apt-get install -y git vim
2526
RUN ./install.sh -e sunbeam -v
2627

2728
ENV PATH="/opt/conda/envs/sunbeam/bin/:${PATH}"
2829
ENV SUNBEAM_DIR="/home/sunbeam"
29-
ENV SUNBEAM_VER="4.4.0"
30+
ENV SUNBEAM_VER ${SUNBEAM_VER}
3031
ENV SUNBEAM_MIN_MEM_MB="8000"
3132
ENV SUNBEAM_MIN_RUNTIME="60"
3233

@@ -38,7 +39,8 @@ RUN rm -r projects
3839

3940
# "Activate" the environment
4041
SHELL ["conda", "run", "-n", "sunbeam", "/bin/bash", "-c"]
41-
#RUN bash /opt/conda/envs/sunbeam/etc/conda/activate.d/env_vars.sh
42+
43+
RUN echo "Python: $(python --version), Snakemake: $(snakemake --version), Conda: $(conda --version)" > installed_packages.txt
4244

4345
# Run
4446
CMD "bash"

README.md

+2-1
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44

55
[![Tests](https://github.com/sunbeam-labs/sunbeam/actions/workflows/pr.yml/badge.svg)](https://github.com/sunbeam-labs/sunbeam/actions/workflows/pr.yml)
66
[![Documentation Status](https://readthedocs.org/projects/sunbeam/badge/?version=stable)](https://sunbeam.readthedocs.io/en/stable/?badge=stable)
7+
[![Release](https://img.shields.io/github/release/sunbeam-labs/sunbeam.svg?style=flat)](https://github.com/sunbeam-labs/sunbeam/releases/latest)
78
[![DockerHub](https://img.shields.io/docker/pulls/sunbeamlabs/sunbeam)](https://hub.docker.com/repository/docker/sunbeamlabs/sunbeam/)
89
[![DOI:10.1186/s40168-019-0658-x](https://img.shields.io/badge/Published%20in-Microbiome-1abc9c.svg)](https://doi.org/10.1186/s40168-019-0658-x)
910

@@ -15,7 +16,7 @@ Sunbeam currently automates the following tasks:
1516
* Taxonomic assignment of reads to databases using [Kraken](https://github.com/DerrickWood/kraken) ([sbx_kraken](https://github.com/sunbeam-labs/sbx_kraken));
1617
* Assembly of reads into contigs using [Megahit](https://github.com/voutcn/megahit) ([sbx_assembly](https://github.com/sunbeam-labs/sbx_assembly));
1718
* Contig annotation using BLAST[n/p/x] and Diamond ([sbx_assembly](https://github.com/sunbeam-labs/sbx_assembly));
18-
* Mapping to reference genomes ([sbx_mapping](https://))
19+
* Mapping to reference genomes ([sbx_mapping](https://github.com/sunbeam-labs/sbx_mapping))
1920
* ORF prediction using [Prodigal](https://github.com/hyattpd/Prodigal) ([sbx_assembly](https://github.com/sunbeam-labs/sbx_assembly)).
2021

2122
More extensions can be found at https://github.com/sunbeam-labs.

dev_scripts/generate_and_push_docker_image.sh

+5-6
Original file line numberDiff line numberDiff line change
@@ -3,13 +3,12 @@
33
# NOT YET AT THE POINT WHERE IT SHOULD BE RUN AS A SCRIPT
44
# THIS SHOULD JUST PROVIDE GUIDANCE FOR DOING IT MANUALLY
55

6-
sunbeam init --data_fp tests/data/reads/ projects/docker/
7-
sunbeam run --profile projects/docker/ --containerize > Dockerfile
8-
9-
docker build -f Dockerfile -t sunbeam4.2.0 .
6+
docker build -f Dockerfile -t sunbeamlabs/sunbeam:4.2.0 .
107

118
docker images # Find image ID in output
129

13-
docker tag image_id ctbushman/sunbeam:4.2.0
10+
docker tag image_id sunbeamlabs/sunbeam:4.2.0 # If not already tagged properly
11+
12+
docker run --rm -it sunbeamlabs/sunbeam:4.2.0 /bin/bash # Test the image
1413

15-
docker push ctbushman/sunbeam:4.2.0
14+
docker push sunbeamlabs/sunbeam:4.2.0

docs/examples.rst

+52-1
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ Now you're ready to run the project, but you want to be able to put the main nod
4343
#SBATCH --no-requeue
4444
#SBATCH --output=slurm_%x_%j.out
4545
46-
# Conda env must be activate for this to work
46+
# Conda env must be active for this to work
4747
set -x
4848
set -e
4949
sunbeam run --profile /projects/my_project all_assembly
@@ -108,6 +108,57 @@ You should now be able to see all the coverage reports and other outputs from th
108108

109109
Including any other database in a containerized run is as simple as mounting the database directory into the container and pointing to it in the config file. For example, if you have a kraken database in ``/kraken_db`` and you want to use it in a containerized run, you would add ``-v /kraken_db:/kraken_db`` to the ``docker run`` command and then set the ``kraken_db_fp`` parameter in the config file to ``/kraken_db`` (after installing ``sbx_kraken``).
110110

111+
Using Slurm with the Containerized Install and Containerized Environments
112+
-------------------------------------------------------------------------
113+
114+
This is a combination of all of the above. You have a dataset on your institution's HPC (which uses slurm to manage jobs) and you want to run sunbeam to automate qc, decontam, and assembly of metagenomic contigs. Your data is paired end and lives in a directory called ``/data``. You have singularity available on the cluster and you want to use containerized environments (apptainer or similar solutions also work). Run:
115+
116+
.. code-block:: bash
117+
118+
### FIND AND LOAD SINGULARITY MODULE IF IT ISN'T ALREADY LOADED ###
119+
modulefiles_list | grep singularity
120+
module load singularity
121+
122+
export SINGULARITY_TMPDIR=/path/to/tmpdir
123+
export SINGULARITY_CACHEDIR=/path/to/cachedir
124+
singularity shell -B /data:/data,/projects:/projects docker://sunbeamlabs/sunbeam
125+
126+
### WITHIN THE CONTAINER ###
127+
sunbeam extend https://github.com/sunbeam-labs/sbx_assembly
128+
sunbeam init --data_fp /data/ --profile slurm /projects/my_project/
129+
pip install snakemake-executor-plugin-slurm
130+
131+
### BACK OUTSIDE THE CONTAINER ###
132+
vi /projects/my_project/config.yaml # edit the config file to make any desired changes (including switching ``software-deployment-method`` to ``apptainer``)
133+
134+
.. note::
135+
136+
Setting the ``SINGULARITY_TMPDIR`` environment variable may be necessary to avoid a bug in singularity that causes it to fail when running snakemake. The path should be a directory that is writable by the user running the container and large enough not to run out of space.
137+
138+
Now you're ready to run the project, but you want to be able to put the main node on the cluster as well so you can logoff, so you create a bash script called ``run_sunbeam.sh``:
139+
140+
.. code-block:: bash
141+
142+
#!/bin/bash
143+
#SBATCH --time=72:00:00
144+
#SBATCH -n 1
145+
#SBATCH --mem=8G
146+
#SBATCH --mail-type=END,FAIL
147+
#SBATCH --mail-user=your_email@your_institution.edu
148+
#SBATCH --no-requeue
149+
#SBATCH --output=slurm_%x_%j.out
150+
151+
# Conda env must be active for this to work
152+
set -x
153+
set -e
154+
singularity run -B /data:/data,/projects:/projects docker://sunbeamlabs/sunbeam sunbeam run --profile /projects/my_project all_assembly
155+
156+
Then you submit the job:
157+
158+
.. code-block:: bash
159+
160+
sbatch run_sunbeam.sh
161+
111162
Running on AWS Batch with AWS S3 Data
112163
======================================
113164

docs/structure.rst

+1-1
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ workflow/envs/
3131
-----
3232
This directory contains ``.yml`` files defining environments that will be managed by snakemake as it runs. Anywhere that a rule is defined with ``conda: /path/to/ENV_NAME.yml``, when snakemake reaches that rule, that environment will be created if it doesn't exist already and then activated while running the rule. These environments are created in ``sunbeam/.snakemake/`` by default.
3333

34-
The accompanying files named something like ``ENV_NAME.ARCH.pin.txt`` are generated with ``snakedeploy``. They list all the packages and exact versions in a given environment (for the architecture they were generated on, e.g. linux-64) so that snakemake can first try to use that exact environment and only if it fails, try to solve the ``.yml`` file for itself.
34+
There may be accompanying files named something like ``ENV_NAME.ARCH.pin.txt`` which are generated with ``snakedeploy``. They list all the packages and exact versions in a given environment (for the architecture they were generated on, e.g. linux-64) so that snakemake can first try to use that exact environment and only if it fails, try to solve the ``.yml`` file for itself.
3535

3636
extensions/
3737
-----------

0 commit comments

Comments
 (0)