Dependencies containerized

[christianwoe]

How to get containers used by nf-core pipelines

(nextflow version 24.10.4.5934)

Option 1: Nextflow inspect

Documentation:
Bytesize: Explaining Wave containers (Minute 22+)
https://www.nextflow.io/docs/latest/reference/cli.html

Nextflow inspect provides an overview of processes, but it seems to required “dummy” input/outout definitions (Could this be solved in new version).

Here an example:
nextflow inspect nf-core/sarek -profile test,singularity --outdir dummy

Output is json and can easily be parsed, e.g., via python or jq.

Limitations/Questions:

1. Only containers of processes executed with the corresponding “run” command are included.
2. Requires running nextflow.

Option 2: Parse from github repo files

Each nf-core pipeline repo has a modules folder and each (?) module should contain a container line showing the information on container used.

Here is how it looks:

    conda "${moduleDir}/environment.yml"
    container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
        'https://community-cr-prod.seqera.io/docker/registry/v2/blobs/sha256/63/6397750e9730a3fbcc5b4c43f14bd141c64c723fd7dad80e47921a68a7c3cd21/data':
        'community.wave.seqera.io/library/bedtools_coreutils:a623c13f66d5262b' }"

Parsing via grep should be no problem.

Limitations/Questions:

1. Does this cover all container used?
2. Would this be possible without download the repo first.

[christianwoe]

Option 3.

As discussed I checked if there is another file providing overview on all the modules used.
There is the modules.json in the nf-core pipelines base directories, which contains all the modules plus git hash allowing to track it in the modules repo.
However, there are limitations:

1. To get information on containers used we need to check the modules github repo as well
2. I did not find the "local" modules included.
   Currently option 2 seems to be most promising, but further checking is required.

[christianwoe]

Option 4.

nf-core pipelines download
This relies on nextflow inspect. See Option 1 and for details: https://nf-co.re/docs/nf-core-tools/pipelines/download#how-pipeline-containers-are-found

[christianwoe]

Currently option 2 is most promising where main.nf files in the in the modules subfolders are searched after cloning the repo.
Here a preliminary code example:

import os
import re
import argparse

def find_main_nf_files(folder_path):
    """Recursively find all 'main.nf' files in the given directory."""
    main_nf_files = []
    for root, _, files in os.walk(folder_path):
        for file in files:
            if file == 'main.nf':
                main_nf_files.append(os.path.join(root, file))
    return main_nf_files

def extract_process_info(file_path):
    """Extract process names and check for 'container' in each 'main.nf' file."""
    with open(file_path, 'r') as file:
        text = file.read()

    # Find all process names
    process_names = re.findall(r'process\s+([^\s{]+)\s*{', text)
    # Check if 'container' is present in the file
    container_present = any(re.search(r'^\s*container', line) for line in text.splitlines())

    return process_names, container_present

def main():
    parser = argparse.ArgumentParser(
        description='Extract process names and check for containers in main.nf files.',
        usage='python process_extractor.py <folder> [--format list|summary]'
    )
    parser.add_argument('folder', type=str, help='The folder to search for main.nf files.')
    parser.add_argument('--format', type=str, choices=['list', 'summary'], default='list',
                        help='Output format: "list" for detailed output, "summary" for count summary.')

    args = parser.parse_args()
    main_nf_files = find_main_nf_files(args.folder)
    total_processes = 0
    processes_with_container = 0
    process_details = []

    for file_path in main_nf_files:
        process_names, container_present = extract_process_info(file_path)
        total_processes += len(process_names)
        if container_present:
            processes_with_container += len(process_names)

        # Collect process details
        for process in process_names:
            process_details.append((process, 1 if container_present else 0))

    if args.format == 'list':
        for process, container_present in process_details:
            print(f"{process},{container_present}")
    elif args.format == 'summary':
        print(f"{total_processes},{processes_with_container}")

if __name__ == '__main__':
    main()

Limitations:

1. String search for "container" only likely causes false-positives
2. Extraction of container names not included yet.

[christianwoe]

To prevent downloading the complete repo alternatively we can extract a list of main.nf files via API and extract only those.

This could be the first step (Here nf-core/rnaseq; Thanks to Björn for sharing his solution):

curl --request GET \
--url "https://api.github.com/repos/nf-core/rnaseq/git/trees/master?recursive=1" \
--header "Accept: application/vnd.github+json" \
--header "X-GitHub-Api-Version: 2022-11-28" | jq -r '[.tree[] | select(.type=="blob" and (.path | startswith("modules/") and endswith("main.nf")))]'

Afterwards we need to screen through all the resulting files.
TBD

[mashehu]

don't do 2. parsing these is a paaaaaain and caused a lot of headache in nf-core pipeline downloads code