feature request: generating call-ready input JSONs from a template

[a-frantz]

I have a Bash script I use to do this "manually", but it would be really cool to add this as an official function in sprocket. Bash script below:

#!/bin/bash
set -euo pipefail

[ $# -ne 3 ] && { echo "Usage: $0 <list_of_paths> <template_json> <out_dir>"; exit 1; }

mkdir -p $3

for sample_path in $(cat $1); do
  sample_name=$(basename $sample_path)
  cp $2 $3/"$sample_name".input.json;
  sed -i 's;foo;'"$sample_path"';g' $3/"$sample_name".input.json
done

where my template JSONs look something like:

{
  "quality_check.gtf": "path/to/gencode.v31.gtf.gz",
  "quality_check.rna": true,
  "quality_check.bam_index": "foo.bai",
  "quality_check.bam": "foo",
  "quality_check.kraken_db": "path/to/kraken2_db.tar.gz",
...
}

The workflow for using these hypothetical sprocket commands:

$ cat qc.wdl
version 1.1

workflow wf {
    input {
        File bam
        File bam_index
        File reference_fasta
        String? strandedness
    }
    ... # irrlevant
}
$ sprocket inputs --name wf qc.wdl > template.json
$ vim template.json # edit JSON to have the proper production values and paths
$ cat template.json
{
    "wf.reference_fasta": "path/to/reference.fa",
    "wf.strandedness": "Stranded-Reverse"
}
$ cat inputs.tsv
name\tbam\tbam_index
sample1\t/path/to/sample1.bam\t/path/to/sample1.bam.bai
sample2\t/path/to/sample2.bam\t/path/to/sample2.bam.bai
$ sprocket generate-inputs --name wf qc.wdl --template template.json --inputs inputs.tsv --output-dir input_jsons/
$ ls input_jsons/
sample1.json
sample2.json
$ cat input_jsons/sample1.json
{
    "wf.reference_fasta": "path/to/reference.fa",
    "wf.strandedness": "Stranded-Reverse",
    "wf.bam": "/path/to/sample1.bam",
    "wf.bam_index": "/path/to/sample1.bam.bai"
}

[a-frantz]

Some extra context here: I was talking to Andrew T about this, and I think both of us, as folks comfortable at the command line, would probably find it easier to just write some Bash one-liners to achieve this behavior rather than go through all the steps I outlined above.

But I think some of our audience for Sprocket is folks with less experience in those kinds of things, and the above behavior is probably easier to follow and copy than some "Bash-fu" that one of us might write.

Sprocket should be lowering the barrier of entry to -omics, and I view this feature as important for making mass workloads less intimidating to run.

[DeepakSilaych]

@a-frantz I’d love to work on implementing this feature. It makes sense to lower the barrier for batch input generation, and I can help translate the Bash script workflow into a sprocket command

[a-frantz]

@DeepakSilaych That's great! But I think it would be best for this feature to be implemented after some other features are merged in.
It makes sense for us to wrap up #66, #69, and #71 (which I created just now) before taking this issue on. This is because those issues and PRs can form some of the necessary building blocks someone would need for this issue.

I'd encourage you to read #71 and consider implementing that first!

[fahdfady]

@a-frantz I'd love to work on that. you can assign me. in rust i would approach it with something like:

fn generate_inputs(template: JsonValue, inputs: csv::Reader, out_dir: Path) -> Result<()> {
    for record in inputs.records() {
        let mut sample_json = template.clone(); // Start with base template
        sample_json["wf.bam"] = record["bam"].into(); // Set workflow inputs
        sample_json["wf.bam_index"] = record["bam_index"].into();
        write_json(out_dir.join(&record["name"]), &sample_json)?; // Save per-sample
    }
    Ok(())
}

I might need more guidance on TSV files, to be honest, i don't fully understand the bash script. but i'll ask you as i'm going through the implementation.