What tool(s) do you use to manage data analysis pipelines?

[gvwilson]

We all have our favorite tools for getting, crunching, and visualizing data; what I'd like to know is, how do you stitch them together? Do you write a shell script? Do you use a Makefile? Do you drive everything from Python, Perl, or R (and if so, how do you handle tools written in other languages)? Do you use web-based workflow tools, and if so, which ones?

[selik]

In general, I prototype everything in an IPython Notebook, then gradually refactor into Python scripts that can chain together via command line pipes. I like to end up with something like this:

$ python collect.py | python wrangle.py | python analyze.py | python report.py

If useful, I'll break out longer parsing, wrangling, and analysis code into their own libraries.

[LJWilliams]

I tend to do my data visualization in R most of the time. I like that it makes pretty graphs without too much work. Sometimes I will use Python, but the final graphs from R are generally more attractive. If I need to stitch things together I tend to use a shell script.

[sveme]

For bioinformatics I started out using shell scripts, however, I'm trying to transfer most of my workflows to julia as they provide a neat way for shelling out commands, something that is unfortunately really big in bioinformatics. Something like this:

using Report, JSON, Gadfly, DataFrames

picardmultiple = `java -jar -Xmx4g $picard/CollectMultipleMetrics.jar INPUT=$infile OUTPUT=$outfile ASSUME_SORTED=false REFERENCE_SEQUENCE=$refseq`

run(picardmultiple)

Nasty, I know, but that's bioinformatics I guess. You construct shell commands using the backtick operator then run it or, and that's the major advantage, detach it or run it in some subprocess.

[gvwilson]

@selik What do you then use to manage those pipelines? Do you store each one in a single-line shell script with a descriptive name for later re-use? Or do you trust yourself to reconstruct the necessary pipelines on the fly?

[gvwilson]

@LJWilliams @sveme So if one of your input data files changes, or if you tweak a parameter, you regenerate affected results manually?

[jkitzes]

I do basically what I teach here, wrapped in some exploratory analysis. The process goes sort of like this (although the real world is, of course, messy and iterative, not linear like the below) -

1. Look at everything in a IPython notebook, use pylab magic to quickly look at the data and make/examine graphs.
2. Once I nail down analysis steps, refactor main "scientific" functions/classes into a python module. Add scripts for do_analysis.py (loads module, loads data file, crunches numbers and saves output results in some format) and a make_tables.py and/or make_figs.py file (loads output results from do_analysis and makes attractive).
3. Use a controller runall (shell or Python) to run all scripts, in order, and save into a results directory. In some sense this is the pipeline part, although it's often pretty lightweight.
4. Write manuscript in LaTeX - every time it compiles, it loads updated figures. (Tables are a problem that I've never effectively solved.)

If my inputs change or parameters change, I just change them and run the runall again. This requires, of course, that the entire pipeline not take too long - if it does, I break up the steps and rerun those that change (manually - a Makefile works in principle here, but I don't generally bother). As a failsafe, I always delete my entire results directory and rerun everything before writing/submitting results somewhere, just to make sure I haven't mucked something up.

For alternate simultaneous parameter sets, I'll create multiple results subdirectories and basically run the entire set of steps above within each subdirectory, loading the appropriate parameter file each time. This process is controlled by runall.

[LJWilliams]

At first I regenerate manually. Once I'm happy with the pipeline I try to put the elements of the script that change into functions and parameters to run the code from the command line. This last step doesn't always happen (even though it should).

[selik]

@gvwilson I make extensive use of argparse to document usage. Like @jkitzes, I store intermediate steps as files and could make use of Makefiles in principle. I rely on memory to track what needs to be re-run.

[sveme]

@gvwilson No, bioinformatics data often come in files that I put into one folder with a certain naming scheme. I then run the whole analysis or parts of it on the whole folder or on files with certain tags. I have written some functions to parse the (file) output from the commands (such as picard, samtools) into julia and then create plots or tables. I also started to write a small julia package Report.jl that generates Markdown documents with tables and figures within my workflow scripts. In the end I run pandoc to create a nicer looking pdf or odt file.

All in all, quite similar to @jkitzes workflow.

[DamienIrving]

@gvwilson My data analysis involves a mix of python scripts (that parse the command line using argparse) and numerous command line utilities that manipulate netCDF files, most of which are specific to the weather/climate sciences. To stitch them all together I use Make. In fact, I've found your seed teaching material on Make to be very useful.

[karinlag]

@gvwilson I usually end up making python functions for each step in my analysis, and end up with a wrapper script that runs the whole shebang. I can then, depending on the complexity of things, add sanity checking for each step, so that it doesn't just run ahead with bad results. While developing, I usually work with a small test set to help with debugging. If there are important parameters that could/should change, I usually end up adding a config file too.

[TomRoche]

@DamienIrving "numerous command line utilities that manipulate netCDF files"

Try NCL: basically a "little language" for netCDF (plus some dubious graphical utilities). Friendly syntax, plus (importantly, IMHO) a REPL (unfortunately not as good as Python's or R's).

For too long I wrangled netCDF using NCO and various IOAPI CLUs, then switched to David Pierce's R packages (notably ncdf4, which I still use whenever getting in and out of R is more pain than gain). But for straight-up netCDF twiddling (put this in, take this out, simple math), NCL rules. Plus it has Python API.

[dpshelio]

In astrophysics we have a lot of tools that communicate each other through SAMP Protocol. Through this protocol one can transfer directly tables from a website (like from vizier catalogue) to a table visualiser (eg. topcat) with just one click. After some merging/filtering there with other tables I could transfer it to python (with astropy). The same workflow can be done with images connecting different imaging software (eg. aladin)

Also, there's a plugin for Taverna (a workflow management system) that it connects through SAMP to all these tools.

The issue/flaw I see in this is the difficulty of reproducibility of the whole process. I find it awesome for testing things quickly, because they are all connected in a seamlessly way. However, if I would have to save it all for publishing my results or to repeat it later I probably would write it all in a python script (I feel identified with this blog post).

[mkcor]

At work we use cron.
My intermediate outputs are CSV files so they can be inputs to programs in either language (R or Python, in my case).