Analysis pipeline tools: make? bpipe? snakemike

[jake13]

I'm interested in putting some of my analysis into an easy to run pipeline. It seems like a make file is the way to go. However, it seems like people have written a number of tools such as snakemake and bpipe to make it easier to run bioinformatic pipeline scripts. I was just wondering if anyone is using any of these and which one is the best?
Thanks

[lindenb]

I use make because

* I already know the syntax
* it fulfills my needs
* it is parallelized (option -j)
* cluster managers like SGE or SLURM have a specialized version of make


my problem with make is 'choosing': telling 'make' at runtime that if, e.g, "a Fastq QC failed, then it shouldn't map the reads for this sample"

[colindaven]

bpipe is great, just make sure the files are clearly named at the start.

The linux utility "rename" is nice for batch renaming files.

[jmw86069]

I'll give a quick shout out to Bpipe, I use it and recommend it. I have limited experience with make, but don't think I could lean on it as heavily as I do Bpipe to handle my workflows. If nothing else, I like the ability to define a library of named Bpipe methods, then just chain them together by name into a pipeline. You can use some clever parentheses to parallelize a certain subset of steps, then have subsequent steps await the results before continuing. Also, it does useful things like keeping files representing paired-end reads together throughout the workflow.

https://code.google.com/p/bpipe/

To address another important criterion when selecting this type of tool, the authors are very active in the discussion group, and are quick to implement sensible updates per feedback and requests. See their Google Groups entry: https://groups.google.com/forum/#!forum/bpipe-discuss

My only critique is that sometimes the tips and tricks are still buried in the discussions and not fully described on their FAQ. However, their FAQ is still quite good.