Data Pipeline Automation, what are you using?

[rdeborja]

Does anyone use GNU makefiles?

I'm a huge fan of makefiles as a very basic workflow system, especially since it works well with sge's qmake.

[sdriscoll]

bash, perl, python and C.

[asiangg]

Does anyone use GNU makefiles?

[dpryan]

Bash scripts and the queuing system on our cluster.

[rhinoceros]

Biopieces -> www.biopieces.org

That's in essence how I construct my pipelines, except that I have individually added all the algorithms to my path. So, for example, instead of 'align_pair_seq' I'd use 'muscle' or 'clustal' or whatever does IMO the best job for the given task..

[maasha]

Biopieces -> www.biopieces.org

[rhinoceros]

Lots of pipes in shell scripts.

[RickBioinf]

I prefer to make my own pipeline workflow, so I know exactly what the code is and I can change everything. But when it's really necessary I like to use Galaxy.

[rdeborja]

I'm curious to know what type of pipeline workflow automation tool people are using these days. At the start of my career, I started cobbling together various shell scripts which quickly turned into an administrative nightmare. Since then, our institute has implemented SeqWare to perform FASTQ file generation, data QC, alignment and some post-processing.

It would be nice to know what others are using and some of their experiences.