Data Pipeline Automation, what are you using?

[rdeborja]

I'm curious to know what type of pipeline workflow automation tool people are using these days. At the start of my career, I started cobbling together various shell scripts which quickly turned into an administrative nightmare. Since then, our institute has implemented SeqWare to perform FASTQ file generation, data QC, alignment and some post-processing.

It would be nice to know what others are using and some of their experiences.

[RickBioinf]

I prefer to make my own pipeline workflow, so I know exactly what the code is and I can change everything. But when it's really necessary I like to use Galaxy.

[rhinoceros]

Lots of pipes in shell scripts.

[maasha]

Biopieces -> www.biopieces.org

[rhinoceros]

Biopieces -> www.biopieces.org

That's in essence how I construct my pipelines, except that I have individually added all the algorithms to my path. So, for example, instead of 'align_pair_seq' I'd use 'muscle' or 'clustal' or whatever does IMO the best job for the given task..

[dpryan]

Bash scripts and the queuing system on our cluster.

[asiangg]

Does anyone use GNU makefiles?

[sdriscoll]

bash, perl, python and C.

[rdeborja]

Does anyone use GNU makefiles?

I'm a huge fan of makefiles as a very basic workflow system, especially since it works well with sge's qmake.