Batch processing of files through tophat etc

[bjoernoest]

If i have a large amount of files i want to run through tophat, htseq and DEseq or other software, do anyone have a neat way (for example a shell script) for doing batch processing so it will start the next run when the previous have finished, so i do not have to start the programs my self with the new parameters?

Bjørn

[dpryan]

I have various shell scripts for these sorts of things (I expect that's common). Generally you would need to tailor something like that to whatever your experimental design is and your particular file structure (i.e., where the sequence and annotation files are). If you need some examples to get you started, just send me your email address in a private message and I can send a few examples to you.

[simonandrews]

For most simple batch processing I often just use a simple bash loop which will run the jobs serially. If you're going to do this a lot you might want to look into adding some error checks and reporting so you can track down what went wrong if it doesn't work. If you want to work on a larger scale you could also look at a proper queueing system, but that's a whole other level of work.

An example of one we did recently to map a pile of fastq files with bowtie would be:

for i in *.fastq.gz
do zcat $i | bowtie -f -m 1 --strata --best -p 6 --chunkmbs=512 --sam /data/public/Genomes/Mouse/NCBIM37/Mus_musculus.NCBIM37 -  > ${i}.sam
done

[bjoernoest]

Thank you both, that was just what i needed to get started