Running Picard MarkDuplicates on multiple bam files

[cnash2]

Hi everyone,

I've had a search around and can't find the answer I'm looking for.

I'm wondering if anyone can tell me if it's possible to run Picard MarkDuplicates on a batch of bam files at once? I have 13 bam files in a single directory and would like to remove duplicates from all files and output 13 new bam files with the duplicates removed.

Since this is a java based program I wasn't sure if the same rules applied as running samtools for example on a batch of files from the command line.....

[GenoMax]

Are you looking to mark duplicates from all 13 files considered together or just within individual files?

[cnash2]

Thanks for your reply. I want to remove the duplicates from each of the 13 individual files and produce 13 new individual files that have had the duplicates removed. If that makes sense.....

[GenoMax]

It would work best if you have access to a cluster. You can start 13 parallel markduplicate jobs. You can use the same procedure (shell script?) for running the jobs from the command line. You should not need to remove those duplicates (http://gatkforums.broadinstitute.org...ove-duplicates).

[cnash2]

Ah. I don't have access to a cluster right now. So I guess I'll have to do them one by one. So it's not possible to run java programs on a batch of files at once?

To note, I have to remove duplicates from these ChIP-seq files to comply with previous analyses so that I can make them comparable.

[GenoMax]

Certainly possible to run multiple. You could start all of them but then they would compete for hardware resources on your local machine and get in each others way.

If you have multiple cores available (and a fast disk, SSD) try starting 3-4 in parallel and see if they all proceed well. Watch for CPU/Disk usage.

[cnash2]

Ok. Thank you.

Let's put my mac to the test.......

[colindaven]

If you don't need to do in parallel, you can sequentially remove duplicates with a shell script like:

## Save a script.sh, run as "sh script.sh" at least on Linux with bash
## 

for i in `ls *.bam`

        do
        java -jar PicardCommandWhatever input=$i

done