There seems to be only a few options for removing PCR duplicates from Illumina fastq data and/or alignment data. I have use FASTX for (fastx_collapse) for removing duplicates in fastq files but this takes a long time when running against human data of ~10Gb per lane.
The other option I have tried is SAMTOOLS (rmdup) but the documentation admits that it doesn't work well (or at all) with single end data. I have also noticed strange results when using this on my own data sets. (e.g. reads removed that were clearly not duplicates).
The alternative posted on the SAMTOOLS site is to use PICARD. Implementation is straightforward and uses percompiled jar files with well documented options.
I am unfortunately getting a strange Exception thrown when I run the MakeDuplicates jar file. It seems it is recognizing something in my data that wants to call it paired end data. However, my data are single end reads.
Does anyone have any experience with using this command in picard? Here is my call and the error being thrown.
Call:
Error:
It seems that it runs okay but then encounters a problem.
Thanks.
Dan
The other option I have tried is SAMTOOLS (rmdup) but the documentation admits that it doesn't work well (or at all) with single end data. I have also noticed strange results when using this on my own data sets. (e.g. reads removed that were clearly not duplicates).
The alternative posted on the SAMTOOLS site is to use PICARD. Implementation is straightforward and uses percompiled jar files with well documented options.
I am unfortunately getting a strange Exception thrown when I run the MakeDuplicates jar file. It seems it is recognizing something in my data that wants to call it paired end data. However, my data are single end reads.
Does anyone have any experience with using this command in picard? Here is my call and the error being thrown.
Call:
Code:
java -jar /home/bornmand/tools/picard/MarkDuplicates.jar INPUT=101119_first4lanes_bwa.bam OUTPUT=101119_first4lanes_bwaNoD up.bam METRICS_FILE=101119_first4lanes_duplicateINFO.txt REMOVE_DUPLICATES=true ASSUME_SORTED=true
Code:
INFO 2011-03-16 08:09:46 MarkDuplicates Start of doWork freeMemory: 62375408; totalMemory: 63111168; maxMemory: 935854080
INFO 2011-03-16 08:09:46 MarkDuplicates Reading input file and constructing read end information.
INFO 2011-03-16 08:09:46 MarkDuplicates Will retain up to 3713706 data points before spilling to disk.
INFO 2011-03-16 08:09:46 MarkDuplicates Assuming input is coordinate sorted.
[Wed Mar 16 08:09:46 EDT 2011] net.sf.picard.sam.MarkDuplicates done.
Runtime.totalMemory()=92864512
Exception in thread "main" java.lang.RuntimeException: SAM validation error: ERROR: Record 1, Read name 4:20:14143:2730:Y, MRNM should not be set for unpaired read.
at net.sf.samtools.SAMUtils.processValidationErrors(SAMUtils.java:334)
at net.sf.samtools.BAMFileReader$BAMFileIterator.advance(BAMFileReader.java:449)
at net.sf.samtools.BAMFileReader$BAMFileIterator.<init>(BAMFileReader.java:413)
at net.sf.samtools.BAMFileReader$BAMFileIterator.<init>(BAMFileReader.java:403)
at net.sf.samtools.BAMFileReader.getIterator(BAMFileReader.java:206)
at net.sf.samtools.SAMFileReader.iterator(SAMFileReader.java:288)
at net.sf.samtools.SAMFileReader.iterator(SAMFileReader.java:37)
at net.sf.picard.sam.MarkDuplicates.buildSortedReadEndLists(MarkDuplicates.java:271)
at net.sf.picard.sam.MarkDuplicates.doWork(MarkDuplicates.java:113)
at net.sf.picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:156)
at net.sf.picard.sam.MarkDuplicates.main(MarkDuplicates.java:97)
Thanks.
Dan
Comment