markduplicates strange results?

[shuoguo]

i have a small bam for testing and found that before and after deduplication, the number reads does not seem to match.
I used the most recent picard tool kit.

before remove duplication, there are 178306 reported duplications
after remove duplication, there are 0

but, the total number of reads just dropped 1586 (453052 - 451466).

can anyone give any insights?

$ samtools flagstat 5.bam
453052 + 0 in total (QC-passed reads + QC-failed reads)
178306 + 0 duplicates
447531 + 0 mapped (98.78%:-nan%)
453052 + 0 paired in sequencing
226526 + 0 read1
226526 + 0 read2
437506 + 0 properly paired (96.57%:-nan%)
443874 + 0 with itself and mate mapped
3657 + 0 singletons (0.81%:-nan%)
5006 + 0 with mate mapped to a different chr
4378 + 0 with mate mapped to a different chr (mapQ>=5)

$ samtools flagstat 5.dedup.bam
451466 + 0 in total (QC-passed reads + QC-failed reads)
0 + 0 duplicates
445945 + 0 mapped (98.78%:-nan%)
451466 + 0 paired in sequencing
225684 + 0 read1
225782 + 0 read2
436046 + 0 properly paired (96.58%:-nan%)
442404 + 0 with itself and mate mapped
3541 + 0 singletons (0.78%:-nan%)
5002 + 0 with mate mapped to a different chr
4374 + 0 with mate mapped to a different chr (mapQ>=5)

[dpryan]

The question then becomes exactly how you removed the duplicates.

[Brian Bushnell]

I don't know the answer to your question, but it looks like after duplicate removal there is a different number of read1's than read2's. That's not good! If two reads are PCR duplicates, then obviously their mates will be as well. I don't know if that behavior is intentional or a bug (I don't use Picard), but I would not want to do that to my data.

[dpryan]

@Brian: Note that there's a change in singletons as well.

[shuoguo]

Oh I missed that part. Does Picard actually remove reads?
I set the remove reads to be true though.

This bam is subsampling from a large one at .5% ratio using samtools.
Could samtools not sampling the paired reads?

Thanks and I will post my cmd once I have my computer...

I don't know the answer to your question, but it looks like after duplicate removal there is a different number of read1's than read2's. That's not good! If two reads are PCR duplicates, then obviously their mates will be as well. I don't know if that behavior is intentional or a bug (I don't use Picard), but I would not want to do that to my data.

[Wort John]

@Brian: Note that there's a change in singletons as well.

What is the singleton sequence ? It means the single read ? Can you explain it in detail ? Thank you !

[Brian Bushnell]

In this context, a singleton is a pair in which only one of the reads mapped.

[juanita]

I have a similar issue because after removing duplicates with Picard, I ran ValidateSamFile and I got a substantial proportion of MATE_NOT_FOUND reads.

Before removing duplicates (with ValidateSamFile):
"No errors found"

After removing duplicates:
"ERROR:MATE_NOT_FOUND 1072882"