Human Illumina Paired-end RNA-Seq remove duplication.

[fabrice]

I am using Human Illumina Paired-end RNA-Seq. I analysis purpose is to
get expression of isoform level. Not for SNP calling.

When I used fastqc(0.94) to examin my RNA-seq data, I found that there
are very high duplication level in it. About 70% are duplication
repost by fastqc. So I tried to use Picard(1.50) to remove duplicate
reads.

The command is:

java -Xmx4g -jar ~/bin/picard/MarkDuplicates.jar REMOVE_DUPLICATES=true
INPUT=accepted_hits.bam OUTPUT=remove_accepted_hits.bam
METRICS_FILE=dup.txt

After run picard, I used fastqc to check again. It is better but it is
still have a high duplication level (63% duplication). Does it mean
picard do not work well or fastqc report have a problem?

I looked the output from Picard,
In the METRICS_FILE of picard output, the PERCENT_DUPLICATION is 0.312927.
But fastqc give the DUPLICATION level percent is 70%.

Why have this difference?


Thanks.

[Robby]

Hi fabrice,
I am not sure, but I think fastqc counts all identical reads as a duplicate. In comparision to that, picard marks only these reads as duplicates, where the position of the forward and the reverse read is the same. So for Picard it is not enough, that for example just the forward reads is the same.

[fabrice]

Thank you.

So does it mean that Fastqc more closer to the truth?

Picard worked as this:

Q: How does MarkDuplicates work?
A: Essentially what it does (for pairs; single-end data is also handled) is to find the 5' coordinates and mapping orientations of each read pair. When doing this it takes into account all clipping that has taking place as well as any gaps or jumps in the alignment. You can thus think of it as determining "if all the bases from the read were aligned, where would the 5' most base have been aligned". It then matches all read pairs that have identical 5' coordinates and orientations and marks as duplicates all but the "best" pair. "Best" is defined as the read pair having the highest sum of base qualities as bases with Q >= 15.

If your reads have been divided into separate BAMs by chromosome, inter-chromosomal pairs will not be identified, but MarkDuplicates will not fail due to inability to find the mate pair for a read.

[Robby]

I think Picard is closer to the truth. If two reads have the same sequence, it is still possible, that it isn't a PCR-duplicate. If the reverse reads of two identical forward reads are different, it is probably no PCR-duplicate. But if the position and orientation of forward and reverse read are identical it is likely a PCR-duplicate. But of course, even the percentage calculated by Picard is overestimated.

[fabrice]

Robby,
Thanks for your explain.
Here you said Picard is overestimated. Does it mean Picard always give an overestimate duplication level?
Because Picard take bam/sam as input, it means that if I want to estimate the duplication level in my sequence. I must map the reads firstly. Is it possible take the fastq file to estimate the duplication level? Fastqc take the fastq files, but it seems not very correctly.

[DZhang]

Hi,

Fastqc and picard work at different levels. Fastqc works on the read level - it takes the read sequences and estimates the duplication level. Picard works on the alignment level - as already explained, it also considers the location of the read (and its mate if applicable). So it depends what you are looking for. As far removing PCR-introduced duplicated reads, picard definitely is more relevant.

[ddaneels]

Your picard value was around 0.13. Is this 0.13% or do you still need to multiply it with 100. So that it's actually 13 % ?

[arkal]

I had the same query. What i personally feel is that Picard is more accurate.

My reassoning is that fastqc is calling around 70% duplication looking at only the read 1 file and a similar amount using the read 2 file without using any information on the mapping.
Picard on the other hand looks at teh mapping and uses both start and end regions of the fragments to call duplicates which logically makes more sense, right? Fragments starting at the same position but ending at varying ones can't be seen as duplicates and discarded IMHO!

[arkal]

I think Picard is closer to the truth. If two reads have the same sequence, it is still possible, that it isn't a PCR-duplicate. If the reverse reads of two identical forward reads are different, it is probably no PCR-duplicate. But if the position and orientation of forward and reverse read are identical it is likely a PCR-duplicate. But of course, even the percentage calculated by Picard is overestimated.

A trivial question: When you say overestimated, do u mean the duplication in his sample is LESS THAN 13% or MORE THAN 13%.