De-duplication of shot-gun metagenomics sequence data

[ademenez]

Hello,

I have several metagenomic datasets (from soil, animal faeces and river water samples) which were all generated using Illumina HiSeq, MiSeq and NovaSeq. In some of them, I detected some sequence duplication, according to FastQC (for example, peaks of duplication between 9->500 duplication level, with 22% of sequences remaining if de-duplicated). When using fastp, the level of duplication detected is lower, but still present (9% for example). I am confused as to whether I should de-duplicate these sequence files or not before analysis. I contacted the administrators at the European Nucleotide Archive (ENA-MGNify) and they were not concerned about duplication. However, is it not unlikely that diverse microbial communities would generate sequences that are 100% identical? Is de-duplication and standard procedure before metagenomic sequence analyses? Or would it be better to keep the duplicates for the first steps of analyses, and remove them optionally for downstream analyses?

Thank you,

Alex

[jorondo]

Far from being a specialist in the field (but aspiring to be!), I am asking myself the same kind of question : why and when would we need to deduplicate (or dereplicate, a synonym I think)

I am mostly posting to see what others will have to say on that matter. Also, this paper (in quality control section) summarises the stakes of deduplication.

[ademenez]

Thanks for the reply. I also think that dereplication and de-duplication basically mean the same thing. Having seen the paper you suggested, it is still unclear to me whether identical reads should be removed. I am not sure these days but my understanding is that most sequencing library preps use some PCR amplification, and for me this would be a more likely source of duplicates than natural duplicates. So my guess is that duplicates in a shot gun metagenomic dataset will be a combination of natural and artificial duplicates, with the proportion of artificial duplicates increasing with the amount of amplification that was carried out. It would be great to know what others in the community think!

A