redundancy in de novo transcriptome assembly

[gevielr]

Hi everyone,

I'm currently looking at ways to reduce redundancy in de novo transcriptomes of some closely related species with the goal of searching for orthologs for phylogenetics afterwards.

I'm interested in using CD-HIT-EST, but I'm unsure about was threshold of similarity is best to use. 95 and 90 aren't that different in terms of how many clusters are formed. Is 90 too stringent a threshold, potentially losing important genes? Since I'm looking for useful orthologs downstream, reducing redundancy as much as possible is important, but I don't want to sacrifice unique genes...

[Brian Bushnell]

I have a neat tool for reducing redundancy in nucleotide space, which is often present in metagenome assemblies, depending on your methodology.

dedupe.sh -Xmx31g in=assembly.fa out=unique.fa

By default, if you run it like that, it will remove all but one copies of any duplicate scaffolds, and remove any scaffolds that are substrings of other scaffolds. But you can additionally specify other parameters, such as a minimum percent overlap (so that a transcript doesn't absorb another fully-contained transcript, for example) and a variable edit distance or percent identity.

For example:
dedupe.sh -Xmx31g in=assembly.fa out=unique.fa maxedits=20 minidentity=98 minoverlappercent=80

...will remove anything that is a substring of another string with identity of at least 98% (up to a maximum of 20 edits, which sets the bandwidth of the banded alignment), as long as the shorter one is at least 80% of the length of the longer one.

It's incredibly fast with no edits allowed, and still very fast with edits allowed. The -Xmx flag, by the way, should be set to around 85% of your system's physical memory. The whole assembly is stored in memory, at about 1 byte per base plus a few hundred bytes per scaffold.

Available here:

http://sourceforge.net/projects/bbmap/

[dongilbert]

Consider using EvidentialGene at http://arthropods.eugenes.org/EvidentialGene/

The EvidentialGene_trassembly pipeline software does a good job of picking best non-redundant mRNA loci and alternates from a large collection of partly redundant mRNA assemblies. It uses coding sequence metrics, and so avoids the problem of selecting errors by picking longest transcripts. It works well with large collections (10 millions) of transcripts produced by several assemblers on the same data, and gives you best results that way (each assembler gets only some complete genes).

Using CD-HIT-EST as you consider is not the right way, as that will select for errors (gene joins and misassemblies) and lose some of your valuable best ortholog genes.

[Apexy]

Hi,
To add to what dongilbert suggested, I would say that care has to be taken when performing post-assembly clustering or re-assembly (across multiple kmers or assemblies from different assemblies). Haznedaroglu et al. BMC Bioinformatics 2012, 13:170 suggested an optimization procedure when employing such post-assembly processing and as you have noted there is the risk of losing potentially unique transfrags. Thus there is generally no concensus as to what identity threshold to implement as this will also vary with transfrag diversity in your assembly.