Hiseq clustering

[cookiemic]

Hi all,

I have a huge dataset from Hiseq (15 million sequences after quality filtering).
I would like to use cd-hit or other clustering software to reduce the redundancy first, and proceed to BLAST against to nr database.

However, for the first step using cd-hit-est (v.4.6.1) with 10 threads, it already took me a week but still not yet finished.

I am new to NGS analysis, anyone could advise to speed up the process?
Your kindly suggestions and help are highly appreciate. Many thanks.

[mastal]

Hiseq clustering

15 million reads from a HiSeq these days is not a large dataset.

You don't say what kind of biological experiment you are analysing that would explain why you want to cluster the data first.

In general, for NGS data you want to do either alignment (with a program like Bowtie or BWA, or there are many others) or de novo assembly, if you don't have a reference genome (with programs like Velvet, Soapdenovo, or other assembler).

Have a look at the list of available software in the SeqWiki.

[cookiemic]

My data is composed of sequences derived from microbes (virus or bacteria) isolated from patients.
And I would like to blastx each sequence to nr database.
But I know blast is too computational intensive, so I would like to reduce the redundancy by clustering first.
I have experience in using bowtie to map the dataset on a particular genome.
But I don't know if bowtie sensitive/accurate to map to nr database?

[Kennels]

The fastx toolkit has a tool to 'collapse' identical reads in your dataset, and output a header containing how many times the read was detected (e.g >1-xxx , >2-xxx, etc)

http://hannonlab.cshl.edu/fastx_toolkit/commandline.html#fastx_collapser_usage

I'm sure there are other tools that would do this faster, but you'll have to search.
Also, it won't make much of a difference if you don't have much duplication in your dataset (it is very good for small RNA datasets).
I wouldn't cluster reads 'too much' as this will mask information when you align to a database. However I'm not sure what the end goal of your experiment is, but there might be very high similarity between sequences from different sources in your sample.

You could pull out only virus and bacterial sequences from the nr database, create a bowtie index from it and bowtie against that using perfect matches first. Bowtie2 will give you more stats. It should be relatively fast to align (shouldn't be a week), and you can interrogate the bowtie output rather than sam output which might be easier as a first pass. In fact, I would try this alignment first before doing any 'collapsing' of reads or clustering.