Very large distance matrix using MOTHUR

[dsher]

Hello,

We are trying to process a large dataset (~200 samples, each with ~100,000 Illumina 250bp 16S reads) using MOTHUR. We now have reached the stage of making a distance matrix to identify OTUs but this distance matrix is very large (~1.7Tb even after using the cluster.split approach). Our computational power is probably enough to run this analysis on the current dataset but we are expecting more samples, and worry that at some stage the OTU table will simply be too large.

1) Is this normal (the size)?
2) Has anyone got any experience with this sort of problem? Is there a way to divide the OTU table into sub-analyses (apart from using cluster.split) and then put them together again?

Thanks
Ashraf and Daniel

[pschloss]

If I had to guess, you are trying to sequence something longer than 250 nt long or you are using the 300PE V3 chemistry. If this is the case, then I'd suggest checking this out:

http://blog.mothur.org/2014/09/11/Why-such-a-large-distance-matrix%3F/

Also, since we don't commonly screen the SeqAnswers forum for questions like these, it would probably be best to address questions over at www.mothur.org/forum.

Thanks,
Pat

[dsher]

Thanks Pat for the reply - I have a couple of questions but will post them on the blog you mentioned rather than here. In any case if anyone else is reading this I highly recommend having a look at that blog.

Cheers
Daniel