bowtie with humongous reference database

[rtyagi]

Hi all,

I have been trying to find a way to build bowtie2 index for a really large reference database (>100GB) but I keep running into memory limits (obviously). Is there a way to do this (in chunks, perhaps... or some other way that trades processing time for low memory requirement)?

[dpryan]

There's currently no way to build a bowtie2 index for something of that size without gobs of memory. Perhaps a different aligner would work better.

Is this a huge plant genome?

[rtyagi]

No. not a single genome. just a very large reference set including many difference genomes. The reads come from a microbiome project.

[dpryan]

You'll be better off mapping to each of the genomes and then post-processing the results (or just using a different tool...though I expect many of them will have issues with something that size).

[GenoMax]

With that large a "genome" you are going to run into memory issues with most aligners. That said what is the maximum memory you have tried so far. It should work with a machine with TB of RAM (which is no longer not that uncommon in shared compute facilities).