Large RAM Requirements and Clusters/Clouds

[pbatzel]

Hello people of the forum, I'm new here and am hoping you all can help me with a hypothetical.

Let's say I have access to a cluster, or even a cloud. I want to run genomic/transcriptomic assembly programs on this cluster/cloud. The problem is that I expect these programs (Velvet/Oases, Trinity, whatever else) to require large amounts of RAM, 100GB+, and the largest node on my cluster/cloud is only 32GB. Now besides switching to an MPI/Hadoop based assembly program (ABySS or whatnot) or writing my own, what are my viable options? Has anyone tried using a distributed operating system (MOSIX, Kerrighed, ...) with shared memory on a cluster/cloud? What about a virtual SMP? What else?

Thanks for any help!

[westerman]

pbatzel:
I do not have an answer. Your problem and proposed solutions are complex thus you may not get an answer. But, since you are a new person to seqanswers, I do want to say hi and encourage you to not get disappointed with seqanswers if no one can help you out on this.

[pbatzel]

@westerman:
Thanks for the welcome! Sadly I know that this is a fairly complex problem (but maybe fairly common too?) and that the solutions I presented may be complex. What I'm really hoping for is someone to raise their hand and say "Hey, I've done that before!" and then I would know I'm not walking down a dead-end corridor. I'd love if someone had a simpler solution that didn't throw efficiency out the window, but I'm not expecting one.

For anyone that's interested, after I posted this question I was referred to a pretty decent blog post that has some relevance to this problem, but doesn't offer a real solution:

http://jermdemo.blogspot.com/2011/06/big-ass-servers-and-myths-of-clusters.html