[krobison]

:The question was - can actually BWA, samtools and blastx+ run with multiple threads when spread across several nodes? If not, this answers the question. If yes, are there any specifics/pecularities in scheduling resources?

I believe the answer to your original question is no. Threads from a single binary cannot be splayed across multiple nodes unless the program is using a framework (such as OpenMPI) that enables this. To my knowledge, none of the tools you are describing are enabled in such as way. Ray & ABySS are two examples of OpenMPI enabled tools.

As other posters have noted, what you would need to do for BLAST/BWA/bowtie/samtools etc is split your job into sub-jobs, run those on the cluster using the cluster scheduling software, and then merge the results at the end.

[rhinoceros]

I believe the answer to your original question is no. Threads from a single binary cannot be splayed across multiple nodes unless the program is using a framework (such as OpenMPI) that enables this. To my knowledge, none of the tools you are describing are enabled in such as way. Ray & ABySS are two examples of OpenMPI enabled tools.

As other posters have noted, what you would need to do for BLAST/BWA/bowtie/samtools etc is split your job into sub-jobs, run those on the cluster using the cluster scheduling software, and then merge the results at the end.

SGE's orte parallel environment can split blast threads over multiple nodes. However, using blast like this is a waste of resources as most of the time blast isn't heavy on the CPU(s) anyway and infiniband is a dog in comparison to smp. Thus, far faster is to run multiple multithreaded instances of blast with split input (not thousands of files, but some 10-50). Our newest cluster has 19 nodes, 2 x 8 cores in 17 nodes with 256 GB ram each and 2 nodes with 2 x 16 cores and 756 GB ram each. In this setup, I've been blasting predicted proteins from metagenomes (100s of thousands to more than a million peptides) against nr in very reasonable times with a script similar to what I mentioned earlier. I don't know it it's 'the best' way to do it, but I know it works..

p.s. I'm not sure blast xml output(s) can be concatenated like the tabular output. Additionally, the xml output files grow ridiculously large very fast. Additional thing to consider: run your blasts from a directory that doesn't get backed up on a regular basis (usually something like /jobs/youruid).

[yaximik]

I believe the answer to your original question is no. Threads from a single binary cannot be splayed across multiple nodes unless the program is using a framework (such as OpenMPI) that enables this. To my knowledge, none of the tools you are describing are enabled in such as way. Ray & ABySS are two examples of OpenMPI enabled tools.

I guess I was confused by FAQ entry from open-mpi.org

http://www.open-mpi.org/faq/?category=running#run-non-mpi-programs

While uptime in the example is certainly not specifically openmpi-enabled, it is not multithreaded per se as, say blastx, so event if multiple instances of it can be launched with mpirun, attempts to use multithreading creates a lot of mess.

[rhinoceros]

I guess I was confused by FAQ entry from open-mpi.org

http://www.open-mpi.org/faq/?category=running#run-non-mpi-programs

While uptime in the example is certainly not specifically openmpi-enabled, it is not multithreaded per se as, say blastx, so event if multiple instances of it can be launched with mpirun, attempts to use multithreading creates a lot of mess.

I'm not 100% sure, but I think your script simply starts the same multithreaded job multiple times. Blast is not mpi compatible.

[dpryan]

I'm not 100% sure, but I think your script simply starts the same multithreaded job multiple times. Blast is not mpi compatible.

I'm ~99% sure you're correct. Starting the same non-mpi-aware process on multiple nodes of a grid or cluster will just cause problems. The various instances won't talk to each other, which is what yaximik is hoping will happen.

[yaximik]

I'm ~99% sure you're correct. Starting the same non-mpi-aware process on multiple nodes of a grid or cluster will just cause problems. The various instances won't talk to each other, which is what yaximik is hoping will happen.

Like I said, it was not entirely groundless hope, as from consultations with openmpi user community it did not sound like unreasonable. But, in addition to openmpi, one has to know more about application itself, this is why I asked some questions here. Indeed, with GenoMax links and rinoceros example script I launched multithreaded blastx using smp interface and it seems running fine so far. I used smp* 8-12 option, and called it as 1-100 array job with max parallel 36 instances, each with as many threads as allocated per node (from the range) as our grid has nodes with different architecture. I see instances are launched on different nodes with different number of threads. Now the most important issue is to figure out how much time I need per particular dataset, as long job is not killed before it completes.