I got a question from our cluster IT that I could not answer. I am having problems runing BWA, samtools or blastx in the multithreaded mode on the cluster. Cluster resorces are normally reserved by_core, that is multithreaded jobs are spread across multiple nodes. BWA, samtools and blast+ run fine on my 2 Quad Core CPU with hyperthreading, that is with 16 threads. The question was - can actually BWA, samtools and blastx+ run with multiple threads when spread across several nodes? If not, this answers the question. If yes, are there any specifics/pecularities in scheduling resources?
Seqanswers Leaderboard Ad
Collapse
Announcement
Collapse
No announcement yet.
X
-
Depending on what scheduling software your cluster uses you can specify multiple threads along with an equivalent number of cores (-n in LSF) to go with them.
That said you would want to be judicious when spreading jobs across cores/nodes. Depending on what else may be running on a particular server/node you may overload that server/node (which your admins may not like). There may also be some I/O problems if you do not have a high performance storage system that is available on the cluster and your jobs get spread across several nodes.
In case of LSF you can use flags (-x) that can exclusively reserve a node for your use with multiple cores which would ensure you do not step on someone else' jobs. There are probably equivalent examples for SGE.
-
Much better than running a single multithreaded instance of blast over many nodes is to split your input (e.g. with fastasplitn) and then run multiple instances of blast so that the maximum number of threads of each instance equals the number of cores of one CPU. With SGE this is achieved by using the smp parallel environment. You can make a simple bash script, e.g.:
Code:cat blastp.sh #!/bin/bash #$ -N blastp #$ -j y #$ -cwd #$ -pe smp 8 #$ -R y blastp -query input.${SGE_TASK_ID} -db nr -lotsOfFlags -outfmt 6 -num_threads 8 -out ${SGE_TASK_ID}.tsv
qsub -t 1-10:1 blastp.sh
To start/queue 10 instances of blastp (each with 8 threads). Input for this would be input.1, input.2, .., input.10 and you could just concatenate the results with cat..Last edited by rhinoceros; 04-29-2013, 12:36 PM.savetherhino.org
Comment
-
Originally posted by rhinoceros View PostMuch better than running a single multithreaded instance of blast over many nodes is to split your input (e.g. with fastasplitn) and then run multiple instances of blast so that the maximum number of threads of each instance equals the number of cores of one CPU. With SGE this is achieved by using the smp parallel environment. You can make a simple bash script, e.g.:
Code:cat blastp.sh #!/bin/bash #$ -N blastp #$ -j y #$ -cwd #$ -pe smp 8 #$ -R y blastp -query input.${SGE_TASK_ID} -db nr -lotsOfFlags -outfmt 6 -num_threads 8 -out ${SGE_TASK_ID}.tsv
qsub -t 1-10:1 blastp.sh
To start/queue 10 instances of blastp (each with 8 threads). Input for this would be input.1, input.2, .., input.10 and you could just concatenate the results with cat..
Comment
-
Originally posted by Khen View PostDefinitely need to find out what your scheduling software is. But in general, I prefer parallelization to multithreading. It is much easier and I find that I run into fewer problems.
Comment
-
Originally posted by yaximik View PostWell, that is a separate interesting question. GNU parallel is one way to go, but I was not yet quite able figuring out how to run it on a cluster. I am not a programmer, I mean I do not have sufficent exerience yet, so how would you request resources for GNU parallel? Unless you parallelize in some other way...
Do I sense some resistance on your side to give up on your dedicated workstation (where you are in control)? In the long run using a dedicated cluster would allow you to get much more done.Last edited by GenoMax; 04-30-2013, 03:35 AM.
Comment
-
Originally posted by GenoMax View PostCan we stick to the original question you had posted? What scheduling software is the cluster you have access to running? On a cluster you are going to get the most benefit by using the job scheduling system.
Do I sense some resistance on your side to give up on your dedicated workstation (where you are in control)? In the long run using a dedicated cluster would allow you to get much more done.
Comment
-
Originally posted by lh3 View PostGNU parallel is not designed for clusters. LSF and SGE/UGE are.
Comment
-
Parallel simply launches multiple jobs on one machine. In principle, you can launch parallel inside qsub/bsub. On LSF, it should be something like: bsub -n 8 'parallel -j 8 ...'. However, with LSF/SGE, this is a bad idea. You should just let LSF/SGE manage all your jobs. Parallel has little use except for constructing command lines. As GenoMax has suggested, just use SGE. You won't crash nodes by submitting multithreaded jobs in a small batch.
Common use of SGE (such as specifying #cpus) works everywhere. If your system admins set resource limits on queues, ask them. We can offer little help.
For most NGS analyses, investing on GPU is a waste of money before you really understand how that works and/or what programs are available.
Comment
-
So we now know that you are going to use SGE.
Can you take a moment and clarify if you are using a true "grid" or a compute "cluster". Here is a posting that would help with the definitions.
If you are truly using a "grid", meaning compute resources that are connected over a non-local network, then trying to split/multi-thread jobs would be a bad idea.
Comment
-
Originally posted by GenoMax View PostSo we now know that you are going to use SGE.
Can you take a moment and clarify if you are using a true "grid" or a compute "cluster". Here is a posting that would help with the definitions.
If you are truly using a "grid", meaning compute resources that are connected over a non-local network, then trying to split/multi-thread jobs would be a bad idea.
It looks like I already have learned the answer in a hard way. My last 64 and 96 thread blastx jobs died with the following error:
Code:A daemon (pid 19390) died unexpectedly with status 137 while attempting to launch so we are aborting. There may be more information reported by the environment (see above). This may be because the daemon was unable to find all the needed shared libraries on the remote node. You may set your LD_LIBRARY_PATH to have the location of the shared libraries on the remote nodes and this will automatically be forwarded to the remote nodes. mpirun noticed that the job aborted, but has no info as to the process that caused that situation.
Code:#$ -S /bin/bash #$ -cwd #$ -N SC3blastx_64-96thr #$ -pe openmpi* 64-96 #$ -l h_rt=24:00:00,vf=3G #$ -j y #$ -M [email protected] #$ -m eas # # Load the appropriate module files # Should be loaded already #$ -V mpirun -np $NSLOTS blastx -query myquery.fasta -db nr -out query.out -evalue 0.001 -max_intron_length 100000 -outfmt 5 -num_alignments 20 -lcase_masking -num_threads $NSLOTS
I was advised earlier by someone from openmpi user group to use –pe openmpi [ARG} , where ARG = number_of_processes x number_of_threads , and then pass desired number of threads as ‘ mpirun –np $NSLOTS cpus-per-proc [ number_of_threads]’. When I did that, I got an error that more threads were requested than number of physical cores.
Oh,well... Looks like splitting large datafiles to thousands of smaller pieces is the only option to use blastx. Any other suggestions?
Comment
-
Here are two useful pages:
Imperial College London is a world-class university with a mission to benefit society through excellence in science, engineering, medicine and business.
Comment
Latest Articles
Collapse
-
by seqadmin
The field of epigenetics has traditionally concentrated more on DNA and how changes like methylation and phosphorylation of histones impact gene expression and regulation. However, our increased understanding of RNA modifications and their importance in cellular processes has led to a rise in epitranscriptomics research. “Epitranscriptomics brings together the concepts of epigenetics and gene expression,” explained Adrien Leger, PhD, Principal Research Scientist...-
Channel: Articles
04-22-2024, 07:01 AM -
ad_right_rmr
Collapse
News
Collapse
Topics | Statistics | Last Post | ||
---|---|---|---|---|
Started by seqadmin, 05-02-2024, 08:06 AM
|
0 responses
17 views
0 likes
|
Last Post
by seqadmin
05-02-2024, 08:06 AM
|
||
Started by seqadmin, 04-30-2024, 12:17 PM
|
0 responses
20 views
0 likes
|
Last Post
by seqadmin
04-30-2024, 12:17 PM
|
||
Started by seqadmin, 04-29-2024, 10:49 AM
|
0 responses
27 views
0 likes
|
Last Post
by seqadmin
04-29-2024, 10:49 AM
|
||
Started by seqadmin, 04-25-2024, 11:49 AM
|
0 responses
28 views
0 likes
|
Last Post
by seqadmin
04-25-2024, 11:49 AM
|
Comment