[dp05yk]

does the samse/sampe support multi threading. There does not seem to be an option listed?

Unfortunately not. This is why running pBWA is all about finding the right balance between multithreading and parallelism. If you ran (as I suggested) 24 processes across 3 cores each with 3 threads, you'll need to just run sampe/samse with 24 processes, no threads. That should work just fine.

[dp05yk]

I don't see the point of using mpi/parallelization for a process that is embarrassingly parallel. BWA has a great multi-threaded functionality and with samtools merging functionalitie is easy. So it is very easy to chunk the reads and run them multi-threaded then merge the bam files.

Actually, BWA only has multi-threaded functionality for half of the process. sampe/samse is not multithreaded. Moreover, BWA's multithreading was inefficient when I initially released pBWA for anything more than ~8 threads (google it... there are multiple threads denoting this issue prior to the update). Upon release of pBWA, running pBWA with 24 processors was faster _just for aln_ than BWA was with 24 threads. Obviously pBWA was faster for sampe/samse. FYI - it was my edits that improved multithreading efficiency in BWA.

You're right - this isn't an enormous breakthrough but it has its advantages. On the cluster I use it's much easier to get a large MPI job scheduled than hundreds of serial jobs... further increasing the usefulness of pBWA.

[ichorny]

It could possibly be a RAM issue... with MPI applications each instance of the program is completely separate from another. Ie. where threaded applications share global variables, MPI applications do not. So if pBWA requires x GB/RAM for 1 processor, it will require p*x GB/RAM for p processors... if you only have 50GB RAM/node and you're running 24 processes on said node, you're only allowing ~2.1GB RAM per process... that's cutting it mighty-fine.

What you may want to try is combining multithreading and pBWA... use 24 processors (again), but tell the system to put 8 on each of your 3 nodes... then in your /pBWA aln command, use -n 3 to spawn 3 threads per node so you'll use all 72 of your cores... tell me how that works.

does the samse/sampe support multi threading. There does not seem to be an option listed?

[rskr]

I don't see the point of using mpi/parallelization for a process that is embarrassingly parallel. BWA has a great multi-threaded functionality and with samtools merging functionalitie is easy. So it is very easy to chunk the reads and run them multi-threaded then merge the bam files.

[dp05yk]

3 nodes, 24 hyperthreaded processors per node (12 actual). About 50G of ram per node.

I am just submitting via a qsub to SGE and letting SGE decide on the distribution. With 32 preocessors I think I got 24 on one and 8 on the other. With 24 I got 23 on one and 1 on the other.

Is it a RAM issue?

It could possibly be a RAM issue... with MPI applications each instance of the program is completely separate from another. Ie. where threaded applications share global variables, MPI applications do not. So if pBWA requires x GB/RAM for 1 processor, it will require p*x GB/RAM for p processors... if you only have 50GB RAM/node and you're running 24 processes on said node, you're only allowing ~2.1GB RAM per process... that's cutting it mighty-fine.

What you may want to try is combining multithreading and pBWA... use 24 processors (again), but tell the system to put 8 on each of your 3 nodes... then in your /pBWA aln command, use -n 3 to spawn 3 threads per node so you'll use all 72 of your cores... tell me how that works.

[ichorny]

Hm... could you answer a couple of questions for me?

1. What is your system's nodal information (# nodes, # cores per node, RAM per node).

2. How are you splitting the jobs up (ie. how many processes are you trying to put on each node)?

3 nodes, 24 hyperthreaded processors per node (12 actual). About 50G of ram per node.

I am just submitting via a qsub to SGE and letting SGE decide on the distribution. With 32 preocessors I think I got 24 on one and 8 on the other. With 24 I got 23 on one and 1 on the other.

Is it a RAM issue?

[dp05yk]

Hm... could you answer a couple of questions for me?

1. What is your system's nodal information (# nodes, # cores per node, RAM per node).

2. How are you splitting the jobs up (ie. how many processes are you trying to put on each node)?

[ichorny]

Seq Fault when running on 12 or more processors

I am getting a seg fault when running on 12,24, or 32 processors. This is output from a 32 processor job. I am using fastqs that are about 250GB each.
Thoughts? It works for 4,8, and 6 processors, so far (job yet to finish). I did not test 10.

[compute-2-1:17105] *** Process received signal ***
[compute-2-1:17105] Signal: Segmentation fault (11)
[compute-2-1:17105] Signal code: Address not mapped (1)
[compute-2-1:17105] Failing at address: (nil)
[compute-2-1:17105] [ 0] /lib64/libpthread.so.0 [0x331e40eb10]
[compute-2-1:17105] [ 1] /lib64/libc.so.6(memcpy+0x15b) [0x331d87c24b]
[compute-2-1:17105] [ 2] /home/galaxy/production/Sept06/galaxy-central/tool-deps/mpirun/1.4.3/lib/libmpi.so.0(ompi_convertor_unpack+0xae) [0x2b6a6db846ae]
[compute-2-1:17105] [ 3] /home/galaxy/production/Sept06/galaxy-central/tool-deps/mpirun/1.4.3/lib/libmpi.so.0 [0x2b6a6dc1fc6e]
[compute-2-1:17105] [ 4] /home/galaxy/production/Sept06/galaxy-central/tool-deps/mpirun/1.4.3/lib/libmpi.so.0 [0x2b6a6dc1cc56]
[compute-2-1:17105] [ 5] /home/galaxy/production/Sept06/galaxy-central/tool-deps/mpirun/1.4.3/lib/libmpi.so.0 [0x2b6a6dbb6e38]
[compute-2-1:17105] [ 6] /home/galaxy/production/Sept06/galaxy-central/tool-deps/mpirun/1.4.3/lib/libopen-pal.so.0(opal_progress+0x5a) [0x2b6a6e1a04ea]
[compute-2-1:17105] [ 7] /home/galaxy/production/Sept06/galaxy-central/tool-deps/mpirun/1.4.3/lib/libmpi.so.0 [0x2b6a6db77135]
[compute-2-1:17105] [ 8] /home/galaxy/production/Sept06/galaxy-central/tool-deps/mpirun/1.4.3/lib/libmpi.so.0 [0x2b6a6dbc5086]
[compute-2-1:17105] [ 9] /home/galaxy/production/Sept06/galaxy-central/tool-deps/mpirun/1.4.3/lib/libmpi.so.0 [0x2b6a6dbc5737]
[compute-2-1:17105] [10] /home/galaxy/production/Sept06/galaxy-central/tool-deps/mpirun/1.4.3/lib/libmpi.so.0 [0x2b6a6dbbb3d0]
[compute-2-1:17105] [11] /home/galaxy/production/Sept06/galaxy-central/tool-deps/mpirun/1.4.3/lib/libmpi.so.0 [0x2b6a6dbcd3c9]
[compute-2-1:17105] [12] /home/galaxy/production/Sept06/galaxy-central/tool-deps/mpirun/1.4.3/lib/libmpi.so.0(MPI_Bcast+0x171) [0x2b6a6db8be11]
[compute-2-1:17105] [13] pBWA(bwt_restore_bwt+0x7c) [0x407fbc]
[compute-2-1:17105] [14] pBWA(bwa_aln_core+0x81) [0x408b01]
[compute-2-1:17105] [15] pBWA(bwa_aln+0x196) [0x409056]
[compute-2-1:17105] [16] pBWA(main+0xec) [0x4281ac]
[compute-2-1:17105] [17] /lib64/libc.so.6(__libc_start_main+0xf4) [0x331d81d994]
[compute-2-1:17105] [18] pBWA [0x404b79]
[compute-2-1:17105] *** End of error message ***
--------------------------------------------------------------------------

[dp05yk]

That's interesting... as the website for pBWA notes, gzipped FASTQ files are not supported since we required random file access to split up the input files.

[ichorny]

pBWA and fastq.gz

I notice that when I run mpirun and gzipped fastq files it returns a sam file containing only the header. If I run without mpirun it works just fine.

BTW I am using v2.

Thanks,

Ilya

[dp05yk]

Hi Sheng,

You need to figure out the alias to use to call the MPI compiler. On most clusters this will be "mpicc"... you'll have to contact your system administrator to figure out what this is, or perform a google search for more popular aliases.

Then, in both makefiles (one in the root folder and one in the bwt_gen folder), change
CC = mpicc
to
CC = youralias

Where youralias = the alias used to call your MPI compiler.

[sheng]

pBWA installation

Hi dp05yk,

Thanks a lot for your reply! I am working on a cluster which have multiple node and core. I am sure we have Openmpi installed in the cluster. So what is the information about openmpi that I need to change the makefile and which part of makefile do I need to change? When I compile it, I just type make? Any other steps?

Cheers,
Sheng

Hi sheng,

These requirements can be broken down as follows. pBWA is a _parallel_ implementation of BWA. This means that unless your computer system has multiple processors, this software will be of no use to you. Essentially what pBWA does is distribute massive input reads files over multiple processors in order to execute BWA in parallel. If you do not have access to a computer cluster or parallel machine, this is impossible for you since you do not have multiple processors to distribute over If you have a standard home computer with a multi-_core_ processor, just use the multithreading option available in the latest release of BWA.

As for the MPICC compiler - if you in fact do have access to a computing cluster, you'll need to ask one of the administrators if the MPI compiler is installed (MPICH or OpenMPI work, actually). If it is installed, it could have an alias over than "mpicc", at which point you'll have to modify the makefile accordingly.

I hope this clears some issues up for you! I have a suspicion you may have been trying to install this on your home or basic lab PC, in which case you will be better off using BWA.

Thanks for posting!

[dp05yk]

Hi sheng,

These requirements can be broken down as follows. pBWA is a _parallel_ implementation of BWA. This means that unless your computer system has multiple processors, this software will be of no use to you. Essentially what pBWA does is distribute massive input reads files over multiple processors in order to execute BWA in parallel. If you do not have access to a computer cluster or parallel machine, this is impossible for you since you do not have multiple processors to distribute over If you have a standard home computer with a multi-_core_ processor, just use the multithreading option available in the latest release of BWA.

As for the MPICC compiler - if you in fact do have access to a computing cluster, you'll need to ask one of the administrators if the MPI compiler is installed (MPICH or OpenMPI work, actually). If it is installed, it could have an alias over than "mpicc", at which point you'll have to modify the makefile accordingly.

I hope this clears some issues up for you! I have a suspicion you may have been trying to install this on your home or basic lab PC, in which case you will be better off using BWA.

Thanks for posting!

[sheng]

Hi dp05yk,

Thanks for releasing the pBWA! The discussion is very helpful for the usage of pBWA. However, I found problems installing pBWA and I could not find any README file in the source code directory. Would you please help me with the following error message I got when trying to compile it? I read the home page of PBWA and know about the requirement for MPI-"pBWA requires a multi-node (or multi-core) *nix system with a parallel scheduler alongside the OpenMPI C library in order to compile and run. " But I am not sure how to add the multi-node (or multi-core) *nix system with a parallel scheduler alongside the OpenMPI C library to compile it.

Thanks a lot!


make
#################Error################
make[1]: Entering directory `/panda_scratch_homes001/shl2018/software/alignment/pBWA'
make[1]: Nothing to be done for `lib'.
make[1]: Leaving directory `/panda_scratch_homes001/shl2018/software/alignment/pBWA'
make[1]: Entering directory `/panda_scratch_homes001/shl2018/software/alignment/pBWA/bwt_gen'
mpicc -c -g -Wall -m64 -O2 -DHAVE_PTHREAD -D_LARGEFILE64_SOURCE bwt_gen.c -o bwt_gen.o
make[1]: mpicc: Command not found
make[1]: *** [bwt_gen.o] Error 127
make[1]: Leaving directory `/panda_scratch_homes001/shl2018/software/alignment/pBWA/bwt_gen'
make: *** [lib-recur] Error 1
############### Error ###################

Just to let everyone know, an alternate version of pBWA is now available that cleans up the workflow a bit. The user is no longer required to enter the number of reads in the FASTQ file, and SAM information is output to one file in parallel by all processors. There are also a few minor stability enhancements that should make pBWA compatible with MPICH. Performance appears to be similar to pBWA-r32. Thanks go to Rob Egan for the enhancements.

It's available at http://sourceforge.net/projects/pbwa ... thanks!

[dp05yk]

Just to let everyone know, an alternate version of pBWA is now available that cleans up the workflow a bit. The user is no longer required to enter the number of reads in the FASTQ file, and SAM information is output to one file in parallel by all processors. There are also a few minor stability enhancements that should make pBWA compatible with MPICH. Performance appears to be similar to pBWA-r32. Thanks go to Rob Egan for the enhancements.

It's available at http://sourceforge.net/projects/pbwa ... thanks!