[gmarco]

I guess the only one who knows if an updated version will be released is dp05yk.

[abi]

Does this work for BWA-MEM ?

Yes, an updated version of this software will be highly desired since the BWA itself has changed to incorporate MEM option and better indexing algorithm.

[narain]

Is there an updated version of pBWA ?

[gmarco]

Dear all,
Currently we try to run pBWA on our cluster as well.

Unfortunately, we have problems with proper index generation. I suppose pBWA index differs from bwa one - there is an additional file test_ref.fa.rbwt which we can not produce using regular bwa index.

Proc 0: [bwa_seq_open] seeked to 0 in reads_1.fq
Proc 0: [bwa_seq_open] seeked to 0 in reads_2.fq
Broadcasting BWT (this may take a while)... done!
Broadcasting BWT (this may take a while)... [bwt_restore_bwt] fail to open file 'test_ref.fa.rbwt'. Abort!

How to prepare a valid pBWA index out of fasta file?

Are you generating the index with the latest BWA version 0.6.2?

I think i got the same problem. If pBWA version is 0.5.9 you should be using an index generated by BWA 0.5.9 also. It won't work otherwise.

Regards,
G.

[stoker]

Dear all,
Currently we try to run pBWA on our cluster as well.

Unfortunately, we have problems with proper index generation. I suppose pBWA index differs from bwa one - there is an additional file test_ref.fa.rbwt which we can not produce using regular bwa index.

Proc 0: [bwa_seq_open] seeked to 0 in reads_1.fq
Proc 0: [bwa_seq_open] seeked to 0 in reads_2.fq
Broadcasting BWT (this may take a while)... done!
Broadcasting BWT (this may take a while)... [bwt_restore_bwt] fail to open file 'test_ref.fa.rbwt'. Abort!

How to prepare a valid pBWA index out of fasta file?

[gmarco]

I'm stuck at sampe step. pBWA internal documentation shows that pBWA sampe usage is the following one:

Usage:   pBWA sampe -f <output.sam> [options] <prefix> <SAI_FILE_PREFIX1> <SAI_FILE_PREFIX2> <in1.fq> <in2.fq>

Options: -a INT   maximum insert size [500]
         -o INT   maximum occurrences for one end [100000]
         -n INT   maximum hits to output for paired reads [3]
         -N INT   maximum hits to output for discordant pairs [10]
         -c FLOAT prior of chimeric rate (lower bound) [1.0e-05]
         -f FILE  sam file name/prefix to output results to
         -M       merge all sam file prefixes into one file
         -r STR   read group header line such as `@RG\tID:foo\tSM:bar' [null]
         -P       preload index into memory (for base-space reads only)
         -s       disable Smith-Waterman for the unmapped mate
         -A       disable insert size estimate (force -s)

Notes: 1. For SOLiD reads, <in1.fq> corresponds R3 reads and <in2.fq> to F3.
       2. For reads shorter than 30bp, applying a smaller -o is recommended to
          to get a sensible speed at the cost of pairing accuracy.
       3. For the SAI prefixes, do NOT include the _1 and _2 generated by aln
          as sampe will auto-detect these.

While pBWA website show the following command as an example:

./pBWA sampe -f SamPrefix /path/to/Index.fa SaiPrefix SaiPrefix[2] /path/to/Read_1.fq /path/to/Read_2.fq

I don't understand why in website example reference fa file is an arg and in shell man it isn't. And what's the meaning of the first <prefix> arg.

I'm trying with the following command and i'm getting an error. I've already tried to re-build the index with bwa index and error persists:

pBWA sampe -f output.sam /share/references/genomes/human/hg19/bwa_ref/hg19.fa aln_left aln_right /home/gmarco/input/data/rawdata/HapMap_1.fastq /home/gmarco/input/data/rawdata/HapMap_2.fastq

Proc 0: Found second SAI file - aln_right-1-00000.sai
Proc 0: [bwa_seq_open] seeked to 0 in /home/gmarco/input/data/rawdata/HapMap_1.fastq
Proc 0: [bwa_seq_open] seeked to 0 in /home/gmarco/input/data/rawdata/HapMap_2.fastq
Proc 0: [bwa_sai2sam_pe_core] 262144 reads
Proc 0: [bwa_sai2sam_pe_core] convert to sequence coordinate...
Broadcasting BWT (this may take a while)... done!
[bwt_restore_sa] SA-BWT inconsistency: seq_len is not the same. Abort!
[sg13:30874] *** Process received signal ***
[sg13:30874] Signal: Aborted (6)
[sg13:30874] Signal code:  (-6)
[sg13:30874] [ 0] /lib64/libpthread.so.0 [0x347b60eb10]
[sg13:30874] [ 1] /lib64/libc.so.6(gsignal+0x35) [0x3cf8c30265]
[sg13:30874] [ 2] /lib64/libc.so.6(abort+0x110) [0x3cf8c31d10]
[sg13:30874] [ 3] pBWA [0x404c52]
[sg13:30874] [ 4] pBWA(bwt_restore_sa+0xce) [0x4081de]
[sg13:30874] [ 5] pBWA(bwa_cal_pac_pos_pe+0x1b35) [0x41ad05]
[sg13:30874] [ 6] pBWA(bwa_sai2sam_pe_core+0x3e3) [0x41b203]
[sg13:30874] [ 7] pBWA(bwa_sai2sam_pe+0x450) [0x41bef0]
[sg13:30874] [ 8] pBWA(main+0x96) [0x428206]
[sg13:30874] [ 9] /lib64/libc.so.6(__libc_start_main+0xf4) [0x3cf8c1d994]
[sg13:30874] [10] pBWA [0x404b79]
[sg13:30874] *** End of error message ***
Aborted

Thanks.

[gmarco]

Hello,

I'm trying the following code for MPI use on my cluster. Which has 2 nodes with 8 CPUs each and 32GB ram per node.

It spawns 16 pbwa processes over the 2 compute nodes. Which seems ok, but checking execution log it seems that pbwa is running the same align process 16 times.

Is my job wrong?
Would appreciate some support.

Of course running this on SGE with Open MPI.

#!/bin/bash
### shell
#$ -S /bin/bash
### env path
#$ -V
### name
#$ -N aln_left
### current work directory
#$ -cwd
### merge outputs
#$ -j y
### PE
#$ -pe mpi 16
### select all.q
#$ -q all.q


mpirun pBWA aln -f aln_left /data_in/references/genomes/human/hg19/bwa_ref/hg19.fa /data_in/rawdata/HapMap_1.fastq > /data_out_2/tmp/mpi/HapMap_1.cloud.left.sai

[xied75]

Dear all,

Just passing by and saw this thread. It seems lots of people would rather have BWA running on multi-core. Given INDEX is a one-off task for a certain batch of files, and ALN is supporting -t, thus only left SAMPE, SAMSE single-threaded.

I have a version of SAMPE able to run multithreaded. Although it's done on Windows under CRT (C Runtime), the basic idea can be easily transfered back to the Linux code base, only needed knowledge of MemoryMappedFiles, and some threading concepts, it's just some plumbing wrapped around some core function calls. SAMSE can be done in the same way ( I didn't do that because I only have PE data in my hand).

My project site is here http://bow.codeplex.com/, you can find some performance data on the release page. Source code is on GitHub: https://github.com/xied75, (oh, should be this branch https://github.com/xied75/bwa/tree/mt-sampe)

I recently tested this MT-SAMPE on Windows Azure (Cloud) large instance with 4 cores 8GB memory, it runs without any problem on -t 4.

Best,

dong

[gmarco]

It could possibly be a RAM issue... with MPI applications each instance of the program is completely separate from another. Ie. where threaded applications share global variables, MPI applications do not. So if pBWA requires x GB/RAM for 1 processor, it will require p*x GB/RAM for p processors... if you only have 50GB RAM/node and you're running 24 processes on said node, you're only allowing ~2.1GB RAM per process... that's cutting it mighty-fine.

What you may want to try is combining multithreading and pBWA... use 24 processors (again), but tell the system to put 8 on each of your 3 nodes... then in your /pBWA aln command, use -n 3 to spawn 3 threads per node so you'll use all 72 of your cores... tell me how that works.

I'm on the same situation. Since i'm on a test environment i have 2 nodes with 8 processors each and 64GB RAM per node.

So my ideal would be to spawn 8 threads per node over the 2 nodes so all 16 processors are used.

I'm using SGE too. And i've MPI installed on my test cluster.
I would like to know how did a SGE user achieved to split up this process with MPI config inside a SGE job file or command (qsub).

That would be awesome, since now i can only process bwa aln and samse over 1 node with 8 threads.

Thanks !

[ymc]

Does pBWA work with multi-core machine with lots of RAM? I have a six core machine with 64GB RAM. Can it run in six cores for samse/sampe in my case?

[odoyle81]

Does anyone know what to do with all the sam files I get from pBWA? Can I just concatenate them into one big sam file for downstream processing, or what do I do?
Thanks

[wencanh]

Hi, dp05yk!

I am a newbie. I analysed my data recently and found the sampe is so slow. Then I encountered pBWA and tried to use it to improve the analysis steps.

However, I got some problems.

What is the parameter 'NumReads' mean? And how can I get that number?

And when I ran the following codes, there was something wrong.

[wencanh@node9 pBWA]$ ./pBWA aln -t 10 -f /data/a.sai /data/hg19/human_g1k_v37.fasta.gz /data/lane2.R1.clean.fq.gz 100000
librdmacm: couldn't read ABI version.
librdmacm: assuming: 4
CMA: unable to get RDMA device list
--------------------------------------------------------------------------
[[12279,1],0]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
Host: node9

Another transport will be used instead, although this may result in
lower performance.
--------------------------------------------------------------------------
[bwa_aln] 17bp reads: max_diff = 2
[bwa_aln] 38bp reads: max_diff = 3
[bwa_aln] 64bp reads: max_diff = 4
[bwa_aln] 93bp reads: max_diff = 5
[bwa_aln] 124bp reads: max_diff = 6
[bwa_aln] 157bp reads: max_diff = 7
[bwa_aln] 190bp reads: max_diff = 8
[bwa_aln] 225bp reads: max_diff = 9
Proc 0: [bwa_seq_open] seeked to 0 in /data/lane2.R1.clean.fq.gz
[bwa_seq_open] fail to open file '100000'. Abort!
[node9:29323] *** Process received signal ***
[node9:29323] Signal: Aborted (6)
[node9:29323] Signal code: (-6)
[node9:29323] [ 0] /lib64/libpthread.so.0 [0x33c400eb10]
[node9:29323] [ 1] /lib64/libc.so.6(gsignal+0x35) [0x33c3430265]
[node9:29323] [ 2] /lib64/libc.so.6(abort+0x110) [0x33c3431d10]
[node9:29323] [ 3] ./pBWA [0x404f0d]
[node9:29323] [ 4] ./pBWA(bwa_seq_open+0x62) [0x412792]
[node9:29323] [ 5] ./pBWA(bwa_aln+0x88b) [0x40974b]
[node9:29323] [ 6] ./pBWA(main+0xec) [0x4281ac]
[node9:29323] [ 7] /lib64/libc.so.6(__libc_start_main+0xf4) [0x33c341d994]
[node9:29323] [ 8] ./pBWA [0x404b79]
[node9:29323] *** End of error message ***
Aborted

And when I removed the NumReads, it seemed OK! But actually the file"a.sai" had nothing in it.

[wencanh@node9 pBWA]$ ./pBWA aln -t 10 -f /data/a.sai /data/hg19/human_g1k_v37.fasta.gz /data/lane2.R1.clean.fq.gz
librdmacm: couldn't read ABI version.
librdmacm: assuming: 4
CMA: unable to get RDMA device list
--------------------------------------------------------------------------
[[12253,1],0]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
Host: node9

Another transport will be used instead, although this may result in
lower performance.
--------------------------------------------------------------------------
[bwa_aln] 17bp reads: max_diff = 2
[bwa_aln] 38bp reads: max_diff = 3
[bwa_aln] 64bp reads: max_diff = 4
[bwa_aln] 93bp reads: max_diff = 5
[bwa_aln] 124bp reads: max_diff = 6
[bwa_aln] 157bp reads: max_diff = 7
[bwa_aln] 190bp reads: max_diff = 8
[bwa_aln] 225bp reads: max_diff = 9
Proc 0: [bwa_seq_open] seeked to 0 in /data/lane2.R1.clean.fq.gz
Broadcasting BWT (this may take a while)... done!
Broadcasting BWT (this may take a while)... done!
Proc 0: Total time taken: 3.88 sec

[ichorny]

New Error. The alignment works on smaller number of processors but all on the same node. This job was run across nodes. Accroding to the log it finished. Thoughts?

Error:
The pBWA alignment failed.
The output file is empty. You may simply have no matches, or there may be an error with your input file or settings.

End of Log File:
Proc 3: [mergeFilesIntoOne] Finished merge in 2.75 secs
Proc 11: [mergeFilesIntoOne] Finished merge in 2.67 secs
Proc 0: [mergeFilesIntoOne] Finished merge in 2.82 secs
Proc 1: [mergeFilesIntoOne] Finished merge in 2.68 secs
Proc 7: [mergeFilesIntoOne] Finished merge in 2.72 secs
Proc 4: [mergeFilesIntoOne] Finished merge in 2.74 secs
Proc 8: [mergeFilesIntoOne] Finished merge in 2.83 secs
Proc 9: [mergeFilesIntoOne] Finished merge in 2.67 secs
Proc 2: [mergeFilesIntoOne] Finished merge in 2.68 secs
Proc 5: [mergeFilesIntoOne] Finished merge in 2.72 secs
Proc 10: [mergeFilesIntoOne] Finished merge in 2.67 secs
Proc 6: [mergeFilesIntoOne] Finished merge in 2.77 secs

real 11m28.892s
user 21m12.224s
sys 20m3.094s

[dp05yk]

Given it's the same error message in the same function, I'm going to say RAM again - sampe/samse require more RAM than aln, because sampe/samse require every processor to have the entire suffix array (hence the 'broadcasting SA') as well as the BWT.

Just play around with different parallel/threaded combinations... eventually you will find the optimal combination for your system.

EDIT: just realized you were only using 8 processors... perhaps there are other users utilizing RAM on your cluster?

[ichorny]

Now I am getting a seq fault in the sam pe part. This is using 8/12 cores on a machine with 50GB of memory. Thoughts?

Proc 1: [bwa_seq_open] seeked to 31248 in /home/galaxy/production/Sept06/galaxy-central/database/files/003/dataset_3168.dat
Proc 4: [bwa_seq_open] seeked to 124248 in /home/galaxy/production/Sept06/galaxy-central/database/files/003/dataset_3168.dat
Proc 3: [bwa_seq_open] seeked to 93248 in /home/galaxy/production/Sept06/galaxy-central/database/files/003/dataset_3168.dat
Proc 6: [bwa_seq_open] seeked to 186248 in /home/galaxy/production/Sept06/galaxy-central/database/files/003/dataset_3168.dat
Proc 3: [bwa_seq_open] seeked to 93248 in /home/galaxy/production/Sept06/galaxy-central/database/files/003/dataset_3169.dat
Proc 7: [bwa_seq_open] seeked to 217248 in /home/galaxy/production/Sept06/galaxy-central/database/files/003/dataset_3168.dat
Proc 7: [bwa_seq_open] seeked to 217248 in /home/galaxy/production/Sept06/galaxy-central/database/files/003/dataset_3169.dat
Proc 7: [skipToNextPairedRecord] found SDUS-BRUNO-106:1:0:4:21:5226:2182 SDUS-BRUNO-106:1:0:4:21:5226:2182
Proc 6: [bwa_seq_open] seeked to 186248 in /home/galaxy/production/Sept06/galaxy-central/database/files/003/dataset_3169.dat
Proc 6: [skipToNextPairedRecord] found SDUS-BRUNO-106:1:0:4:21:4608:2141 SDUS-BRUNO-106:1:0:4:21:4608:2141
Proc 7: [bwa_sai2sam_pe_core] 124 reads
Proc 7: [bwa_sai2sam_pe_core] convert to sequence coordinate...
Proc 6: [bwa_sai2sam_pe_core] 125 reads
Proc 6: [bwa_sai2sam_pe_core] convert to sequence coordinate...
Proc 0: [bwa_seq_open] seeked to 0 in /home/galaxy/production/Sept06/galaxy-central/database/files/003/dataset_3168.dat
Proc 0: [bwa_seq_open] seeked to 0 in /home/galaxy/production/Sept06/galaxy-central/database/files/003/dataset_3169.dat
Proc 2: [bwa_seq_open] seeked to 62248 in /home/galaxy/production/Sept06/galaxy-central/database/files/003/dataset_3168.dat
Proc 1: [bwa_seq_open] seeked to 31248 in /home/galaxy/production/Sept06/galaxy-central/database/files/003/dataset_3169.dat
Proc 2: [bwa_seq_open] seeked to 62248 in /home/galaxy/production/Sept06/galaxy-central/database/files/003/dataset_3169.dat
Proc 1: [skipToNextPairedRecord] found SDUS-BRUNO-106:1:0:4:21:1817:2166 SDUS-BRUNO-106:1:0:4:21:1817:2166
Proc 2: [skipToNextPairedRecord] found SDUS-BRUNO-106:1:0:4:21:2392:2241 SDUS-BRUNO-106:1:0:4:21:2392:2241
Proc 5: [bwa_seq_open] seeked to 155248 in /home/galaxy/production/Sept06/galaxy-central/database/files/003/dataset_3168.dat
Proc 5: [bwa_seq_open] seeked to 155248 in /home/galaxy/production/Sept06/galaxy-central/database/files/003/dataset_3169.dat
Proc 5: [skipToNextPairedRecord] found SDUS-BRUNO-106:1:0:4:21:4089:2188 SDUS-BRUNO-106:1:0:4:21:4089:2188
Proc 4: [bwa_seq_open] seeked to 124248 in /home/galaxy/production/Sept06/galaxy-central/database/files/003/dataset_3169.dat
Proc 3: [skipToNextPairedRecord] found SDUS-BRUNO-106:1:0:4:21:2841:2141 SDUS-BRUNO-106:1:0:4:21:2841:2141
Proc 4: [skipToNextPairedRecord] found SDUS-BRUNO-106:1:0:4:21:3574:2130 SDUS-BRUNO-106:1:0:4:21:3574:2130
Proc 3: [bwa_sai2sam_pe_core] 125 reads
Proc 3: [bwa_sai2sam_pe_core] convert to sequence coordinate...
Proc 4: [bwa_sai2sam_pe_core] 125 reads
Proc 4: [bwa_sai2sam_pe_core] convert to sequence coordinate...
Proc 1: [bwa_sai2sam_pe_core] 125 reads
Proc 1: [bwa_sai2sam_pe_core] convert to sequence coordinate...
Proc 2: [bwa_sai2sam_pe_core] 125 reads
Proc 2: [bwa_sai2sam_pe_core] convert to sequence coordinate...
Proc 5: [bwa_sai2sam_pe_core] 125 reads
Proc 5: [bwa_sai2sam_pe_core] convert to sequence coordinate...
Proc 0: [bwa_sai2sam_pe_core] 126 reads
Proc 0: [bwa_sai2sam_pe_core] convert to sequence coordinate...
Broadcasting BWT (this may take a while)... done!
Broadcasting SA... done!
Broadcasting BWT (this may take a while)... [compute-2-0:15546] *** Process received signal ***
[compute-2-0:15546] Signal: Segmentation fault (11)
[compute-2-0:15546] Signal code: Address not mapped (1)
[compute-2-0:15546] Failing at address: (nil)
[compute-2-0:15546] [ 0] /lib64/libpthread.so.0 [0x3b43e0eb10]
[compute-2-0:15546] [ 1] /lib64/libc.so.6(memcpy+0x15b) [0x3b4327c24b]
[compute-2-0:15546] [ 2] /home/galaxy/production/Sept06/galaxy-central/tool-deps/mpirun/1.4.3/lib/libmpi.so.0(ompi_convertor_unpack+0xae) [0x2b904780c6ae]
[compute-2-0:15546] [ 3] /home/galaxy/production/Sept06/galaxy-central/tool-deps/mpirun/1.4.3/lib/libmpi.so.0 [0x2b90478a7c6e]
[compute-2-0:15546] [ 4] /home/galaxy/production/Sept06/galaxy-central/tool-deps/mpirun/1.4.3/lib/libmpi.so.0 [0x2b90478a4c56]
[compute-2-0:15546] [ 5] /home/galaxy/production/Sept06/galaxy-central/tool-deps/mpirun/1.4.3/lib/libmpi.so.0 [0x2b904783ee38]
[compute-2-0:15546] [ 6] /home/galaxy/production/Sept06/galaxy-central/tool-deps/mpirun/1.4.3/lib/libopen-pal.so.0(opal_progress+0x5a) [0x2b9047e284ea]
[compute-2-0:15546] [ 7] /home/galaxy/production/Sept06/galaxy-central/tool-deps/mpirun/1.4.3/lib/libmpi.so.0 [0x2b90477ff135]
[compute-2-0:15546] [ 8] /home/galaxy/production/Sept06/galaxy-central/tool-deps/mpirun/1.4.3/lib/libmpi.so.0 [0x2b904784d086]
[compute-2-0:15546] [ 9] /home/galaxy/production/Sept06/galaxy-central/tool-deps/mpirun/1.4.3/lib/libmpi.so.0 [0x2b904784d737]
[compute-2-0:15546] [10] /home/galaxy/production/Sept06/galaxy-central/tool-deps/mpirun/1.4.3/lib/libmpi.so.0 [0x2b90478433d0]
[compute-2-0:15546] [11] /home/galaxy/production/Sept06/galaxy-central/tool-deps/mpirun/1.4.3/lib/libmpi.so.0 [0x2b90478553c9]
[compute-2-0:15546] [12] /home/galaxy/production/Sept06/galaxy-central/tool-deps/mpirun/1.4.3/lib/libmpi.so.0(MPI_Bcast+0x171) [0x2b9047813e11]
[compute-2-0:15546] [13] pBWA(bwt_restore_bwt+0x7c) [0x407fbc]
[compute-2-0:15546] [14] pBWA(bwa_cal_pac_pos_pe+0x1b8b) [0x41ad4b]
[compute-2-0:15546] [15] pBWA(bwa_sai2sam_pe_core+0x3af) [0x41b1bf]
[compute-2-0:15546] [16] pBWA(bwa_sai2sam_pe+0x415) [0x41be45]
[compute-2-0:15546] [17] pBWA(main+0x96) [0x428156]
[compute-2-0:15546] [18] /lib64/libc.so.6(__libc_start_main+0xf4) [0x3b4321d994]
[compute-2-0:15546] [19] pBWA [0x404b79]
[compute-2-0:15546] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 3 with PID 15546 on node compute-2-0.local exited on signal 11 (Segmentation fault).