Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • BWA multithreading

    Hi all,

    I have BWA installed on a Mac OS 10.6 installation with 8 cores (64 bit machine). I have tried using '-t 8' or '-t 4' to use multiple cores, but it doesn't appear to be working? I have built the program from source, but don't seem to be able to find anything I can change in the Makefile.

    Any help would be much appreciated.

    Thanks

  • #2
    Any thoughts?

    Comment


    • #3
      You do not give sufficient detail (input data, command line) to debug the problem. Thanks!

      Comment


      • #4
        Hey Nils,

        Sorry, didn't realize that. Everything has been run pretty standard - here's an example:

        bwa aln -q 5 -l 32 -k 2 -o 1 -t 4 database.fasta reads.fastq > aligned.sai

        As I said - the '-t 4' parameter is the problem as bwa doesn't appear to be able to use multiple cores on my install - and in fact, when I use -t 4, it will slow down the program compared to if that option isn't used.

        I have tried rebuilding bwa from source (two different versions - - either using 'make' or the 'sh autogen.sh' method - neither works).

        At the end of the day, I guess bwa isn't very suited for my purposes (my data is quite divergent), but the Broad is using bwa as their pipeline, so I will have to try and work with it for now - multi threading would help ;-).

        Comment


        • #5
          Provide more detail and maybe we can help. We run BWA on a 64 core machine and it runs great. It scales nearly perfectly (64 cores = 64X speed up).

          Comment


          • #6
            make fails for bwa?
            What's the error message?

            Make sure you're compiling with -DHAVE_PTHREAD When you type make you should see this "string" as parameter to gcc. If you don't have the pthread library, then it won't link.

            Comment


            • #7
              make appears to be working and I don't see any errors popping up. On a fresh install, this is what is written to the terminal window:

              $make
              make[1]: Nothing to be done for `lib'.
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD bwt_gen.c -o bwt_gen.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD QSufSort.c -o QSufSort.o
              ar -cru libbwtgen.a bwt_gen.o QSufSort.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD utils.c -o utils.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD bwt.c -o bwt.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD bwtio.c -o bwtio.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD bwtaln.c -o bwtaln.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD bwtgap.c -o bwtgap.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD is.c -o is.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD bntseq.c -o bntseq.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD bwtmisc.c -o bwtmisc.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD bwtindex.c -o bwtindex.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD stdaln.c -o stdaln.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD simple_dp.c -o simple_dp.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD bwaseqio.c -o bwaseqio.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD bwase.c -o bwase.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD bwape.c -o bwape.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD kstring.c -o kstring.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD cs2nt.c -o cs2nt.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD bwtsw2_core.c -o bwtsw2_core.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD bwtsw2_main.c -o bwtsw2_main.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD bwtsw2_aux.c -o bwtsw2_aux.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD bwt_lite.c -o bwt_lite.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD bwtsw2_chain.c -o bwtsw2_chain.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD bamlite.c -o bamlite.o
              gcc -c -g -Wall -O2 -DHAVE_PTHREAD main.c -o main.o
              gcc -g -Wall -O2 -DHAVE_PTHREAD utils.o bwt.o bwtio.o bwtaln.o bwtgap.o is.o bntseq.o bwtmisc.o bwtindex.o stdaln.o simple_dp.o bwaseqio.o bwase.o bwape.o kstring.o cs2nt.o bwtsw2_core.o bwtsw2_main.o bwtsw2_aux.o bwt_lite.o bwtsw2_chain.o bamlite.o main.o -o bwa -lm -lz -lpthread -Lbwt_gen -lbwtgen

              My computer is a Mac Pro running 10.6.8 and as far as I can tell pthread should be there. I just use the standard Makefile with no modifications - everything appears to be in order - here's a copy of the file:

              CC= gcc
              CXX= g++
              CFLAGS= -g -Wall -O2
              CXXFLAGS= $(CFLAGS)
              DFLAGS= -DHAVE_PTHREAD #-D_FILE_OFFSET_BITS=64
              OBJS= utils.o bwt.o bwtio.o bwtaln.o bwtgap.o is.o \
              bntseq.o bwtmisc.o bwtindex.o stdaln.o simple_dp.o \
              bwaseqio.o bwase.o bwape.o kstring.o cs2nt.o \
              bwtsw2_core.o bwtsw2_main.o bwtsw2_aux.o bwt_lite.o \
              bwtsw2_chain.o bamlite.o
              PROG= bwa
              INCLUDES=
              LIBS= -lm -lz -lpthread -Lbwt_gen -lbwtgen
              SUBDIRS= . bwt_gen

              .SUFFIXES:.c .o .cc

              .c.o:
              $(CC) -c $(CFLAGS) $(DFLAGS) $(INCLUDES) $< -o $@
              .cc.o:
              $(CXX) -c $(CXXFLAGS) $(DFLAGS) $(INCLUDES) $< -o $@

              all:$(PROG)

              lib-recur all-recur clean-recur cleanlocal-recur install-recur:
              @target=`echo $@ | sed s/-recur//`; \
              wdir=`pwd`; \
              list='$(SUBDIRS)'; for subdir in $$list; do \
              cd $$subdir; \
              $(MAKE) CC="$(CC)" CXX="$(CXX)" DFLAGS="$(DFLAGS)" CFLAGS="$(CFLAGS)" \
              INCLUDES="$(INCLUDES)" $$target || exit 1; \
              cd $$wdir; \
              done;

              lib:

              bwa:lib-recur $(OBJS) main.o
              $(CC) $(CFLAGS) $(DFLAGS) $(OBJS) main.o -o $@ $(LIBS)

              bwt.o:bwt.h
              bwtio.o:bwt.h
              bwtaln.o:bwt.h bwtaln.h kseq.h
              bwt1away.o:bwt.h bwtaln.h
              bwt2fmv.o:bwt.h
              bntseq.o:bntseq.h
              bwtgap.o:bwtgap.h bwtaln.h bwt.h

              bwtsw2_core.o:bwtsw2.h bwt.h bwt_lite.h stdaln.h
              bwtsw2_aux.o:bwtsw2.h bwt.h bwt_lite.h stdaln.h
              bwtsw2_main.o:bwtsw2.h

              cleanlocal:
              rm -f gmon.out *.o a.out $(PROG) *~ *.a

              clean:cleanlocal-recur

              Comment


              • #8
                Just in case anybody else ran into this problem. The lack of multi-threading was caused by the size of the reference file I was using - my genome is only 4kb, so I guess bwa couldn't throw all the data at 8 cores - this just slowed down the program, rather than speeding it up. Using hg19 as a reference, everything is working great.

                Comment

                Latest Articles

                Collapse

                • seqadmin
                  Best Practices for Single-Cell Sequencing Analysis
                  by seqadmin



                  While isolating and preparing single cells for sequencing was historically the bottleneck, recent technological advancements have shifted the challenge to data analysis. This highlights the rapidly evolving nature of single-cell sequencing. The inherent complexity of single-cell analysis has intensified with the surge in data volume and the incorporation of diverse and more complex datasets. This article explores the challenges in analysis, examines common pitfalls, offers...
                  06-06-2024, 07:15 AM
                • seqadmin
                  Latest Developments in Precision Medicine
                  by seqadmin



                  Technological advances have led to drastic improvements in the field of precision medicine, enabling more personalized approaches to treatment. This article explores four leading groups that are overcoming many of the challenges of genomic profiling and precision medicine through their innovative platforms and technologies.

                  Somatic Genomics
                  “We have such a tremendous amount of genetic diversity that exists within each of us, and not just between us as individuals,”...
                  05-24-2024, 01:16 PM

                ad_right_rmr

                Collapse

                News

                Collapse

                Topics Statistics Last Post
                Started by seqadmin, Yesterday, 07:23 AM
                0 responses
                9 views
                0 likes
                Last Post seqadmin  
                Started by seqadmin, 06-17-2024, 06:54 AM
                0 responses
                12 views
                0 likes
                Last Post seqadmin  
                Started by seqadmin, 06-14-2024, 07:24 AM
                0 responses
                24 views
                0 likes
                Last Post seqadmin  
                Started by seqadmin, 06-13-2024, 08:58 AM
                0 responses
                18 views
                0 likes
                Last Post seqadmin  
                Working...
                X