Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • parallelizing MAQ & BWA

    I'm a guy who involved in Bio-informatics.
    Theses days, we have a lot of intesests in genome sequencing
    and consider some mapping programs, especially MAQ & BWA.

    We are trying to run MAQ & BWA in parallel way,
    having a some questions.

    1. How to divide & distribute the data?
    We have two possible options for this.
    - divide the reads.
    - divide the reference. (1 chromosome per machine)
    For example, ABi Corona recommends us to divide the genome.
    What is the option MAQ & BWA recommend?

    2. Does MAQ processs color-space data efficiently?
    We cannnot find any text about how to process CS data with MAQ.
    Could you give us some tutorial about processing CS data?

    We are looking forward to your valuable advice.

  • #2
    Originally posted by mhmckm View Post
    I'm a guy who involved in Bio-informatics.
    Theses days, we have a lot of intesests in genome sequencing
    and consider some mapping programs, especially MAQ & BWA.

    We are trying to run MAQ & BWA in parallel way,
    having a some questions.

    1. How to divide & distribute the data?
    We have two possible options for this.
    - divide the reads.
    - divide the reference. (1 chromosome per machine)
    For example, ABi Corona recommends us to divide the genome.
    What is the option MAQ & BWA recommend?
    Divide the reads definitely. With each slide of a SOLiD machine generating 200M-500M of paired end reads this is a must. As for the reference (say human), given enough memory (~4Gb), for BWA and MAQ, you do not need to divide the reference. Other programs you may need to given RAM requirements (please let me know your RAM availability).

    2. Does MAQ processs color-space data efficiently?
    We cannnot find any text about how to process CS data with MAQ.
    Could you give us some tutorial about processing CS data?

    We are looking forward to your valuable advice.
    MAQ and BWA are written by the same author, so they work similarly. BWA can find small indels (1-3bp) whereas MAQ cannot. Both are meant for low-error data (<%2 error rates) and so you may to either tune them for intrinsic higher error-rate SOLiD data (use the -n option), which will dramatically reduce the running times. For longer indels and higher sensitivity, consider BFAST, which is admittedly my own software.

    What genomes do you plan to sequence (human)?

    Comment


    • #3
      Originally posted by mhmckm View Post
      I'm a guy who involved in Bio-informatics.
      Theses days, we have a lot of intesests in genome sequencing
      and consider some mapping programs, especially MAQ & BWA.

      We are trying to run MAQ & BWA in parallel way,
      having a some questions.

      1. How to divide & distribute the data?
      We have two possible options for this.
      - divide the reads.
      - divide the reference. (1 chromosome per machine)
      For example, ABi Corona recommends us to divide the genome.
      What is the option MAQ & BWA recommend?

      2. Does MAQ processs color-space data efficiently?
      We cannnot find any text about how to process CS data with MAQ.
      Could you give us some tutorial about processing CS data?

      We are looking forward to your valuable advice.
      divide reads, after mapping to the reference, MAQ can merge diferent bach of reads together.

      As for the second question, I'm also searching for open source tools that can be applied to process ABI Solid data. BFAST may deserve a try, if you concern about higher sensitivity.

      Best

      Comment


      • #4
        Originally posted by mhmckm View Post
        We are trying to run MAQ & BWA in parallel way,
        having a some questions.

        1. How to divide & distribute the data?
        We have two possible options for this.
        - divide the reads.
        - divide the reference. (1 chromosome per machine)
        For example, ABi Corona recommends us to divide the genome.
        What is the option MAQ & BWA recommend?
        Hi, if you are using BWA and speed is your concern you may try to align with the "-t" option, which enables multithread analysis (i.e. you don't need to split the data).
        About splitting the data... One should ask to the algorithm developer to understand how it scales on the size of the genome and on the size of the input (and length of reads...). Suppose it scales O(log(n)) with the genome and O(n) with the input, I think you should split the data, but in the end you will test both, just to see what's better (and results may differ for differently sized genomes...).
        On the other side, splitting the genome in a "persistent" way is less flexible, as you will need always a certain machine for a certain chromosome (unless you don't split each time).
        HTH
        d

        Comment


        • #5
          btw, I also used ABI corona lite. to my experience, it's a disaster. it is not a good tool. it should be worst option, for your consideration,

          Comment


          • #6
            i get the impression that corona lite and maq can map the same amount of sequence for you usually but maq is easier to use. there are differences in the # of reads mapped independently to a given location though. i still haven't figured out how corona lite really works compared to bfast and maq.

            changing -n with maq .map function doesn't affect the total % mappable sequence on the data that i've ran when colorspace is used.

            Comment


            • #7
              I'm sure you've found a solution by now, but we split fastq files using either maq's built in tool for doing this, which makes it quite easy to split your reads, or by simply executing the split command in linux and creating batch jobs for bwa. With BWA its also nice to use the -t multi threading option. I tend to stick to about 2 million reads per input. Splitting the reference is not a good idea, and may result in somewhat inaccurate results IMO. Merging the results its also quite easy. In MAQ you can do this automatically with merge, with BWA it's best to use samtools to merge your results once you've sorted your bam file.

              Comment


              • #8
                I have a question as well. admittedly its more bwa related.
                for bwa
                how does the -t (multithread) option differ from splitting the reads fasta itself and running on a single core?
                i.e. should i
                a) split reads into 20 and run bwa as a single thread in a PBS cluster
                or
                b) split reads into 5 nodes and run bwa as -t 4 in the same PBS cluster
                http://kevin-gattaca.blogspot.com/

                Comment


                • #9
                  I've found that a combination of both works best. Split -t #cores on your node, and split your input reads in about 2million reads per file chunks, more if you have plenty of memory.

                  Comment


                  • #10
                    Originally posted by jperin View Post
                    I've found that a combination of both works best. Split -t #cores on your node, and split your input reads in about 2million reads per file chunks, more if you have plenty of memory.
                    ic.
                    if I have 2 GB ram per core is that enough for 2 million reads?
                    http://kevin-gattaca.blogspot.com/

                    Comment


                    • #11
                      yes, but i probably wouldn't thread more than 2-4 on a node with 2GB.

                      Comment


                      • #12
                        Well all I can say is if you want to use MAQ on many cores or on a cluster ( built using commodity machines) then please try our version at http://www.geschickten.com/PaCGeE.html

                        Your suggestions and feedback are always appreciated. Thank you.

                        Comment

                        Latest Articles

                        Collapse

                        • seqadmin
                          Strategies for Sequencing Challenging Samples
                          by seqadmin


                          Despite advancements in sequencing platforms and related sample preparation technologies, certain sample types continue to present significant challenges that can compromise sequencing results. Pedro Echave, Senior Manager of the Global Business Segment at Revvity, explained that the success of a sequencing experiment ultimately depends on the amount and integrity of the nucleic acid template (RNA or DNA) obtained from a sample. “The better the quality of the nucleic acid isolated...
                          03-22-2024, 06:39 AM
                        • seqadmin
                          Techniques and Challenges in Conservation Genomics
                          by seqadmin



                          The field of conservation genomics centers on applying genomics technologies in support of conservation efforts and the preservation of biodiversity. This article features interviews with two researchers who showcase their innovative work and highlight the current state and future of conservation genomics.

                          Avian Conservation
                          Matthew DeSaix, a recent doctoral graduate from Kristen Ruegg’s lab at The University of Colorado, shared that most of his research...
                          03-08-2024, 10:41 AM

                        ad_right_rmr

                        Collapse

                        News

                        Collapse

                        Topics Statistics Last Post
                        Started by seqadmin, Yesterday, 06:37 PM
                        0 responses
                        12 views
                        0 likes
                        Last Post seqadmin  
                        Started by seqadmin, Yesterday, 06:07 PM
                        0 responses
                        10 views
                        0 likes
                        Last Post seqadmin  
                        Started by seqadmin, 03-22-2024, 10:03 AM
                        0 responses
                        52 views
                        0 likes
                        Last Post seqadmin  
                        Started by seqadmin, 03-21-2024, 07:32 AM
                        0 responses
                        68 views
                        0 likes
                        Last Post seqadmin  
                        Working...
                        X