Header Leaderboard Ad

Collapse

parallelizing MAQ & BWA

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • parallelizing MAQ & BWA

    I'm a guy who involved in Bio-informatics.
    Theses days, we have a lot of intesests in genome sequencing
    and consider some mapping programs, especially MAQ & BWA.

    We are trying to run MAQ & BWA in parallel way,
    having a some questions.

    1. How to divide & distribute the data?
    We have two possible options for this.
    - divide the reads.
    - divide the reference. (1 chromosome per machine)
    For example, ABi Corona recommends us to divide the genome.
    What is the option MAQ & BWA recommend?

    2. Does MAQ processs color-space data efficiently?
    We cannnot find any text about how to process CS data with MAQ.
    Could you give us some tutorial about processing CS data?

    We are looking forward to your valuable advice.

  • #2
    Originally posted by mhmckm View Post
    I'm a guy who involved in Bio-informatics.
    Theses days, we have a lot of intesests in genome sequencing
    and consider some mapping programs, especially MAQ & BWA.

    We are trying to run MAQ & BWA in parallel way,
    having a some questions.

    1. How to divide & distribute the data?
    We have two possible options for this.
    - divide the reads.
    - divide the reference. (1 chromosome per machine)
    For example, ABi Corona recommends us to divide the genome.
    What is the option MAQ & BWA recommend?
    Divide the reads definitely. With each slide of a SOLiD machine generating 200M-500M of paired end reads this is a must. As for the reference (say human), given enough memory (~4Gb), for BWA and MAQ, you do not need to divide the reference. Other programs you may need to given RAM requirements (please let me know your RAM availability).

    2. Does MAQ processs color-space data efficiently?
    We cannnot find any text about how to process CS data with MAQ.
    Could you give us some tutorial about processing CS data?

    We are looking forward to your valuable advice.
    MAQ and BWA are written by the same author, so they work similarly. BWA can find small indels (1-3bp) whereas MAQ cannot. Both are meant for low-error data (<%2 error rates) and so you may to either tune them for intrinsic higher error-rate SOLiD data (use the -n option), which will dramatically reduce the running times. For longer indels and higher sensitivity, consider BFAST, which is admittedly my own software.

    What genomes do you plan to sequence (human)?

    Comment


    • #3
      Originally posted by mhmckm View Post
      I'm a guy who involved in Bio-informatics.
      Theses days, we have a lot of intesests in genome sequencing
      and consider some mapping programs, especially MAQ & BWA.

      We are trying to run MAQ & BWA in parallel way,
      having a some questions.

      1. How to divide & distribute the data?
      We have two possible options for this.
      - divide the reads.
      - divide the reference. (1 chromosome per machine)
      For example, ABi Corona recommends us to divide the genome.
      What is the option MAQ & BWA recommend?

      2. Does MAQ processs color-space data efficiently?
      We cannnot find any text about how to process CS data with MAQ.
      Could you give us some tutorial about processing CS data?

      We are looking forward to your valuable advice.
      divide reads, after mapping to the reference, MAQ can merge diferent bach of reads together.

      As for the second question, I'm also searching for open source tools that can be applied to process ABI Solid data. BFAST may deserve a try, if you concern about higher sensitivity.

      Best

      Comment


      • #4
        Originally posted by mhmckm View Post
        We are trying to run MAQ & BWA in parallel way,
        having a some questions.

        1. How to divide & distribute the data?
        We have two possible options for this.
        - divide the reads.
        - divide the reference. (1 chromosome per machine)
        For example, ABi Corona recommends us to divide the genome.
        What is the option MAQ & BWA recommend?
        Hi, if you are using BWA and speed is your concern you may try to align with the "-t" option, which enables multithread analysis (i.e. you don't need to split the data).
        About splitting the data... One should ask to the algorithm developer to understand how it scales on the size of the genome and on the size of the input (and length of reads...). Suppose it scales O(log(n)) with the genome and O(n) with the input, I think you should split the data, but in the end you will test both, just to see what's better (and results may differ for differently sized genomes...).
        On the other side, splitting the genome in a "persistent" way is less flexible, as you will need always a certain machine for a certain chromosome (unless you don't split each time).
        HTH
        d

        Comment


        • #5
          btw, I also used ABI corona lite. to my experience, it's a disaster. it is not a good tool. it should be worst option, for your consideration,

          Comment


          • #6
            i get the impression that corona lite and maq can map the same amount of sequence for you usually but maq is easier to use. there are differences in the # of reads mapped independently to a given location though. i still haven't figured out how corona lite really works compared to bfast and maq.

            changing -n with maq .map function doesn't affect the total % mappable sequence on the data that i've ran when colorspace is used.

            Comment


            • #7
              I'm sure you've found a solution by now, but we split fastq files using either maq's built in tool for doing this, which makes it quite easy to split your reads, or by simply executing the split command in linux and creating batch jobs for bwa. With BWA its also nice to use the -t multi threading option. I tend to stick to about 2 million reads per input. Splitting the reference is not a good idea, and may result in somewhat inaccurate results IMO. Merging the results its also quite easy. In MAQ you can do this automatically with merge, with BWA it's best to use samtools to merge your results once you've sorted your bam file.

              Comment


              • #8
                I have a question as well. admittedly its more bwa related.
                for bwa
                how does the -t (multithread) option differ from splitting the reads fasta itself and running on a single core?
                i.e. should i
                a) split reads into 20 and run bwa as a single thread in a PBS cluster
                or
                b) split reads into 5 nodes and run bwa as -t 4 in the same PBS cluster
                http://kevin-gattaca.blogspot.com/

                Comment


                • #9
                  I've found that a combination of both works best. Split -t #cores on your node, and split your input reads in about 2million reads per file chunks, more if you have plenty of memory.

                  Comment


                  • #10
                    Originally posted by jperin View Post
                    I've found that a combination of both works best. Split -t #cores on your node, and split your input reads in about 2million reads per file chunks, more if you have plenty of memory.
                    ic.
                    if I have 2 GB ram per core is that enough for 2 million reads?
                    http://kevin-gattaca.blogspot.com/

                    Comment


                    • #11
                      yes, but i probably wouldn't thread more than 2-4 on a node with 2GB.

                      Comment


                      • #12
                        Well all I can say is if you want to use MAQ on many cores or on a cluster ( built using commodity machines) then please try our version at http://www.geschickten.com/PaCGeE.html

                        Your suggestions and feedback are always appreciated. Thank you.

                        Comment

                        Latest Articles

                        Collapse

                        • seqadmin
                          A Brief Overview and Common Challenges in Single-cell Sequencing Analysis
                          by seqadmin


                          ​​​​​​The introduction of single-cell sequencing has advanced the ability to study cell-to-cell heterogeneity. Its use has improved our understanding of somatic mutations1, cell lineages2, cellular diversity and regulation3, and development in multicellular organisms4. Single-cell sequencing encompasses hundreds of techniques with different approaches to studying the genomes, transcriptomes, epigenomes, and other omics of individual cells. The analysis of single-cell sequencing data i...

                          01-24-2023, 01:19 PM
                        • seqadmin
                          Introduction to Single-Cell Sequencing
                          by seqadmin
                          Single-cell sequencing is a technique used to investigate the genome, transcriptome, epigenome, and other omics of individual cells using high-throughput sequencing. This technology has provided many scientific breakthroughs and continues to be applied across many fields, including microbiology, oncology, immunology, neurobiology, precision medicine, and stem cell research.

                          The advancement of single-cell sequencing began in 2009 when Tang et al. investigated the single-cell transcriptomes
                          ...
                          01-09-2023, 03:10 PM

                        ad_right_rmr

                        Collapse
                        Working...
                        X