parallelizing MAQ & BWA

[mhmckm]

I'm a guy who involved in Bio-informatics.
Theses days, we have a lot of intesests in genome sequencing
and consider some mapping programs, especially MAQ & BWA.

We are trying to run MAQ & BWA in parallel way,
having a some questions.

1. How to divide & distribute the data?
We have two possible options for this.
- divide the reads.
- divide the reference. (1 chromosome per machine)
For example, ABi Corona recommends us to divide the genome.
What is the option MAQ & BWA recommend?

2. Does MAQ processs color-space data efficiently?
We cannnot find any text about how to process CS data with MAQ.
Could you give us some tutorial about processing CS data?

We are looking forward to your valuable advice.

[nilshomer]

I'm a guy who involved in Bio-informatics.
Theses days, we have a lot of intesests in genome sequencing
and consider some mapping programs, especially MAQ & BWA.

We are trying to run MAQ & BWA in parallel way,
having a some questions.

1. How to divide & distribute the data?
We have two possible options for this.
- divide the reads.
- divide the reference. (1 chromosome per machine)
For example, ABi Corona recommends us to divide the genome.
What is the option MAQ & BWA recommend?

Divide the reads definitely. With each slide of a SOLiD machine generating 200M-500M of paired end reads this is a must. As for the reference (say human), given enough memory (~4Gb), for BWA and MAQ, you do not need to divide the reference. Other programs you may need to given RAM requirements (please let me know your RAM availability).

2. Does MAQ processs color-space data efficiently?
We cannnot find any text about how to process CS data with MAQ.
Could you give us some tutorial about processing CS data?

We are looking forward to your valuable advice.

MAQ and BWA are written by the same author, so they work similarly. BWA can find small indels (1-3bp) whereas MAQ cannot. Both are meant for low-error data (<%2 error rates) and so you may to either tune them for intrinsic higher error-rate SOLiD data (use the -n option), which will dramatically reduce the running times. For longer indels and higher sensitivity, consider BFAST, which is admittedly my own software.

What genomes do you plan to sequence (human)?

[henry]

I'm a guy who involved in Bio-informatics.
Theses days, we have a lot of intesests in genome sequencing
and consider some mapping programs, especially MAQ & BWA.

We are trying to run MAQ & BWA in parallel way,
having a some questions.

1. How to divide & distribute the data?
We have two possible options for this.
- divide the reads.
- divide the reference. (1 chromosome per machine)
For example, ABi Corona recommends us to divide the genome.
What is the option MAQ & BWA recommend?

2. Does MAQ processs color-space data efficiently?
We cannnot find any text about how to process CS data with MAQ.
Could you give us some tutorial about processing CS data?

We are looking forward to your valuable advice.

divide reads, after mapping to the reference, MAQ can merge diferent bach of reads together.

As for the second question, I'm also searching for open source tools that can be applied to process ABI Solid data. BFAST may deserve a try, if you concern about higher sensitivity.

Best

[dawe]

We are trying to run MAQ & BWA in parallel way,
having a some questions.

1. How to divide & distribute the data?
We have two possible options for this.
- divide the reads.
- divide the reference. (1 chromosome per machine)
For example, ABi Corona recommends us to divide the genome.
What is the option MAQ & BWA recommend?

Hi, if you are using BWA and speed is your concern you may try to align with the "-t" option, which enables multithread analysis (i.e. you don't need to split the data).
About splitting the data... One should ask to the algorithm developer to understand how it scales on the size of the genome and on the size of the input (and length of reads...). Suppose it scales O(log(n)) with the genome and O(n) with the input, I think you should split the data, but in the end you will test both, just to see what's better (and results may differ for differently sized genomes...).
On the other side, splitting the genome in a "persistent" way is less flexible, as you will need always a certain machine for a certain chromosome (unless you don't split each time).
HTH
d

[henry]

btw, I also used ABI corona lite. to my experience, it's a disaster. it is not a good tool. it should be worst option, for your consideration,

[der_eiskern]

i get the impression that corona lite and maq can map the same amount of sequence for you usually but maq is easier to use. there are differences in the # of reads mapped independently to a given location though. i still haven't figured out how corona lite really works compared to bfast and maq.

changing -n with maq .map function doesn't affect the total % mappable sequence on the data that i've ran when colorspace is used.

[jperin]

I'm sure you've found a solution by now, but we split fastq files using either maq's built in tool for doing this, which makes it quite easy to split your reads, or by simply executing the split command in linux and creating batch jobs for bwa. With BWA its also nice to use the -t multi threading option. I tend to stick to about 2 million reads per input. Splitting the reference is not a good idea, and may result in somewhat inaccurate results IMO. Merging the results its also quite easy. In MAQ you can do this automatically with merge, with BWA it's best to use samtools to merge your results once you've sorted your bam file.

[KevinLam]

I have a question as well. admittedly its more bwa related.
for bwa
how does the -t (multithread) option differ from splitting the reads fasta itself and running on a single core?
i.e. should i
a) split reads into 20 and run bwa as a single thread in a PBS cluster
or
b) split reads into 5 nodes and run bwa as -t 4 in the same PBS cluster

[jperin]

I've found that a combination of both works best. Split -t #cores on your node, and split your input reads in about 2million reads per file chunks, more if you have plenty of memory.

[KevinLam]

I've found that a combination of both works best. Split -t #cores on your node, and split your input reads in about 2million reads per file chunks, more if you have plenty of memory.

ic.
if I have 2 GB ram per core is that enough for 2 million reads?

[jperin]

yes, but i probably wouldn't thread more than 2-4 on a node with 2GB.

[geschickten]

Well all I can say is if you want to use MAQ on many cores or on a cluster ( built using commodity machines) then please try our version at http://www.geschickten.com/PaCGeE.html

Your suggestions and feedback are always appreciated. Thank you.