why should BAM be shuffled before extracting to FASTQ?

[splaisan]

I came across this apparent bias when recovering FASTQ paired read data from BAM for re-mapping purpose (public data obtained from SRA or new reference genome available).

It seems that if I extract reads from an existing BAM file, the order in which the reads are presented in the BAM (and therefore extracted thereof) will affect the later remapping results!

People having a piece of answer told me that the bias affects computation of the read insert distance during remapping and will 'tend' to reproduce results obtained during the first alignment.

This is second-hand information (thanks a lot to Geraldine who shared this in http://gatkforums.broadinstitute.org...o-fastq-format) but is not very satisfactory and I would like to read a more conclusive discussion.

Thanks to all who know the detailed answer and could make it clear for us all.

[dpryan]

The explanation given at the link is actually pretty good. What part of that wasn't satisfactory? I should note that if the input BAM file is name-sorted (or even unsorted) then you probably don't need to shuffle things (it'l. The idea is that the insert sizes that will map to a single region of the genome might not be representative of the entire experiment. If you don't shuffle the reads, then your aligner might estimate the insert size incorrectly, which will slightly bias the alignments downstream (obviously, bias is an issue if you're going to call SNPs).

[splaisan]

Thanks for this answer, it already sheds some light.

No part is unsatisfactory but the process leading to a wrong estimate is not really explained (or I did not understand it correctly). Is it only the initial step where BWA collects a read sample and measures distance to fine tune the remaining alignments?

In my understanding, the bias was present also in name-sorted BAM which did not make sense to me since the only thing exported to FASTQ are seq and quals.

Considering that public BAM often only report nicely mapped reads I guess that the bias is unavoidable, right!

S

[dpryan]

To a certain extent yes, particularly if the original authors didn't do a good job with things. It's probably a good idea to be cautious if you can't get the original fastq files.