Sorting BAM by chromosome produces a much larger file.

[aramadhan]

Hi,

I am working with BAM files and I need to sort them.

I've realized that sorting by chromosome produces a file that's around the same size as the original BAM file but then I cannot index it. When I sort it by read name, I can index it, but the resulting BAM is usually significantly larger than the original, sometimes as much as three times larger. Can't I sort by chromosome and have it be indexable?

Is this normal behaviour or can someone explain what's going on here? I've only found one other thread discussing this but all I understood is that you must sort by read name.

Thank you in advance.

EDIT: I forgot to mention that I am using samtools 0.1.17 on Linux if it matters.

[Richard Finney]

indexing on sorted by chromosome bam should definitely work.

Here are some diagnostics to try ...

1) check header.
2) what is the samtools sort command that produces a sorted bam that fails indexing.
3) what is the error output from the samtools index command ?
4) can you dump the bam to a sam file that looks right?

Latest samtools is 0.18 , but yours should be good enough.

[aramadhan]

1. Hmmm, the two headers seem exactly the same.

2. I cannot index it when I use

samtools sort -on original.bam > sorted_on.bam

although emitting the -o flag still produces an unindexable BAM file.

3. I get this

$ samtools index sorted_on.bam 
[bam_index_core] the alignment is not sorted (HWUSI-EAS1720_0021_FC63A8AAAXX:2:1:1330:17918#0/1): 13-th chr > 12-th chr
[bam_index_build2] fail to index the BAM file.

and similar output for other BAM's except for the chromosomes being different. Trying to index the one sorted without -n produces a perfectly good index file.

4. Yes I can. When I sort it with -on, the SAM file looks fine and is sorted by read name and when I sort it without -n, it's definitely sorted by chromosome. The two files have the same number of lines as well so I'm assuming that one is much easier to compress but that still doesn't explain why I can't index it when I sort by chromosome.
Here are the first few reads.

Thank you.

[BAMseek]

The -n means "Sort by read names rather than by chromosomal coordinates". If you do this, you won't be able to index it. The file will need to be sorted by chromosomal coordinates for you to index (which is why you get the alignment is not sorted error).

As far as why one is bigger than the other - it may just be that gzip has an easier time compressing the file when sorted by read name rather than chromosomal position.

Justin

[swbarnes2]

I'm a bit confused. sort -n should sort by read name, not chromosome. So if index is failing to sort that because it's not sorted by chromosome, it's because it's not.

Also,

sort in.bam sort

is the same as

sort -o in.bam > sort.bam

[aramadhan]

Ah I see. Thank you guys for clearing that up. I was not aware that indexing was only possible if the BAM is sorted by chromosome.

Would anyone know why though? I've only started dealing with these files and googling hasn't really helped.

Thanks!

[etorstenson]

The index is used as a way for a program to very quickly find a reasonable starting point for the relevant information. I'm assuming it's used when you want to extract reads from a particular region. Without the index files, and without sorting by chromosome, the program would have to scan ALL reads within the bam which could be very time consuming. With the index, it knows that it can start a very specific position and read until it has finished it's job.

The compression algorithm used in BAM compression is very sensitive to certain types of patterns. So, by simply moving reads around, you'll find that the end file size will be very different even if the actual amount of data didn't change at all.