BAM speed: i/o, uncompress, etc.

[cycomatto]

Is anybody familiar with the relative speeds of .bam file i/o and decompression? My algorithm is bottlenecking when I try to load reads from a .bam file.

Using BAMTools, I jump to certain, specific locations in the genome, and load the reads in a small region there (this involves decompression). I do this many, many times, loading many reads.

I am thinking it may be more efficient to create a mini-BAM file (only the regions of concern), then load this into memory (cache it), and then search and load the reads. But will the required decompressing from the compressed .bam format still make things too slow?

[Richard Finney]

What data structure are you using to store the "loaded reads"?
Seeking to an indexed bam file, reading in and uncompromising the data should not be a big problem.

[maubp]

You could try using an uncompressed BAM file (use the samtools view -u option), and see if that helps.

Also does BAMtools attempt to cache any of the data to avoid repeated reading and decompressing the same BGZF blocks?

[cycomatto]

What data structure are you using to store the "loaded reads"?
Seeking to an indexed bam file, reading in and uncompromising the data should not be a big problem.

I am using the BamTools::BamAlignment data structure (std::string for qualities, read name, sequence, etc. bunch of ints and some bools)

[jkbonfield]

If you're on a linux box I'd advise "strace -o trace.out -T program args" to list the system calls and times taken. This can help a lot in seeing what takes up the time.

Generally random access is always going to be slow as disk seeks are typically slow. Some filesystems are better than others here too. Ultimately if you're doing a LOT of random access it may prove faster to copy it off a slow storage and onto something quick before doing the random I/O, eg a solid-state drive, or at the very least /tmp so the OS can do a better job of caching local data without making assumptions on networked I/O.

A quick test: is it faster to do a full "samtools view filename.bam > /dev/null" than to do your random I/O? If so then seeking is probably your bottleneck.

Do you have the option to sort the random access prior to running the algorithm? I.e. to ensure the small portions you're reading are always sequential.

[adaptivegenome]

I/O is a big bottleneck. I would not recommend using uncompressed SAMs because then you have even more data to move around. Is your tool parallelized? If so BAM reading could be a bottleneck because bamtools implements its IO serially.