Fast BAM sorting

[genomeHunter]

Hello,

I have lots of unsorted BAM files and the sorting process is very slow. One solution I could think of was to run multiple instances of samtools:

cat files.txt | xargs -n 1 -P $num_threads samtools sort

The problem is that this method is I/O bound. Are there any (free) faster alternatives?

Should I try novosort?

Cheers
GH

[Heisman]

Are you aware you can use samtools sort with multiple threads with the "-@" option?

[GenoMax]

If you are doing this on a cluster (with a storage subsystem that can support multiple I/O streams) Heisman's solution will work.

If you are I/O bound on a single server then your only option is patience.

[genomeHunter]

Are you aware you can use samtools sort with multiple threads with the "-@" option?

No, I was not. I saw the "-@" option in a fork of samtools, I guess it was called samtools-mt. I did not know it was added to samtools.

Thanks,
GH

[genomeHunter]

If you are doing this on a cluster (with a storage subsystem that can support multiple I/O streams) Heisman's solution will work.

If you are I/O bound on a single server then your only option is patience.

I tried it on a moderate-sized BAM file and it was not very fast on our mini-server (32 Cores and RAID-6 disk). I guess I will be using xargs :-)

Cheers,
GH

[Richard Finney]

Please note the -m parameter for samtools sort , this is the amount of memory to use.
Using available memory will speed up thing. This also works for novocraft's novosort. You may also wish to copy to a fast disk for any temp files that spill to disk.

[genomeHunter]

Please note the -m parameter for samtools sort , this is the amount of memory to use.
Using available memory will speed up thing. This also works for novocraft's novosort. You may also wish to copy to a fast disk for any temp files that spill to disk.

Thanks. I am using 4GB/threads and 16 threads. Our main disk is a 12-disk RAID-6 arrays with enterprise-level HDDs.

Do you know how does samtools sort compare to SortSam and NovoSort?

Cheers,
GH

[GenoMax]

Our main disk is a 12-disk RAID-6 arrays with enterprise-level HDDs.

Cheers,
GH

How is it connected to the server: 6 Gbps SAS/8 Gbps Fiber Channel HBA?

[genomeHunter]

How is it connected to the server: 6 Gbps SAS/8 Gbps Fiber Channel HBA?

I am not 100% sure but it must be 6 Gbps SAS.

GH

[lh3]

Multi-threading is available in the released package. Novosort is faster than samtools.

[genomeHunter]

Multi-threading is available in the released package. Novosort is faster than samtools.

Do you have any rough speedup factors? I tried

samtools view -buS exp.sam | samtools sort -@ 16 -m 4G - sorted

And it takes 10 minutes for a moderate size SAM file.

Cheers,
GH

[GenoMax]

GH: Here are some systems tools to check if any hardware bottlenecks exist on your system. Start with the "iostat": http://www.thegeekstuff.com/2011/07/...stat-examples/