Speeding up .sam to .vcf creation

[swbarnes2]

Is there a reason you have a .sam file?

whatever is making your .sam file, you should pipe that into samtools view -bS and convert it to a .bam on the fly. .sam files are so huge, you should never have your whole data set as one. You should always pipe straight into samtools view, and make .sam file of subsets of your data, if you have to eyeball it.

You can take a peek at your .bcf by running bcftools view on it to convert it to vcf as it's being made. (It might complain about the last line having not enough columns, but that doesn't matter) Just see if it makes sense. You'd hate to wait forever and find that something is wrong.

I used to pipe view into sort, and then one day, it spontaneously stopped working. It was making no temp files at all, so I stopped doing it. But if you are only doing one .bam file, you can probably pipe that into mpileup too.

[Richard Finney]

You can get some small improvements by
1) piping your commands together (view | sort | mpileup | bcftools view > output.bcf)
2) specifying -m (use memory) parameters on sort. Use as much memory as you can get away with.

There are "multi-thread bam sorters" but they don't offer much real speed up.

It's not unusual for it to take several days to process a bam file in this fashion.

You probably have multiple cores on your computer. You can break down the bam|mpileup>bcf part into chunks (by chromosome?) and run them in parallel. example: samtools view x.bam chr22 | samtools mpileup |bcftools > x22.bcf
Be sure to index the bam file if you choose this method.

[falarkys]

I'm trying to get .vcf files from a .sam file via the following commands:

samtools view -bt ${fai} ${prefix}.sam > ${prefix}_unsorted.bam
samtools sort ${prefix}_unsorted.bam ${prefix} 
samtools mpileup -ugf ${fasta} ${prefix}.bam | bcftools view -bvcg -p - > ${prefix}.raw.bcf

The sorted .bam file is ~130GB, and the final command I'm running seems like it will take forever to complete. It's been running for a couple of days, and the .raw.bcf file is only 85MB so far.

Is this normal? Is there an option in samtools/bcftools for multithreading, or am I doing something incorrectly?

Thanks!