bcftools is slow

[craigt]

Hi,

I am subsetting a vcf by positions stored in a tab delimited file using bcftools. I noticed the program is very slow. Here is the command:

bcftools view -R ./chr1.passing.markers.txt chr1.vcf.gz -Oz -o ./chr1.reduced.vcf.gz

where chr1.passing markers is tab delimited chromosome and position for muliple positions, no header. 68K positions. Original vcf has 524K positions.

The bcftools command is not using NFS (reading/writing to local disk, executable running from the analysis directory), no competing jobs. It is taking a really long time. Still running after 120 minutes.

I wrote an equivalent perl script that completes this in 10 minutes but uses flat files and so should be even slower than bcftools with its binary file format.

bcftools version is up to date.

Does anyone have an idea how I can speed this up or what might be wrong?

Thanks,
Craig

[Markiyan]

RDBMS are usually more efficient with these types of queries...

It looks like it tries to ungzip and scan whole input file for each chromosome postion range...
So to do it 68 thousand times... it takes a bit of time...

to fix:
1. Your perl script need the following in the VCF open section in order to be able to read gzipped files:

if($vcf_file_in=~m/\.gz$/i or $vcf_file_in=~m/\.Z$/i){
open (VCF_IN, "zcat $vcf_file_in |") or die "\nUnable to open gzipped vcf input file: $vcf_file_in\n";
}else{
open (VCF_IN, $vcf_file_in) or die "\nUnable to open vcf input file: $vcf_file_in\n";
}


OR:
2. try running it on ungzipped input file...

OR:
3. If you are good with perl, DBI, SQL and MySQL/postgres/etc you can try loading input vcf file into mysql table(s), index it properly and run a set of SQL queries to select needed data.
PS: Make sure to crank up the MySQL server memory limits (/etc/my.cnf) before attempting to do it...

[craigt]

"It looks like it tries to ungzip and scan whole input file for each chromosome postion range...
So to do it 68 thousand times... it takes a bit of time..."

> It is my impression that the program doesn't unzip anything, I believe it streams an indexed binary file and writes a binary file, that can also be indexed.

The call I made is very standard, if it is poorly phrased, let me know.

Has anyone experienced this issue of slow performance with bcftools compared to some other benchmark program and been able to resolve it?

Craig

[jkbonfield]

If you don't have an index or have *many* regions, you're probably better off using -T instead of -R which will stream and filter. I have vague recollections though that there may be a difference too regarding overlapping regions and whether records in the overlaps get reported once or twice. (Using an index would report them twice IIRC.)

-R attempts to use the index, which *should* be faster than streaming, but only if the density is low.

Eg if you're pulling 68000 positions each of which will mean seeking and uncompressing a gzip block (ideally it'd cache, but this may not be done) then possibly it's actually decoding more than the entire files worth of data, in which case streaming (-T) becomes more efficient. On the flip side, if you have a 10Gb file and 2 positions, seeking and decoding will obviously be faster.

[craigt]

Thank you for your response. I am pulling a lot of regions from the original file. I will try -T and see if there is some improvement.

Thanks again!

Craig