bfast match "hangs" while reading index

[sleight82]

When I run bfast match (version 0.7.0a), the program often "hangs" while reading the index (as evidenced by the last line of output to stderr):

Reading index from /path/to/NFSfilesystem/hg19.fasta.nt.2.1.bif.

If I look at the process with "top" it is still active. Generally, loading an index file takes 5-8 minutes (from the timing info provided from bfast); but, often times I've had it take up to 15 hours! This most often happens when I submit the processes to our PBS cluster. When I run them 'locally' (but on the same machines/disks that are in the PBS cluster), I don't have this issue (though the sample size is much smaller than when I submit to the cluster, since I'm submitting 20+ bfast match commands at the same time). I realize this could be a consequence of high disk/network usage (the index files and FASTQ files are located on a shared NFS GPFS filesystem on high performance dedicated fileservers), but I thought I would see if anyone else sees this issue.

An example command line call:

bfast match -A 0 -t -n 16 -f /path/to/NFSfilesystem/hg19.fasta -r /path/to/NFSfilesystem/Sample.fastq.gz  -i 4 -z -T /localdisk/scratch/  > /path/to/NFSfilesystem/D1BUKACXX_MN02_002_4.bmf

Thanks!

[sleight82]

Not directly a problem with BFAST

I've determined this is not an issue with BFAST per se, but a combination with running BFAST match through my ruffus-based pipeline. A similar issue has been noticed when using SGE (I'm using PBS Pro), also not specific to BFAST. It seems to be related to the submission script polling for input (see http://ruffus.org.uk/faq.html under "qrsh eats up all my processor time under ruffus"). Though, in my pipeline, this only happens when using bfast match (not localalign, postprocess, or merge).