bwa - fastq file size limit? and more...

[mikeworth]

hello,

running bwa v0.5.9 i get this error:
fail to open file 'long_path/filename.fastq'. Abort!
Abort (core dumped)

yes, the file does actually exist. i think the problem is that it's too big - 4.6G.
this is one side of illumina paired end reads - 100bp reads and there are 18,517,594 reads in the file. i shortened the file to ~2 million reads and bwa handled it just fine. is there a limit to the size of the fastq input file?

if there is a limit i was thinking i could break up each of the paired end files and create multiple .sai files - but i think that would be problematic on the 'bwa sampe' end. i don't think the 'sampe' command supports multiple pairs of .sai files. maybe i could cat all the .sai files together?

anyone else have a problem like this? too much data!

thanks,
mike

[swbarnes2]

I think what you should do is split the fastq's, since that seems to work, and after you generate multiple bams, use samtools to merge the bams into one file.

[RDW]

Sounds like a system limitation - 64-bit bwa can handle much bigger files than this with quite modest RAM. Are you running a 32-bit OS, or might you have built a 32-bit executable?

[mikeworth]

thanks very much to both of you for your replies.

i was indeed using a 32 bit version of bwa. i added "-m64" to the makefile gcc options and recompiled, creating a 64 bit executable.

the large fastq file was read without error. problem solved. thanks again for the help.
mike