tophat error with large file

[smleighton]

I am getting errors running large rna files. If the file is larger than about 140 GB then I get the following error. If I split the files up and run each individually then it works fine. Does anyone know if there is a limit to the size of input to the tophat application?

[Wed Mar 21 00:07:31 2012] Beginning TopHat run for step PreProcess (v1.3.1)
-----------------------------------------------
[Wed Mar 21 00:07:31 2012] Preparing output location ./
[Wed Mar 21 00:07:32 2012] Checking for Bowtie index files
[Wed Mar 21 00:07:32 2012] Checking for reference FASTA file
[Wed Mar 21 00:07:32 2012] Checking for Bowtie
Bowtie version: 0.12.7.0
[Wed Mar 21 00:07:32 2012] Checking for Samtools
Samtools Version: 0.1.18
[Wed Mar 21 00:07:32 2012] Generating SAM header for /hg19/data/hg19
format: fastq
quality scale: solexa33 (reads generated with GA pipeline version < 1.3)
[Wed Mar 21 00:07:54 2012] Preparing reads
[FAILED]
Error running 'prep_reads'
Error: beginning of quality values record not found! (@CCFFFDDHHHHHJJJJJJHIJJIJIGIIIGJIJJIIIIIGJDHIJJJJIJ>F=FHHIIIIHHHGHHFFDDDEEECEEDCDEDDDDCDDDDBDDDDDCCC)

[aoifemcm]

I'm having exactly the same problem. Anyone have a suggestion?

I'm using tophat v2.0.10.

[WhatsOEver]

Did you concatenate the files somehow? The fastq format is normally easy to parse as a new entry has to have 4 lines (header, seq, def, qual). If this order gets somehow corrupted, such errors can appear (It looks to me like tophat identifies the "@CCFFF..." as the header, but its actually the quality line...).

what do you get from the following command:

grep "@CCFFFDDHHHHHJJJJJJHIJJIJIGIIIGJIJJIIIIIGJDHIJJJJIJ>F=FHHIIIIHHHGHHFFDDDEEECEEDCDEDDDDCDDDDBDDDDDCCC" -A 4 -B 4 ./yourFastqFile.fastq

[EDIT]
@aoifemcm: you need to exchange "@CCF..." with what you have in the error, of course

[aoifemcm]

Yes, these files were concatenated. Tophat ran perfectly well on each constituent file separately, and when I redid the concatenation it solved the problem.
I guess corruption was at fault.

Thanks!

[WhatsOEver]

Though its good to hear you solved your problem, I would be very interested to hear how you concatenated the files and if the error is reproducible.