SOAPdenovo: Cannot open file... now exit to system!

[mittymat]

Hello.
I am new to genome assembly, and I'm having some problems with SOAP denovo. For some reason, it can't open my fastq file specified in the .config file. I've made sure that the path is correct, and I've tried on both gzipped and unzipped fastq files, but to no avail. I've also double checked that the fastq file has the header before both the sequence and quality data (I read somewhere that this could be a problem). But I'm now out of ideas! Can anyone suggest why it won't open it? It's possible it something really simple that I'm overlooking, so apologies!
Config file:


max_rd_len=50

[LIB]
avg_ins=200
reversed reverse_seq=0
asm_flags=3
rd_len_cutoff=50
rank=1
pair_num_cutoff=3
map_len=17

q=/home/barnes/uvba002/backups/NEXTGEN/Projects/Nesophontes/Data/LANE_8_all.fastq


and the error message I'm getting is:

>SOAPdenovo-31mer pregraph -s LANE_8.small.config -K 23 -o Nes.LANE_8

Version 1.05: released on July 29th, 2010

In LANE_8.small.config, 1 libs, max seq len 50, max name len 256

8 thread created
read from file:
/home/barnes/uvba002/backups/NEXTGEN/Projects/Nesophontes/Data/LANE_8_all.fastq
Cannot open /home/barnes/uvba002/backups/NEXTGEN/Projects/Nesophontes/Data/LANE_8_all.fastq . Now exit to system...

If anyone can help, I'd be very grateful!
Many thanks.

[maubp]

Have you double checked the filename and path - especially if running on a cluster? What about the file's access permissions?

[ians]

can we see the pwd, ls, and head of the sequence?

[mittymat]

Hello,
I did double check the filename and path, which were:

pwd: /home/barnes/uvba002/backups/NEXTGEN/Projects/Nesophontes/Data
filename: LANE_8_all.fastq

header:
less LANE_8_all.fastq
@HWI-ST558:720JEGACXX:8:1101:1570:1981_1:N:0:_GATGGT
CGGCGACAGCACGTCCCGTGATCAGCAAAGCAATGTTCGGCGCAAACGCG
+HWI-ST558:720JEGACXX:8:1101:1570:1981_1:N:0:_GATGGT
CCCFFFFFHHDHFHIIJGCFEGGGHIJDGIGIJJJIJJJIIJHHHFFDDD
@HWI-ST558:720JEGACXX:8:1101:7912:2032_1:N:0:_GATGGT
CGCTGGCCTCACCGCGGAAGTTCCCGAGCGCGTCGACCACCACACGTC
+HWI-ST558:720JEGACXX:8:1101:7912:2032_1:N:0:_GATGGT
@@@FFFFEDHGHGGEFHEGEBFHHGIDG@FGHAE>>AC?B?BDB@B?A

etc...

One slight development is that SOAPdenovo actually worked fine when I converted the fastq file into a fasta file, and changed the config file to: f=/home/barnes/uvba002/backups/NEXTGEN/Projects/Nesophontes/Data/LANE_8.converted.fasta

Presumably this would suggest there is a problem with the fastq file, rather than the path? The fastq file is also much bigger at 41GB, rather than 14GB for the fasta. Both seem to have the same permissions (at least, I think these are the permissions: -rw-r--r-- ).

[maubp]

... when I converted the fastq file into a fasta file...

Presumably this would suggest there is a problem with the fastq file, rather than the path? The fastq file is also much bigger at 41GB, rather than 14GB for the fasta. Both seem to have the same permissions (at least, I think these are the permissions: -rw-r--r-- ).

How did you convert FASTQ to FASTA? Given your FASTQ includes the redundant repeat of the record name on the + lines, going from FASTQ to FASTQ should halve the file size. In your case, your FASTA file sounds too small - perhaps the converter got only part way and encountered and error and stopped?

You can check this very easily by counting the reads in the FASTQ and FASTA files, and also compare the first and last records. e.g. Compare this:

grep -c "^@HWI" LANE_8_all.fastq
grep -c "^>" LANE_8_all.fasta

and

head LANE_8_all.fastq
head LANE_8_all.fasta

and:

tail LANE_8_all.fastq
tail LANE_8_all.fasta

Note: A naive grep on "^@" (lines starting with at sign) could include false positives from quality strings which can also start with the @ mark. I'm assuming all your reads start with the identifier HWI here, if not adjust as required.

[ians]

be careful with Illumina 1.8+ encoding, since '@' can be used as a quality and could certainly start a line.
If you covert manually, you'll need to include some prefix of the header.

I wonder if SOAP is getting caught on this ambiguity. I choose to think something that simple was not overlooked, however.

[mittymat]

Thanks for your responses. Yes, I agree - I was surprised that the fasta file wasn't half the size of the fastq file, however, I checked the sequences at the end for both, and they were the same:



[uvba002@cluster 17:06:18 Data]$ head LANE_8_all.fastq
@HWI-ST558:72: D0JEGACXX:8:1101:1570:1981_1:N:0:_GATGGT
CGGCGACAGCACGTCCCGTGATCAGCAAAGCAATGTTCGGCGCAAACGCG
+HWI-ST558:72: D0JEGACXX:8:1101:1570:1981_1:N:0:_GATGGT
CCCFFFFFHHDHFHIIJGCFEGGGHIJDGIGIJJJIJJJIIJHHHFFDDD
@HWI-ST558:72: D0JEGACXX:8:1101:7912:2032_1:N:0:_GATGGT
CGCTGGCCTCACCGCGGAAGTTCCCGAGCGCGTCGACCACCACACGTC
+HWI-ST558:72: D0JEGACXX:8:1101:7912:2032_1:N:0:_GATGGT
@@@FFFFEDHGHGGEFHEGEBFHHGIDG@FGHAE>>AC?B?BDB@B?A
@HWI-ST558:72: D0JEGACXX:8:1101:10882:2455_1:N:0:_GATGGT
GTCTACTACGGACTTGCGTATCCCCGGTAACCTCGCGTCGCACCGGATCA

[uvba002@cluster 17:09:51 Data]$ head LANE_8.converted.fasta
>HWI-ST558:72: D0JEGACXX:8:1101:1570:1981_1:N:0:_GATGGT
CGGCGACAGCACGTCCCGTGATCAGCAAAGCAATGTTCGGCGCAAACGCG
>HWI-ST558:72: D0JEGACXX:8:1101:7912:2032_1:N:0:_GATGGT
CGCTGGCCTCACCGCGGAAGTTCCCGAGCGCGTCGACCACCACACGTC
>HWI-ST558:72: D0JEGACXX:8:1101:10882:2455_1:N:0:_GATGGT
GTCTACTACGGACTTGCGTATCCCCGGTAACCTCGCGTCGCACCGGATCA
>HWI-ST558:720JEGACXX:8:1101:11553:2298_1:N:0:_GATGGT
CCGGTTTCGAGACCGACTTCGATGAACAATACCCGGGCGCGGTCGATG
>HWI-ST558:720JEGACXX:8:1101:8179:2666_1:N:0:_GATGGT
GAGTTGAGCACCTGCGAGCCCGCGGCAACCCGGCCCGAGTACACCCGTAC

[uvba002@cluster 17:09:55 Data]$ tail LANE_8_all.fastq
+HWI-ST558:72: D0JEGACXX:8:2307:20845:200790_1:Y:0:
===A222@C=CCAA2+2+++2+2:??A
@HWI-ST558:72: D0JEGACXX:8:2307:21099:200799_1:N:0:
CAACGTGGATTTGGGTGGGTTGGGGCAGGGGGTGGTGGTGGGG
+HWI-ST558:72: D0JEGACXX:8:2307:21099:200799_1:N:0:
1:;=4=22+22232222)21:CD10)00)(0-@A,,;?,;???
@HWI-ST558:72: D0JEGACXX:8:2307:21334:200758_1:Y:0:
AGGATTTTTTTTTGTGGTTTGGG
+HWI-ST558:72: D0JEGACXX:8:2307:21334:200758_1:Y:0:
===A?<2?C)<?)111?::=A

[uvba002@cluster 17:10:26 Data]$ tail LANE_8.converted.fasta
>HWI-ST558:720JEGACXX:8:2307:20994:200776_1:Y:0:
GGGAGGAGGTTGGGTGGGTTGG
>HWI-ST558:720JEGACXX:8:2307:20752:200777_1:N:0:
TCCATGGGGTTGTGGGGGTG
>HWI-ST558:720JEGACXX:8:2307:20845:200790_1:Y:0:
TCTGGTTGTGGTGGTTGAGTTCGTGGG
>HWI-ST558:720JEGACXX:8:2307:21099:200799_1:N:0:
CAACGTGGATTTGGGTGGGTTGGGGCAGGGGGTGGTGGTGGGG
>HWI-ST558:720JEGACXX:8:2307:21334:200758_1:Y:0:
AGGATTTTTTTTTGTGGTTTGGG

However, I notice that these numbers aren't the same:

[uvba002@cluster 16:51:36 Data]$ grep -c "^@HWI" LANE_8_all.fastq
217491778

[uvba002@cluster 17:05:04 Data]$ grep -c "^>" LANE_8.converted.fasta
147978064

I used the fastx toolkit script to generate the fasta file from the fastq one though... any suggestions how to find the problem? (Except for eyeballing - the file is pretty big, and I can't program at all unfortunately, something I need to remedy!)
I guess if there's something strange going on in the middle of the file, perhaps this could explain the problem with SOAPdenovo.

[ians]

convert it yourself !

grep -A 1 "^@HWI" LANE_8_all.fastq | grep -v "\-\-" | tr '@' '>' >LANE_8_all.fna

[maubp]

It does look like you may have found a bug in the fastx toolkit. The EMBOSS tool seqret, BioPerl, Biopython etc FASTQ parsers http://dx.doi.org/10.1093/nar/gkp1137 should all be robust and catch mangled FASTQ, hopefully with an error message that will tell you more. An independent conversion like that should be informative.

My guess is the FASTQ file got corrupted somewhere in the middle, perhaps a disk or network error?

[mittymat]

Right, I've been doing some investigations, and yes it looks like the file has become corrupted somewhere. I created this fastq file out of several files, by concatenating them together. However I now realise that you can't just concatenate two gzipped files together (this probably is obvious, apologies, I did google to see if you could - but I believe this is an example of not believing everything you read on the internet). Anyway, so last night I unzipped the original files, and then concatenated these into one fastq file, and tried to run SOAPdenovo on this again. Usefully, this time SOAP gave an error which indicated where there was a problem with a particular line in the file - it claimed that the quality info was incorrect on one line. When I went and looked at the line, there was a massive line of ^@^@^@^@^@ symbols - but with white lettering on black background (I haven't seen this before - it doesn't just seem like normal text?). So, something weird has happened, and I guess I need to go back to the beginning and work through the quality and adapter trimming process, to try and get a file which isn't messed up.

One thing, do you know if there is a way to search for the ^@^@ motifs if it isn't just standard text? Or if you can simply tell me what it is, that would also be very helpful! I've tried searching for the actual text in the fastq file, but it isn't found...?

As I said, I'm completely new to this, so I'm sorry if my questions seem stupid! Thank you very much for helping!

[maubp]

Right, I've been doing some investigations, and yes it looks like the file has become corrupted somewhere.

Glad that's sorted.

The original SOAP denovo message could have been clearer.

It does seem you've found a bug in the fastx toolkit (silently ignoring the corrupted FASTQ and in the process missing many of the reads).

I created this fastq file out of several files, by concatenating them together. However I now realise that you can't just concatenate two gzipped files together (this probably is obvious, apologies, I did google to see if you could - but I believe this is an example of not believing everything you read on the internet).

In principle you should be able to concatenate two gzipped files together - but clearly something went wrong somewhere.

[mgbfx9]

Hi,

I was using Soapdenovo2 for de novo assembly of my MiSeq data for a while. But as my contig values were pretty low, I have been suggested to do preprocessing of my data.
I used Trimmomatic tool, Trimmomatic-0.33 to preprocess my data which removed my adapter sequences and the low-quality reads and generated fastq files.
Now when I took this processed fastq files to assemble with Soapdenovo2, the assembler could not open my file. It has the same header as the original. Output says that "Import reads from file...... Cannot open. Now exiting the system". Exactly the same as the earlier post has said.

What could be wrong? Was there a bug in my trimmomatic tool or were my files corrupted?

Any help would be appreciated, thanks!

[GenoMax]

Are your trimmed files still in gzip format? Did you concatenate more than one of these files together?

[mgbfx9]

Hi GenoMax,

I just responded to you in the other site with different question.

No, my files were converted prior to trimming to FastQ files, and the Trimmomatic tool generated fastq files too. So they are not in gzip format.
I did not concatenate any of these files together, although I thought about that. But earlier when I ran with the unprocessed files, I didn't concatenate them, but they were in gzip files.

[kimyam]

Hello, I was encountered the same problem with SOAPdenovo2.

"Cannot open <mydata.fastq>. Now exit to system."

I checked the file names and path, converted the fastq files to fasta with 'grep' command.
However this error did not be resolved.

Then, I could assemble the fastq data with Velvet 1.2.10.
So it was not the problem about file conversion errors.

-- for our information