Illumina FastQ Question

[shandley]

Hi,

New to working with Illumina data. I am running into some inconsistency issues on what programs will recognize my file. For example, bowtie works just fine with the file, however, Galaxy, the FastX Toolkit and Stampy all give me error messages in regards to line 4 containing the quality information. I have read lots of posts about converting to Sanger, the variety of Illumina generated FastQ formats, but have yet to completely identify the issue. Some of the data is below.

Any thoughts or recommendations?

@HWUSI-EAS-100R_0003:8:1:1034:19859#NCCCCC/1
GGTATGTTAACATTCANTGAGCTATACACTTAAGATTTGTGCACTTTATCATAGTAAGTTATTTGTCAGTTTGAA
+
ddadcdeeeedddd^bBa]\ZX[^Xdddaddca^a\dddab^J]`[Y^`^^cLa^c`a`YJWOSSRb\_BBBBBBB
@HWUSI-EAS-100R_0003:8:1:1034:9367#NCCCTC/1
CACTGAATGACATGGGACTGTTTGGACAAAACGTGCTATACCTCTACCTCGGGAGGGCCGGTTACACCATACATG
+
eceeeeedefcdfcfdcc^a`c^_`_Y^^\M^][^baYa\a^K^^T[]Y^BBBBBBBBBBBBBBBBBBBBBBBBBB
@HWUSI-EAS-100R_0003:8:1:1035:9515#NCTATG/1
CGCCGTTTCCCAGTAGGTCTCCTAGAACACTATTTCAATGATGAACAAGGCGAGATCGAGCTCATTGAGACCTCG
+
feefffffdfffedcdcbe\bdaa^T^^YaT^_\Sa^\^^YY^^^BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB

EDIT: Just noticed that there are 75 bp and 76 ascii quality scores. Am I missing something here?

Many thanks,

Scott

[fkrueger]

Hi Scott,

regarding FastQ files it is normally a good idea to run FastQC on the data, it might tell you a lot about the sequencing data.


I guess most programs will complain about the space in the sequence line, for which there is a quality value though. If you delete this space then FastQC wouldn't complain anymore.

Apparently your data is in Illimunia 1.3+ encoding (Phred64 scale), however you need to find out why the sequence line and the quality value lanes are of different lengths, and why there is a space in the sequence in the first place...

[shandley]

Thanks fkrueger,

FastQC recognizes and works with the data as Illumina 1.3+ format. When I run bowtie I turn the --phred64-quals option on.

The Galaxy complaint is specifically that there are 76 quality scores and only 75 nucleotides ... I guess this is where I will direct my attention and try to sort it all out. If anyone has any suggestions or has run into a similar problem please let me know!

Scott

[maubp]

I guess most programs will complain about the space in the sequence line, ...

There is no space in the sequence - it is just the forum rendering it badly, trying to help it line wrap. If the OP had used the [ code ] tag I think it would have looked like this:

@HWUSI-EAS-100R_0003:8:1:1034:19859#NCCCCC/1
GGTATGTTAACATTCANTGAGCTATACACTTAAGATTTGTGCACTTTATCATAGTAAGTTATTTGTCAGTTTGAA
+
ddadcdeeeedddd^bBa]\ZX[^Xdddaddca^a\dddab^J]`[Y^`^^cLa^c`a`YJWOSSRb\_BBBBBBB
@HWUSI-EAS-100R_0003:8:1:1034:9367#NCCCTC/1
CACTGAATGACATGGGACTGTTTGGACAAAACGTGCTATACCTCTACCTCGGGAGGGCCGGTTACACCATACATG
+
eceeeeedefcdfcfdcc^a`c^_`_Y^^\M^][^baYa\a^K^^T[]Y^BBBBBBBBBBBBBBBBBBBBBBBBBB
@HWUSI-EAS-100R_0003:8:1:1035:9515#NCTATG/1
CGCCGTTTCCCAGTAGGTCTCCTAGAACACTATTTCAATGATGAACAAGGCGAGATCGAGCTCATTGAGACCTCG
+
feefffffdfffedcdcbe\bdaa^T^^YaT^_\Sa^\^^YY^^^BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB

This is not valid FASTQ since the quality strings are one character longer than the sequences. Tools which ignore the quality strings may accept this as input anyway if they don't check the length.

[shandley]

Thanks maubp,

The data was provided by a collaborating lab. I have written them to ask if they have any insight as to why there is an extra character in the quality string. I am tempted to just eliminate the last value. The quality is low here anyways. But this does assume that the extra character was entered at the end of the string instead of the beginning or middle.

Thanks again

[maubp]

The data was provided by a collaborating lab. I have written them to ask if they have any insight as to why there is an extra character in the quality string.

Probably a bug in their filtering script or something

I am tempted to just eliminate the last value. The quality is low here anyways. But this does assume that the extra character was entered at the end of the string instead of the beginning or middle.

Yeah - hard to say where the error is.

P.S. If you hadn't seen this before, I think the trailing BBBB... qualities are Illumina's Read Segment Quality Control Indicator, and can be stripped off, see:

http://seqanswers.com/forums/showpost.php?p=17491&postcount=3

http://news.open-bio.org/news/2010/04/illumina-q2-trim-fastq/

[shandley]

Ahhh ... I thought that trailing BBBBB were suspicious. Based on glancing at the overall run data in FastQC the quality drops off dramatically in the last 10 nt's or so. I think I will just eliminate the trailing quality score and run with it.

Thanks for all of the rapid an insightful responses.

[bioinfosm]

There is always the trade-off of trimming and losing information, or including and adding noise. Is it really an accepted convention now to trim all trailing B's from Illumina data? Or the aligner and variant caller using Quality values takes care of odd low quality bases..

[shandley]

I am only going to trim the final B to alleviate the conflict I am having (at least until the sequencing center gets back with me and explains the extra quality value). I would actually prefer to include as much data as possible and tune out the low quality stuff using assemblers. My thought on this is that 'low-quality' data actually implies there is some accurate data there. I hate to loose any of the precious information, particularly with illumina short-read data.

[maubp]

I think people running out of memory with Illumina assemblies tend to be much more ruthless with their quality trimming. Read errors push up the unique kmer count, and therefore the memory requirements for something like velvet. As a result, I think the optimal trimming strategy will vary according to the amount and nature of your data, and your assembler and computer resources.

Or in other words, YMMV (your mileage may vary)

[bioinfosm]

Certainly. Depending on application one can get deep coverage and then trim low Q to end up with really good quality data. Though have not seen anything to point how helpful that is in the end