[NikTuzov]

Yes, although in this case you could use PHRED quality 0 (and I recall some tools may use the upper bound 93 as a special value).

Thanks. I take it the encoding of missing values depends on the sequencer and it cannot be changed downstream. Therefore, I don't see how some downstream tool can use 93 as a missing value when it is not seen as such by the sequencing instrument.

[maubp]

Hi:

I have a question about the so-called Sanger format where Q can be in [0;93] represented by [!-~]. Since this is the set of all visible ASCII characters, then it looks like there is no symbol reserved for a missing value. Does it mean that specifying missing values for the quality of individual base pairs is impossible? If yes, why?

Yes, although in this case you could use PHRED quality 0 (and I recall some tools may use the upper bound 93 as a special value).

Why? None of the early technologies needed a missing value quality score.

[NikTuzov]

Hi:

I have a question about the so-called Sanger format where Q can be in [0;93] represented by [!-~]. Since this is the set of all visible ASCII characters, then it looks like there is no symbol reserved for a missing value. Does it mean that specifying missing values for the quality of individual base pairs is impossible? If yes, why?

[maubp]

As discussed earlier in this thread, you can have lone B qualities (PHRED 2) in the quality string, and a trailing block of B markers as well. The second example here specifically copes this:

http://news.open-bio.org/news/2010/04/illumina-q2-trim-fastq/

[SamH]

Isolated B's

Hi, hopefully I can revive this old thread for a little. I just got a big dataset from a HiSeq2000 machine at Berkeley, I'm not sure which version of the Illumina pipeline was used, but I do see single "B" qualities in some reads
i.e.


@HS1_0077:5:1101:1205:2082#0/1
NCCCCAAAGCATGATGTTTCCACCCCCATGCTNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNCAGACCGATATCGTATGCCGTCTTCCGC
+HS1_0077:5:1101:1205:2082#0/1
BTSTTV[VYYc_ac_cccccccc[YUYV_cBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
@HS1_0077:5:1101:1231:2094#0/1
NTGTGGTATATATGCATGTAGTTACTTGGCCANNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNCTCTCCACGATCTCCACACACACCCTCT
+HS1_0077:5:1101:1231:2094#0/1
BUPNUSSUUUcccac_c_ccccc_ccc_caBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB


SamH

[juan]

[QUOTE=maubp;25709]Solexa's negative quality scores only went down to -5, so something else is going on.

I figured it out. 10 is the ASCII code for newline. bug in code not bizarre quality score.

[clive]

Do they still emit varying quality values for N bases?

That always confused me. Most were 4 I think, but we'd occasionally see N with quality all the way up to 10. I can only assume they change bases to N at some stage, but don't do anything with the Q value. It seemed broken at the time anyway, but maybe it's a bit saner now.

If I can understand what they have done here? -- they take low scoring bases and convert them to N (rather than calling the highest signal with a low score) ? -- when you align these reads are the N's counted as errors ?? or ignored ??

[maubp]

Solexa's negative quality scores only went down to -5, so something else is going on.

Could you post a couple of reads with these funny quality scores? Wrap it in [ code ] and [ /code ] tags for display in the forum.

[juan]

Quality score of -54(10)

Before mapping and before subtracting 64, I checked the distribution of quality scores for my reads (PIPELINE 1.6). I noticed what everyone mentioned here (quality scores starting at 66 - 64 = 2).

However, I also noticed thousands of quality scores of 10 - 64 = -54. I thought negative quality scores were "phased out" according to the Wiki? What are these? More importantly, do they say anything about run quality? One end of my paired-end run has more -54 quality bases in the second end for every lane, what does that mean?

Second question, do any of the current mapping programs (Bowtie, BWA, BFAST, SOAP, etc) automatically do end-clipping of "B" quality bases at ends of reads? I am guessing that the -54 scores are converted to zero.

Cheers,
Juan

[Torst]

Do they still emit varying quality values for N bases? That always confused me. Most were 4 I think, but we'd occasionally see N with quality all the way up to 10. I can only assume they change bases to N at some stage, but don't do anything with the Q value. It seemed broken at the time anyway, but maybe it's a bit saner now.

I agree with your experience. I never understood what an "N" of quality "10" actually meant; Q10 means probability of error is 10% so does that mean there is a 90% chance it is not an "N" ? :-)

I just wrote a quick Perl script to check how N is being qualitied on a recent Pipeline 1.6 for the first 2M reads of a random fastq file from the run (QVALUE => FREQUENCY):

'6' => 7,
'11' => 22,
'7' => 57,
'9' => 80,
'12' => 18,
'2' => 281517,
'15' => 1,
'14' => 5,
'8' => 62,
'4' => 51799,
'13' => 3,
'10' => 23,
'5' => 72

As you can see, most are Q02, which is "B" and is part of the 'rejected section' of the read, so they can be ignored. Most of true Ns are Q4 ("D") as they were in your experience, however there are still smatterings of Ns with qualities all the way up to Q15 !

*sigh*

[jkbonfield]

Do they still emit varying quality values for N bases?

That always confused me. Most were 4 I think, but we'd occasionally see N with quality all the way up to 10. I can only assume they change bases to N at some stage, but don't do anything with the Q value. It seemed broken at the time anyway, but maybe it's a bit saner now.

[maubp]

Correct - for data from recent pipelines that do use the RSQC, your script will do the right thing. But if you modify it to only work if the Bs are anchored to the end of the read, it will still do the same thing for new reads (as no isolated Bs) and won't do too much damage if used on older reads (the Bs, which are awful quality 2 anyway, would have to be at the end). I always prefer to program defensively.

I quite agree about defensive programming, and have worked out a neat way to do this alternative (trim trailing B characters only). I'll update the blog post.

Thanks for your advice Torst.

Peter

[Torst]

maubp,

That makes sense. Do you agree that would mean Hena's data predates the new "Read Segment Quality Control Indicator" with the special meaning for character "B", and thus using "B trimming" as implemented in my script on it isn't a good idea. Some sort of window based average quality threshold filtering would be more sensible.

Yes, for reads from older pipelines that don't use the RSQC, your script is not useful.

However, if the FASTQ files from the latest Illumina pipeline really do only use B at the end of a read, my script is fine as is. Yes?

Correct - for data from recent pipelines that do use the RSQC, your script will do the right thing. But if you modify it to only work if the Bs are anchored to the end of the read, it will still do the same thing for new reads (as no isolated Bs) and won't do too much damage if used on older reads (the Bs, which are awful quality 2 anyway, would have to be at the end). I always prefer to program defensively.

[maubp]

Actually, the pipeline guarantees there will NOT be any isolated Bs in the quality string anymore. Quality values 0,1,2 are no longer used. B=2 is only used at the END of reads in a contiguous fashion backwards. I suspect the disallowed B as a quality value now, so you can tell the difference between older pipleline quality strings (which use B for Q2) and recent pipelines which use it for the "read quality indicator".

That makes sense.

Do you agree that would mean Hena's data predates the new "Read Segment Quality Control Indicator" with the special meaning for character "B", and thus using "B trimming" as implemented in my script on it isn't a good idea. Some sort of window based average quality threshold filtering would be more sensible.

However, if the FASTQ files from the latest Illumina pipeline really do only use B at the end of a read, my script is fine as is. Yes?

(P.S. Apologies if my recent posts were too terse, typing on a mobile device is a pain - I'm back at a full keyboard now)

[Torst]

So are any B characters in the middle simply PHRED quality 2 then? That was not clear to me.

Actually, the pipeline guarantees there will NOT be any isolated Bs in the quality string anymore. Quality values 0,1,2 are no longer used. B=2 is only used at the END of reads in a contiguous fashion backwards. I suspect the disallowed B as a quality value now, so you can tell the difference between older pipleline quality strings (which use B for Q2) and recent pipelines which use it for the "read quality indicator".