[GenoMax]

It was ultimately determined that "the operation was successful but the patient had died".

A replacement machine is predictably behaving significantly better.

Our upgraded MiSeq (which had passed optical tests), required a service call because a phiX run (done after the fact due to quality issues with samples) did not pass illumina quality standards.

[mcnelson.phd]

These are some pictures showing the quality of our first run (2x250 bp and 2 indices) where we tried to do amplicon sequencing. The same library had been run with the 300 kit successfully giving nice reads and good Q values.
I was told there was not enough diversity. We will give it a try and spike in more phiX. But I am not sure this is the only problem. The constructs are based on the published TrueSeq sequences. A second run went even worse... Any feedback is appreciated.

Illumina really dropped the ball on this and they'll admit it. The new MCS software freaks out if the phasing/pre-phasing metrics start to move out of what they consider an acceptable range, and causes per-base quality estimates to go to hell. The problem is, the samples really aren't phasing/pre-phasing, it's just that the base distribution is so skewed that the software can't correctly calculate the real values. We've talked with a number of Illumina people about this, and basically there's no easy fix except to spike in massive amounts of phiX depending on how low the base diversity is. I know for 16S runs we've had problems even when using 75% phiX, which means the MiSeq is no longer cost-effective for the number of high-quality reads being returned.

Rest assure, you're not alone, and there are a lot of very disgruntled researchers right now.

[pmiguel]

Illumina really dropped the ball on this and they'll admit it. The new MCS software freaks out if the phasing/pre-phasing metrics start to move out of what they consider an acceptable range, and causes per-base quality estimates to go to hell. The problem is, the samples really aren't phasing/pre-phasing, it's just that the base distribution is so skewed that the software can't correctly calculate the real values.

Do you mean this is particularly a problem for 250 base reads? The low diversity issue has been among Illumina's major weaknesses since the beginning.

I have a document from somewhere that describes how to "hard code" matrix and phasing for the MiSeq running RTA 1.14. Should work for v2 as well, right? Alas, the document says it is not a complete solution to diversity issues, but can improve results somewhat.

--
Phillip

[mcnelson.phd]

Do you mean this is particularly a problem for 250 base reads? The low diversity issue has been among Illumina's major weaknesses since the beginning.

I have a document from somewhere that describes how to "hard code" matrix and phasing for the MiSeq running RTA 1.14. Should work for v2 as well, right? Alas, the document says it is not a complete solution to diversity issues, but can improve results somewhat.

--
Phillip

From what I've been told, and a lot of this has been "we're telling you this but we didn't officially tell you this", the issue affects the newest version of RTA that goes along with the software/hardware upgrades and v2 chemistry kits. Apparently during their beta-testing, Illumina only focused on genome libraries, so when the issues with the low-diversity amplicons came up they were caught with their pants down. I was speaking with our FAS about different potential "hacks" we could try to keep quality up, but she said that they've tried those in house and they still don't resolve the quality drop-offs (or else the problem would have been solved with a software update by now).

Hardcoding new values might help a little, so feel free to try it and let us all know how it goes, but from what I've been hearing from Illumina is that it's not a simple fix.

Added: This problem affects both the new 300 cycle and the 500 cycle kits. We haven't tried a 50 cycle but if the problem is in RTA then I could see it still popping up, although with such a short run time it might not have a measurable effect on overall quality like it does on the longer cycle kits. The low-diversity has always been a thorn in Illumina's side, but post-upgrade it's turned into a gaping wound now that people are ditching 454s for MiSeq for 16S.

[Vinz]

Thanks, Mcnelson for your comments. I definitely can confirm that the issue really popped up for us with the upgrade. We have a mix of about 30 amplicons. So diversity is not very low, but obviously not to compare to genomic libraries.
Meanwhile we have tried the hardcoded phasing/matrix. After the upgrade this is kind of supported by the software. So the instructions to do so changed. We have done one run so far which looked considerably better. On the reverse read quality drops at the last 50 cycles but is still usable.
We will try to improve diversity anyway to avoid this temporary solution.


Instructions for hardcoded phasing/matrix with upgraded MiSeqs:
Choose a good run (phiX).
Run folder\Data\Intensities\Basecalls\Matrix\s_1_1_matrix.txt
Run folder\Data\Intensities\Basecalls\Phasing\s_1_1_phasing.txt
Copy these 2 files and rename them as hardcodedmatrix.txt and hardcodedphasing.txt respectively.
Place them in C:\Illumina\RTA

[nickloman]

I know for 16S runs we've had problems even when using 75% phiX, which means the MiSeq is no longer cost-effective for the number of high-quality reads being returned.

I'm thinking that if the run is 75% PhiX and there are still problems, then the issue is not low-diversity but something else ...

[Vinz]

I'm thinking that if the run is 75% PhiX and there are still problems, then the issue is not low-diversity but something else ...

Nick, do I understand you correctly that you managed to do amplicon sequencing with v2 250 bp on an upgraded MiSeq? I would be eager to learn the tricks!

[mcnelson.phd]

I'm thinking that if the run is 75% PhiX and there are still problems, then the issue is not low-diversity but something else ...

You're right and you're wrong.

Depending on the amplicon libraries, they can be so heavily skewed at certain base positions that even with 75% phiX the software will still run into the phasing/pre-phasing estimation error and from that point on it can't recover. This easily occurs when you have a fusion primer design where you sequence the template primer early in the read.

The other possible problem is that we've had too many hands involved lately and errors are being made in dilutions or maths. My suspicion is that because of the software issue, any small errors in library prep can easily kill a run because all it takes is a few cycles with very skewed base distribution and that read is lost, especially if they happen early.

[mcnelson.phd]

Thanks, Mcnelson for your comments. I definitely can confirm that the issue really popped up for us with the upgrade. We have a mix of about 30 amplicons. So diversity is not very low, but obviously not to compare to genomic libraries.
Meanwhile we have tried the hardcoded phasing/matrix. After the upgrade this is kind of supported by the software. So the instructions to do so changed. We have done one run so far which looked considerably better. On the reverse read quality drops at the last 50 cycles but is still usable.
We will try to improve diversity anyway to avoid this temporary solution.


Instructions for hardcoded phasing/matrix with upgraded MiSeqs:
Choose a good run (phiX).
Run folder\Data\Intensities\Basecalls\Matrix\s_1_1_matrix.txt
Run folder\Data\Intensities\Basecalls\Phasing\s_1_1_phasing.txt
Copy these 2 files and rename them as hardcodedmatrix.txt and hardcodedphasing.txt respectively.
Place them in C:\Illumina\RTA

Vinz, did you use your phiX validation run after the upgrade or did you go back to a V1 chemistry run? I talked with my FAS and she said she's surprised that you got it to work out and help improve data quality since their internal testing didn't produce promising results (but who knows what they're testing with?).

[Vinz]

Vinz, did you use your phiX validation run after the upgrade or did you go back to a V1 chemistry run? I talked with my FAS and she said she's surprised that you got it to work out and help improve data quality since their internal testing didn't produce promising results (but who knows what they're testing with?).

We used a phiX run with V2 chemistry which looked really good. Our amplicon data has some diversity due to the number of amplicons. Base composition is usually between 10 and 50% per base. Maybe we were just lucky...

[GenoMax]

This seems to be a recurring theme with 2 x 250 bp runs. The drops are not drastic but they are there.

On the reverse read quality drops at the last 50 cycles but is still usable.

The unofficial word seems to be that this solution is so new that it is technically unsupported.

Thanks from providing the instructions. I had heard that the instrument will automatically decide (after looking at the first 4 cycles) if it needs to use the hard coded matrix and there is no way (yet) to override that. But your instructions seem to indicate otherwise.

In our case we have some 16S libraries that have a diverse 4-bp tag at the beginning of the read so if the instrument was to decide on whether to use the "good" matrix then it will not work.

Instructions for hardcoded phasing/matrix with upgraded MiSeqs:
Choose a good run (phiX).
Run folder\Data\Intensities\Basecalls\Matrix\s_1_1_matrix.txt
Run folder\Data\Intensities\Basecalls\Phasing\s_1_1_phasing.txt
Copy these 2 files and rename them as hardcodedmatrix.txt and hardcodedphasing.txt respectively.
Place them in C:\Illumina\RTA

[mcnelson.phd]

In our case we have some 16S libraries that have a diverse 4-bp tag at the beginning of the read so if the instrument was to decide on whether to use the "good" matrix then it will not work.

We've been using that same format, with the 4 N bases at the beginning, for some runs and have had some success. Success in this case though is ~2 millions usable reads that aren't phiX before you even get into quality trimming and processing. So far we've been able to squeeze out enough data to make it work, and it's more cost effective than the 454, but obviously this is a non-optimal solution for an instrument that's stated to give ~15 million read pairs.

[nickloman]

Nick, do I understand you correctly that you managed to do amplicon sequencing with v2 250 bp on an upgraded MiSeq? I would be eager to learn the tricks!

No, not yet. Hopefully we will be up and running later this week or next week and I will report back our findings.

[nickloman]

We've been using that same format, with the 4 N bases at the beginning, for some runs and have had some success. Success in this case though is ~2 millions usable reads that aren't phiX before you even get into quality trimming and processing. So far we've been able to squeeze out enough data to make it work, and it's more cost effective than the 454, but obviously this is a non-optimal solution for an instrument that's stated to give ~15 million read pairs.

Have you considered multiplexing a few primers of different lengths such that each starts with a different base?

[mcnelson.phd]

Have you considered multiplexing a few primers of different lengths such that each starts with a different base?

That's been another thought, but it's an issue of how much $ does one need to spend on primers just to get this to work? Ideally Illumina comes out with a software fix within the next month that obviates a lot of these hacks.