v4 communcation patch causes problem.

[pmiguel]

FYI:

Watch your instrument closely if you have the most recent v4 "communications issue" patch installed -- it no longer will error out when communication between the console and the camera hangs. In this circumstance your instrument will hang, collecting no data, but it will not warn you.

If you do notice you are in this state use the same work around as normal: just restart ICS (pausing your run(s), if possible, beforehand), or in the most extreme case, reboot the windows blade. Then the instrument re-establishes communications and the run continues merrily on its way once ICS restarts and you unpause each run.

This happened to us around noon yesterday. But because of the lack of warning the instrument collected no data for 16 hours before I noticed the issue.

Applied Biosystems: masking a problem is not the same as solving it!

--
Phillip

[bioits]

You meant the latest v4.0.4? I haven't seen any issue since it was patched about a month ago.
After you restarted ICS, were you able to get your data back? How did you do if not? Rerun those primers w/o data?

[VanessaS]

Version 4.03 did have some bugginess. Fixed one problem then had issues with the imager hanging during focal range detection. v4.04 seems to have fixed the issues that we had. So far.

[pmiguel]

Hi Vanessa,
We are running ICS 4.0.4FC3.

We did not lose anything but time, as it turned out. I exited ICS, started ICS again, set the run progress points back to "image prep" and the runs continued without incident.

So, as issues go, this was not a particularly bad one. More an irritation. To avoid it in the future we are keeping an eye on the instrument to make sure it has not entered a perpetual wait. (Prior to this patch, ICS would time out fairly rapidly and enter a "pausedManual" state that was obvious from SETS.)

I am not objecting to its new "never give up" attitude, I just want to know when the run is no longer progressing so I can intervene to get it back on track.

--
Phillip

[VanessaS]

Thankfully I've not had the hang-and-chill-forever-with-no-error-thing since before the 4.03 patch.
I have noticed that our two instruments behave differently. One never showed certain errors, the other always did, and vice versa depending on the patch and the price of tea. Most quirky and temperamental, my gals are. I find they behave better in general if I restart ICS before every set of runs, when possible.
I wonder if reinstalling the patch would prove to be beneficial?
I have an angel of a FSE who is very responsive to coming out and giving them the proverbial whack on the side and reporting errors, working with my equally angelic FAS to help get things ironed out. I did notice on one of the machines, after the latest patch, that on some random ligations, data was not collected, or not reporting. This was new but the bioI FAS logged in and fixed. I wish I knew how to do that little bit of wizardry!

ETA I now set my alarm on my phone for every 4 waking hours to log in and check on the gals. First thing I do when I wake up and last thing I do before heading to bed. Ounce of prevention and all that jazz.

[pmiguel]

I did notice on one of the machines, after the latest patch, that on some random ligations, data was not collected, or not reporting. This was new but the bioI FAS logged in and fixed. I wish I knew how to do that little bit of wizardry!

Yes, we have seen that. Image data is collected, but no processing occurs for a ligation. We tell our bioinformatics contact at ABI and he initiates the processing for that ligation. Then everything is okay.

--
Phillip

[selupe]

We had the same issue with one of 3 instruments running 4.0.4. The mouse is frozen right now during the F5 tag of a paired end run, but I can control the computer by VNC from another terminal.

[pmiguel]

We had the same issue with one of 3 instruments running 4.0.4. The mouse is frozen right now during the F5 tag of a paired end run, but I can control the computer by VNC from another terminal.

That sounds like a windows issue, not what we saw.
The issue we saw was that low level communications problem (possibly preventing control of the camera?). No noticeable change there with respect to earlier patch versions. Rather, the way the communications problem was being handled changed.

Previously when the communications problem cropped up, it would cause a timeout that would lead to a paused run. Now, instead, there appears to be no time out. Rather the console continues to attempt to re-establish communication with the sub-system -- possibly forever? (I finally noticed something was awry 16 hours after the "failure to communicate" event began.)

Arguably the new behavior is better -- but since we had absolutely no warning of the new behavior, it caused us a problem. Not a big problem -- ultimately we lost 16 hours of machine time, but no apparent loss of samples or data.

I will add that this situation strikes me as Applied Biosystems-typical. We have had similar (rare) failures on our 3730XL's. Where a run has crashed, but the instrument placidly displays a "green light". Some issue with error propagation I guess.

I think the easiest way to monitor the situation are the "Ganglia:: Cluster Reports". As long as these show the expected load spikes every few hours, everything is okay.

--
Phillip