[ploverso-pgdx]

The code in savR doesn't seem to support version 6 from what I can see (https://github.com/Bioconductor-mirr...savR-methods.R) although it seems to support v5.

FWIW, here's the format for V6 (very similar to v5 as described in https://tracker.tgac.ac.uk/browse/MISO-138):


byte 0: file version number (6)
byte 1: length of each record
byte 2: quality score binning (byte flag representing if binning was on), if (byte 2 == 1) // quality score binning on
byte 3: number of quality score bins, B
bytes 4 - (4+B-1): lower boundary of quality score bins
bytes (4+B) - (4+2*B-1): upper boundary of quality score bins
bytes (4+2*B) - (4+3*B-1): remapped scores of quality score bins (QRemapped)
The remaining bytes are for the records, with each record in this format:
2 bytes: lane number (uint16)
2 bytes: tile number (uint16)
2 bytes: cycle number (uint16)
4 x B bytes: number of clusters assigned score (uint32) (QRemapped)


For the MiniSeq, B = 7.

SavR definitely supports version 6- I'm the one that wrote the code to support it. Here's the commit that added support: https://github.com/bcalder/savR/comm...506336250cec1c

[dpryan]

Here's the original github repository that people can make PRs against and issues for.

[alito]

The code in savR doesn't seem to support version 6 from what I can see (https://github.com/Bioconductor-mirr...savR-methods.R) although it seems to support v5.

FWIW, here's the format for V6 (very similar to v5 as described in https://tracker.tgac.ac.uk/browse/MISO-138):


byte 0: file version number (6)
byte 1: length of each record
byte 2: quality score binning (byte flag representing if binning was on), if (byte 2 == 1) // quality score binning on
byte 3: number of quality score bins, B
bytes 4 - (4+B-1): lower boundary of quality score bins
bytes (4+B) - (4+2*B-1): upper boundary of quality score bins
bytes (4+2*B) - (4+3*B-1): remapped scores of quality score bins (QRemapped)
The remaining bytes are for the records, with each record in this format:
2 bytes: lane number (uint16)
2 bytes: tile number (uint16)
2 bytes: cycle number (uint16)
4 x B bytes: number of clusters assigned score (uint32) (QRemapped)


For the MiniSeq, B = 7.

[ploverso-pgdx]

Illuminate is not up-to-date, it only supports up through v5 of interOp files. There is a package called savR in Bioconductor which supports up through v6, for RTA versions 2.7 and up (for the HiSeq 4k, etc).

https://www.bioconductor.org/packages/release/bioc/html/savR.html

[ophir]

A new tool for the job is illumate - https://bitbucket.org/invitae/illuminate
Installation is pretty straightforward, was able to get it up and running in a few minutes.

[earonesty]

Module Bio::IlluminaSAV at CPAN is available for these files. Version 1.0, release 7013 works well (prev release was missing files). You can install using:

sudo cpanm -i Bio::IlluminaSAV

Lastest version passes all the relevant CPAN test reports for MiSEQ and GAII data. the failed reports are on test systems that can't handle the volume of data. HiSeq passes as well, but is too large to upload to CPAN at all.

[mchen1]

InterOp parsers in R and perl

Hi,

I'm from Illumina and over the past few months, I've built a package in R and some scripts in perl to accurately parse the binary InterOp files.

These are unsupported, which means tech support will not be able to help you with them, but we've tested these internally and I've received approval from my manager to share them with those that ask. So please PM me if these scripts would be useful for you.

Cheers,
mchen1

[earonesty]

Module Bio::IlluminaSAV at CPAN is available for these files. Version 1.0, release 7013 works well (prev release was missing files). You can install using:

sudo cpanm -i Bio::IlluminaSAV

[ECO]

I was so annoyed with not being able to do get cluster density easily (Illumina, if you're reading this, you're retarded ), that I wrote the below to get it:

class TileMetrics:
    def __init__(self,filename):
        self.f = filename 
        import pandas as pd
        from bitstring import BitString
               
        a = BitString(bytes=open(self.f, 'rb').read())
        self.filever = a.read('uintle:8')  # version number == "2"
        self.recordlen = a.read('uintle:8')  # length of each record == 10 (for TileMetrics)
        a.pos = 16  # skip the above bytes which are invariant for this

        #setup data
        self.data = {'lane' : [], 'tile' : [], 'met' : [], 'value' : []}
        
        #read records bytewise per specs in technote_rta_theory_operations.pdf from ILMN
        for i in range(0,((a.len - 16) / (self.recordlen * 8 ))):  # 80 == record length in bits
            self.data['lane'].append(a.read('uintle:16'))  #lane number
            self.data['tile'].append(a.read('uintle:16'))  #tile number
            self.data['met'].append(a.read('uintle:16'))  #metric code
            self. data['value'].append(a.read('floatle:32')) #metric value
        
        #make it fuzzy
        self.df = pd.DataFrame(self.data)
        
        #get some stuff
        self.cdens = self.df[self.df.met == 100].reset_index(drop = 1)
        self.pfcdens = self.df[self.df.met == 101].reset_index(drop = 1)
        

if __name__ == '__main__':
    tm = TileMetrics('TileMetricsOut.bin')
    print '###############'    
    print 'filename %s' % tm.f
    print '###############'
    print 'average clusterdensity == %.2f' % tm.cdens.value.mean()  
    print 'average perc pf clusters == %.2f' %  (100 * tm.pfcdens.value.mean() / tm.cdens.value.mean())

Easily extensible to other metrics files, I'm working on it (slowly) Metrix looks awesome but is beyond my skills to efficiently implement.

[iamh2o]

Thanks Bernd-

I'll reply with our experience, and contribute back to the project where I can.

j

[Rhizosis]

Hi everybody.

As promised. The initial release of Metrix.
For feedback, idea's, bug reports and future communication please see the Metrix thread.

I hope this helps.

Bernd

[behavin]

This would be quite helpful for us as well, I'd love to be able to parse all the metrics without resorting to SAV.

[Rhizosis]

I have talked to my supervisor and i've been given the green light to release this software under the GPLv3 license. I need to put it through some beta tests first, but it shouldn't be too long.

In about a week or two I will create a separate thread for this in the bioinformatics section for feedback, support and bug reports.

[ECO]

I am also interested in this. It blows my mind that ILMN (or someone) hasn't released this code yet.

[Rhizosis]

I have managed to build a fully functional server client model Illumina metrics parser which interprets data written by HiSeq's and MiSeq's in real time.

However, I dont know yet if I am allowed to release it to the public. Need to talk to my supervisor about this first.

Most likely it will be integrated with the GNomEx LIMS system in Q1 2013.
Let me know if you need pointers.