HiSeq 2500 : Rawdata Storage (permanent)

[sklages]

Hi Folks,

as an academic sequencing service provider we are obligated to store the sequencing "rawdata".

We are currently defining "rawdata" as the "standard" rundata plus CIFs (for HiSeq devices).
Amount of data is roughly 3-4TB per run. 80%-85% being CIFs.
In the past this enabled us to run OLB on lanes/runs to "optimize" basecalling in some cases.

OLB development has been dropped some time ago, RTA was developed continously.

So my questions are:

* What files/data are you (permanently) storing of HiSeq run? And why?
* Are you storing the data for (potential) re-analysis (OLB?) or just for documentation purposes?
* Anyone with experience in running OLB 1.9.x on HiSeq data created with current machine software?

I am just curious :-)

best,
Sven

[GenoMax]

We store a tar copy of the flowcell folder from the sequencer (No CIF's, with amount of sequencing we do that is impractical). In addition we store fastq files after sample demultiplexing for ready retrieval in case customers ask us to re-release their data (generally because of drive failures, on their end).

[sklages]

We store a tar copy of the flowcell folder from the sequencer (No CIF's, with amount of sequencing we do that is impractical). In addition we store fastq files after sample demultiplexing for ready retrieval in case customers ask us to re-release their data (generally because of drive failures, on their end).

Well, it becomes even impractical with our 4 machines. It works, but it is still impractical.

We store the fastq results without any time limit. That's ok ..

[GenoMax]

Are you backing the data up to tape?

With new nearline storage from google that is becoming a tempting option as a replacement for tape (especially with the announcement that netbackup will support nearline storage in v.7.7) at just 1 cent/GB/month.

[sklages]

No, we are backing up to disks, jBods.
Google is not an option as we require the data to be stored on internal servers.

[GenoMax]

Wow that is a lot of data on plain JBOD. What are you doing for ensuring reliability? Or are you storing data on external drives that are then put away in a closet (not continuously available)?

[sklages]

Wow that is a lot of data on plain JBOD. What are you doing for ensuring reliability? Or are you storing data on external drives that are then put away in a closet (not continuously available)?

That's how we do it. We keep the results data "online" without any time limit (internal) and put the raw rundata on jbods after a few months to be stored offline in a closet as backup :-)

[GenoMax]

That's how we do it. We keep the results data "online" without any time limit (internal) and put the raw rundata on jbods after a few months to be stored offline in a closet as backup :-)

Ah. This is not standard JBOD (something that always stays online).

Hopefully you have an assistant who can spin the disks up periodically to make sure they are working

I have had bad experiences with external drives stored in a closet that have refused to spin up.

[Brian Bushnell]

Our long-term storage is tape, and we store the raw demultiplexed gzipped fastqs (as well as some processed files, but I don't think that's necessary). Tape is much cheaper per GB, and since your disks are offline anyway, I don't see there as being a disadvantage. I suspect tape may be more reliable than HDDs, as well.

[sklages]

Our long-term storage is tape, and we store the raw demultiplexed gzipped fastqs (as well as some processed files, but I don't think that's necessary). Tape is much cheaper per GB, and since your disks are offline anyway, I don't see there as being a disadvantage. I suspect tape may be more reliable than HDDs, as well.

We stored our sanger sequencing and project on tape; nowadays it's too much data, so it is easier to use harddisks.

You do not store any raw run data, e.g. BCLs and/or CIFs etc? What if you/your customer chose a wrong index? No chance of re-processing?

We are currently calculating very loosely 2x4TBx4(HiSeqs) = 32TB/two weeks if we'd run WGS PE100. Rawdata, including CIFs. With the 1TB update this amount would be more than doubled (in theory and when saving CIFs).

[Brian Bushnell]

We generally don't let customers choose indices. Typically, we prepare the libraries, sequence, quality-control, and assemble or otherwise analyze the data to verify the process was successful before archiving. Most of our customers don't want the raw data anyway.

It's not entirely clear to me what additional benefit you get from storing anything prior to the fastqs (demultiplexed or not), as long as all the relevant information (lane, bar codes, etc) is in the read headers.

[dpryan]

We're also storing the raw hiseq output to tape. We're talking internally about moving to some sort of cloud-based storage, at least for the majority of our runs that don't contain datasets that require being kept in-house. I had been thinking of amazon glacier, but google looks a bit cheaper at the moment.

Has anyone made the jump to cloud-based storage a la google or amazon? I'd love to hear about experiences.

[GenoMax]

We are currently calculating very loosely 2x4TBx4(HiSeqs) = 32TB/two weeks if we'd run WGS PE100. Rawdata, including CIFs. With the 1TB update this amount would be more than doubled (in theory and when saving CIFs).

That is ~ 6 x 5 TB drives every two weeks?

Do you get bare drives or full external enclosures? I hope you have a storage room set aside for those drives.

[GenoMax]

We're also storing the raw hiseq output to tape. We're talking internally about moving to some sort of cloud-based storage, at least for the majority of our runs that don't contain datasets that require being kept in-house. I had been thinking of amazon glacier, but google looks a bit cheaper at the moment.

Has anyone made the jump to cloud-based storage a la google or amazon? I'd love to hear about experiences.

Devon: There is a subtle but significant difference. Google's nearline storage supposedly offers access with just a 3-5 second delay (so you could compute on it via Google compute, Edit: Not 100% certain about this). Glacier is truly meant for long term storage.

[sklages]

We're also storing the raw hiseq output to tape.

What is the "raw hiseq output" referring to? Does it include CIF files or is this just the "standard" rundata with BCLs etc?