Data storage

[rdeborja]

In our centre, we have been completely overwhelmed at the amout of data that is kept on spinning disk. I'm curious to find out what others are doing with their data storage.

For our GAIIx runs, we no longer analyze from images and keep thumbnails for troubleshooting. We perform our analysis from intensities on both the GAIIx's and the HiSeq's. We use Illumina's pipeline to generate QC results and sequence (fastq files). It produces a considerable number of intermediate files. I'm curious to know what people keep and what is discarded. Also, what do people backup onto tape for safekeeping?

[drio]

Keep the whole bustard output until you have validated the data. After that, just keep the BAM file with all the alignments (and the unmapped reads also).

[simonandrews]

We keep the whole of the run folder, minus the images. For quite a while now Illumina have put their temp files into folders called Temp, so it's easy enough to get rid of these when moving the run folder to the permanent backup.

tar cf - /run_folder --exclude=Temp --exclude=Images | (cd /backup && tar xf -)

Although we only really need the fastq files for the backup we're still keeping intensities until we're completely sure that no journal is going to require these when publishing results. Frankly a statement from the sequence archives and the journals that only BAM/SAM/FastQ files will be required would save ridiculous amounts of money at sequencing facilities around the world.