Managing next gen data, HELP!

[rdeborja]

With higher density runs, I'm curious to know how people are handling data. We have Illumina and AB instruments and handling the data off the instruments is becoming very cumbersome. We currently have 1.2PB of storage which is constantly being pruned for space since we're ramping up production.

We perform the analysis of Illumina runs from Intensities (I.e. Running Bustard from Intensities) and do not work from the on instrument Basecalls. No Images are copied over other than thumbnails for troubleshooting purposes. Prior to backing up our data we perform a cleanup (delete the on instrument Basecall directory, leave the export, sequence and Summary.htm related files in the Gerald directory).

For AB data, we keep only the csfasta and qual files and the analyzed data output. All other intermediate files are deleted prior to backup.

Any advice from others on handling all the data?

[maubp]

Could you store CSFASTQ instead of CSFASTA and QUAL files? Pre-compression that should save a lot of disk space (in fact, even CSFASTA + CSFASTQ should still take less space than CSFASTA + QUAL).

The reason is simply QUAL needs 2 or 3 bytes per score (separator and one or two digits), while (colour space) FASTQ just needs 1 byte per score.

Assuming you are compressing the files anyway this may not make such a big difference, but it might be worth trying.