Storage Gotchas

[Tom Bair]

We are in the process of getting a short read system. Having been burned by a couple of things with the minor amount of storage demands from the 454 (ext3 <16T, fdisk <2T)

I thought I would ask about what the short read community is doing with storage of runs. It sounds like alot of people are going the "here is usb 2.0 2T drive with all your data you keep it" route. Anybody not doing that?

Any good storage solutions? Ones to avoid?

Also details of file systems xfs, jfs etc that people like/don't like

More of a brainstorming thread than anything, wouldn't feel to bad about drift

Thanks

Tom

[Torst]

We use a 20T volume to store our Illumina run data in a priority queue - so when space is needed we delete the oldest/lowest priority run folders. Of course we always keep the base call and probality files, but delete the images. The 20T is LVM of four 5T RAID5 devices. We use XFS filesystem.

If a user/customer wants to keep the whole raw data set, we will provide it on a USB2 external HDD at their cost.

Ultimately, the price is dropping such that re-sequencing your sample (if possible) is cheaper than keeping the image files etc. I have yet to hear of anyone actually re-running their old images on newer versions of the software, but in theory we could do it. I think the "best practice" today seems to just keep the PRB and SEQ files, and delete the rest.

If you have IPAR with your GA2 (Illumina) it comes with a fair amount of storage anyway.

[joa_ds]

we have an IPAR and we keep a small portion of the images (<1%) together with the sequences on a 12TB raid server. We then delete the images on the IPAR. You can of course move them to an external HD and store them (just see it as a backup).

We keep a small portion of the images for QC-ing. Imagine their is a new algorithm, we can then see what the improvents are on a smaller subset of known results before we implement them on real data. Or to compare several runs in time and compare the results. We don't really know what we will do with it, but might be interesting, certainly not for reanalysis. Just rerun the experiment then, store the DNA in a freezer, it is cheaper.