What do you do with your database?

[quantrix]

Dear All,
In a different thread which I posted earlier (Core cluster setup...), westerman suggested I post a new thread with the question what do you do with your DB. I am indeed curious what do you do with your database? I would believe that trying to store the NGS data in something like a SQL database is a lost enterprise. So my questions are

1) What do you do with your Database?

2) How do you store your NGS data?

3) Do you have any troubles with accessing your data on a repeated basis?

4) What are the biggest bottlenecks you commonly encounter with regards to the data management?

5) Do you have a commercial solution or a home-grown one?

Thank you for your time and I shall look forward to your replies.
Regards
Quantrix

[colindaven]

At the moment we don't use a database. As you say the files are huge. It would be important to store variants etc for comparison if you are always working on one large project, but here we have a lot of smaller/medium projects which aren't relevant to compare to each other.
Also keep in mind your users might not be trained in database-based analysis, so a good front end will be important.

[mapper]

Hi

Here are some links which will be helpful

http://blog.bioteam.net/wp-content/uploads/2010/03/cdag-xgen-storageForNGS_v3.pdf

http://bioteam.net/dag/CHI-09-NGS/NextGenDataWorkshop_20090920.pdf

http://bioteam.net/dag/CHI-09-NGS/Scalable%20Storage%20for%20Life%20Sciences.pdf

http://bioteam.net/dag/UniCluster/

[jkbonfield]

We had an oracle DB housing our SRF + fastq files when we were still generating those. It was huge, but worked well and had a fuse layer to transparently make it visible to the users. The DB was in two halves - a large set of partitions holding blobs (actually oracle "secure files" I think) and a far smaller meta-data component that tracked where things were. It would have worked OK using a filesystem instead of the binary blobs though - there are pros and cons to each method.

We've since switched both format and DB mechanism for raw data: we store BAM files in an iRODs system.

The analysis bams & co (ie mapped or assembled data, vcf files, etc) are less clearly divided - stored in various project/group directories over a variety of file system types; slow & fast NFS storage, lustre, etc.

The only real bottlenecks are if someone tries to access a single DB layer (like the fuse layer) from 1000+ cores on our cpu farm. We require that people copy data to something more scalable first which we use Lustre for.

[quantrix]

Hi Mapper and jkbonfield,
That is very helpful indeed!
Thanks
Quantrix