VCF Database

[DataDiviner]

Hi all, I am looking into a DBMS for database storage. We have analyzed VCF Files.

http://www.1000genomes.org/node/101

When I say analyzed I mean data that has been aligned with some other steps but is still in a VCF format.

My question is does anyone have any experience with good database systems? Our problem is when you choose an indexing you may be indexing for one question and when you go to ask a different question your query may not be optimized for the next question. We are looking at systems such as Key/Value, RDBMS, Graph, and BigTable. I only have experience with RDBMS systems would be interested in hearing any and all experience on this problem. Thanks!

[swNGS]

I'm in the process of doing something similar, and rather than reinventing the wheel, am looking into adapting ensembl's variant database as a means of storing vcfs derived from many similar targeted sequencing projects. It's based on a MySQL database and you can get the database structure from ENSEMBL with fairly comprehensive instructions. The attraction is that they have a set of scripts that, for example automate the import process amongst other things

If anyone has any solutions to store e contents of vcf files in a database structure, I would be interested to hear about it too

[xied75]

How big is your dataset? What if you can't hold them in one physical machine? You'll scale-up or scale-out?

[swNGS]

small, multiple repeats of the same experiment generating a few 100's variants per sample, approx a dozen samples at a time. I would however be interested in something that can scale up to exome-sized vcfs