Why is the growth rate of the SRA decreasing?

[pmiguel]

Growth of the short read archive at EMBL appears to be plateauing:



That is the doubling time is trending upwards:



This is well below the doubling time for raw megabases/$ -- which is around 6 months.

Is some other archive for raw data being used? Or is the raw data simply not being submitted to archives any longer?

--
Phillip

[NicoBxl]

the scale is logarithmic

[GenoMax]

People are hitting ISP data caps trying to upload data

[mgogol]

I think people are slowing down a little on generating data after they realized how much it takes to analyze it.

Or maybe just not sharing.

[pmiguel]

the scale is logarithmic

Yeah, I know. But I would expect the doubling time to be similar to the doubling time for megabases/$--currently about 6 months. Instead is appears to be at 14 months and is trending upwards.

--
Phillip

[maubp]

People are hitting ISP data caps trying to upload data

Or their Institute's bandwidth isn't up to it

Some of our local sequencing providers can submit direct to the ENA/SRR on your behalf - the only tricky bit is providing the metadata.

[kmcarr]

Perhaps people are realizing there isn't sufficient value in archiving every scrap of raw sequence data produced to justify the cost. I think there is an argument to made that as the cost for each Gbp of sequence decreases so does the value. Not long ago when producing even a Mbp of sequence meant a substantial investment in both dollars and person hours you made sure that every bp of DNA you sequenced was meaningful and to protect that investment by having your data safely stored for posterity. Now one can produce hundreds of Gbp for orders of magnitude less effort and money so researchers are a somewhat less choosey about what and how much they sequence.

Let's be honest, how much raw sequence is ever downloaded from the ENA or SRA for research purposes. I agree with the NCBI's current stance on submission of raw sequence to the SRA. They will accept submissions of raw sequence that are directly reported on in a publication or that correlate to an analyzed data set in some other repository at NCBI (e.g. GEO, Genome, etc.)

[james hadfield]

We have been sequencing like this for three to four years (see the jump in 2008) and thats about as long as most PhDs and post-docs work on a project before moving on. Maybe everyone is enjoying a long summer after a crazy time in the lab and before writing all this data up!

[srasdk]

A growing portion of sequencing capacity is occupied by human disease studies(cancer, diabetes,etc..) and private medical/pharma sequencing. The former is not exchanged between archives due to differences in privacy laws, the latter stays private.

[Richard Finney]

"differences in privacy laws".

How come there's not much public cancer data sets?

Are there laws preventing people from making their genome public? I imagine the motivation to help others suffering from a disease that is killing them might be pretty strong.

If ethics is the problem, perhaps the ethics needs to be over-hauled. A thousand eyeballs looking at some of the problems in cancer might bring a lot of solutions, particularly if patients are willing to let their genomes out.

[srasdk]

A researcher can get access to cancer data through an application process, where he/she is effectively promising not to use it for non-consented research. The data is public, but with concent-based limitations.
What I was pointing out is that the data is not exchanged between archives due to different application processes which is due differences in privacy laws. So ENA does not count NCBI cancer data and vise versa. As a result, it is hard to calculate how much data is currently produced and archived.

[damiankao]

People just aren't submitting to the SRA because its a pain in the ass honestly. Sequences are being generated in such huge volumes and speed, I think it's hard for users to keep up with submissions.

[samanta]

We saw the same trend (slowing down of SRA growth) -

http://www.homolog.us/CI/index.php/charts/growth_sra

There are three possibilities -

i) The exponential spike was due to US stimulus spending. Now we are seeing Tea Party decline.

ii) SRA scared people about shutting down early last year, and that may have forced some to change submission style.

iii) Everyone has too much data and they are down to analysis.