Cuffdiff q-values

[agwe]

I am having problem understanding the Cuffdiff q-values. I am using Cufdiff v.2.1.1 and I understand that in this version the lowest possible p-value one would see is 5e-05. For my analysis, with 2 conditions and 10 biological replicates each, I am using a GTF file with ~29,000 genes and ~205,000 transcripts.

As a result I can see that e.g. 300 isoforms in the isoform_exp.diff file have a p-value of 5e-05, but because of the multiple testing correction the q-values for those isoforms are all equal to 0.0504822, so not significant... When I look at the actual values of the replicates FPKMs, they look like truly differentially expressed. It seems that with my setup, using a GTF file with so many possible transcripts, I will never arrive at anything statistically significant with Cuffdiff...

Similarly, in case of differential gene expression, I have two such experiments, both using the same Cuffmerge generated GTF file, I see:
1) 199 significant genes, while only 136 of them have p-value of 5e-05
2) no significant genes, while 12 genes have p-values of 5e-05

In both experiments I am looking at the same number of genes, shouldn't then the 12 genes from experiment 2) be marked as significant?

Does anyone see similar behavior?

[TiborNagy]

If you use a statistical test many times, you would get significant results by chance. To reduce the number of false positives come from this phenomenon, multiple test correction was used. So you always get equal or smaller number of significant results after multiple test correction.

[agwe]

I understand the principle behind multiple testing correction. What I don't understand is why in my results I get 12 genes with p-value of 5e-5, which is the lowest value one can see in Cufflinks, yet those are not significant after correction...

And as the multiple testing correction q-value depends on the number of tests run, or in other words genes/transcripts tested, if I test 200,000 transcripts I will never get a significant p-value in this setup.

[dpryan]

The multiple testing q-value will depend on the distribution of p-values, not just their number. At least that's the case with the method that's likely being used (Benjamini-Hochberg). It's the incorporation of their distribution (in the null case, we expect p-values to be uniformly distributed, so we can use that expectation to adjust things) that allows you to have significant adjusted p-values even after 10s or 100s of thousands of tests, even with a minimum p-value of 5e-5 (though this doesn't help things).