[billstevens]

Thanks.

One very challenging aspect is to go from quantifying differential RNA to quantifying differential protein. I'm using epithelial cells, and I have a few popular cytokines that have differential expression. I tried ELISA's on these cytokines and I couldn't resolve any differences. ELISpot requires cells that can grow in suspension which my experiment is not. Does anyone know any other high-resolution assays to measure cytokines? One thing I'm thinking is maybe its producing more cytokines, but many are staying intracellularly, so I should lyse the cells?

[dpryan]

The ELIspot assay will be the lynchpin for your argument, so I would put most of my efforts there. How many (and which) of the genes you need to also show changed via qPCR will depend a lot on which journal you're sending things too.

[billstevens]

So I used SPIA, and I came up with a couple very interesting pathways. In the pathways, there are some genes that are fold changes of 1.3, 1.5 and some genes that are fold changes over 2. There are about 20 or so genes in this pathway. So I clearly can't get the ones that are not showing much fold change. What I'm thinking is that a the ones that are over 2 are of very high interest, and if I prove just those genes with qPCR, do you think that is enough to prove my statement?

The crux of the paper will be that exposure to this antigen causes this pathway to be upregulated, and it is verified via RNA-Seq, and a couple of highest fold change genes are verified via qPCR, and then those translated proteins are also verified via ELIspot.

[ThePresident]

Once you've clarified what the publication is actually about, usually the answer to validate or not is pretty much self-evident.

Yeah, you're 100% right...! I guess I already knew it but since it's all new for me, I needed a second opinion.

Thanks!

[Wallysb01]

Thanks Wallysb01, that's what I figured. But here's the question, what is "real power"? I have 4-5 biological replicates, and I will be doing it in triplicate (5 technical triplicates, and I just wouldn't have any room left on my plate). Could I get down to 1.5X? What, quantitavely, is your experience?

Well, its hard to say exactly how much power you'll have without knowing what kind of biological and technical noise you're dealing with in your system. But in my experience ~15 biological x technical replicates and you have a good chance at 1.5x becoming statistically significant. It will help if your genes are fairly highly expressed and easy to amplify. When this many replicates fail for me, its often because the ct-values are up close to 30 or more and jump around too much. When ct-values are <25 the technical noise is usually quite low, at least in my experience.

How many genes are you trying to do this with? I highly doubt anyone is going to expect this kind of validation across an entire RNA-seq experiment. Is the issue that a few of you genes of highest interest do not show much fold change?

[mbblack]

I was just about to ask for qPCR validation. What is the routine for RNA-seq , i.e. is it necessary to validate data by qPCR? If I'm about to publish those RNA-seq results, I'm afraid reviewers will ask for qPCR validation even for significantly (<0.01) DE genes (fold change > 2). And, as I see it, it is better to use new RNA samples in order to control for biological differences...?

Thanks in advance,

TP

Whether or not an editor or reviewer will insist on diff. gene validation is entirely dependent on just what the publications is about. If this is a some sort of general genomic characterization experiment (eg. a chemical or other exposure experiment) then very likely no. Just like such an experiment conducted using gene microarrays, as long as the experiment was properly controlled and included sufficient biological replication for robust results, nobody will be expecting validation of quite likely hundreds of putative DE genes.

However, if you are trying to characterize something more specific - a particular metabolic pathway, or you are trying to pick genes to base a bio-assay or clinical assay upon, then validation will likely be deemed a necessary step, at least for your final candidate gene selections.

But it is impossible to say whether your study really needs DE validation without knowing the specifics of what question(s) you are specifically addressing in your publication. Once you've clarified what the publication is actually about, usually the answer to validate or not is pretty much self-evident.

[billstevens]

Thanks Wallysb01, that's what I figured. But here's the question, what is "real power"? I have 4-5 biological replicates, and I will be doing it in triplicate (5 technical triplicates, and I just wouldn't have any room left on my plate). Could I get down to 1.5X? What, quantitavely, is your experience?

[Wallysb01]

I am just about to embark on this for the first time, obviously using MIQE (and help from a post-doc), but everyone in my lab has told me that unless it has at least 2 fold differential expression, I won't able to determine any differences. Is this the experience of other people on here? If so, do you not do qPCR validation for some of these genes that are closer in expression and simply rely on the sequencing data?

This depends on your experimental design. If you're just using two samples with technical triplicates, yes, >2x is about all you can hope to detect. The noise in qPCR is just that high. And if you step back and think about it, it is clear why. PCR relies on a doubling of the DNA with every round of replication, so its hard to pick up on differences less than 2x, even with triplicates. If you start using several biological replicates and expand to 5 technical triplicates you'll start see some real power.

[billstevens]

dpryan, that's good advice. I want to input my results into SPIA, and a couple of pretty cool pathways pop up, but some of thse are fold changes of 1.3 or so (very highly expressed, which is why DESeq called it). I doubt I could get it to pop up on qPCR. Would you say its fair to then say that RNA-Seq has better resolution than qPCR?

ThePresident, if you have fold changes over 2, you should do qPCR. You should be able to show fold changes and it strengthens your case. What I would say is that you do qPCR on your RNA samples, and assuming you have replicates, do qPCR on the replicates as well. Have it in triplicate, that way you are now doing technical noise (RNA-Seq vs. qPCR) and you are doing biological noise (from doing 3 replicates).

[ThePresident]

I was just about to ask for qPCR validation. What is the routine for RNA-seq , i.e. is it necessary to validate data by qPCR? If I'm about to publish those RNA-seq results, I'm afraid reviewers will ask for qPCR validation even for significantly (<0.01) DE genes (fold change > 2). And, as I see it, it is better to use new RNA samples in order to control for biological differences...?

Thanks in advance,

TP

[dpryan]

I am just about to embark on this for the first time, obviously using MIQE (and help from a post-doc), but everyone in my lab has told me that unless it has at least 2 fold differential expression, I won't able to determine any differences. Is this the experience of other people on here? If so, do you not do qPCR validation for some of these genes that are closer in expression and simply rely on the sequencing data?

My experience seems to parallel those of your lab mates. While it's certainly not impossible to detect <2x changes in qPCR, it's not trivial (you'll probably just need more samples). I would also question whether 20% changes in RNA are biologically meaningful. Given the number of samples that would have to be looked at to significantly discern that sort of RNA level change, I probably wouldn't bother unless the candidates were highly explanatory.

[ETHANol]

In regards to qPCR validation, if you are using the same RNA as you did your RNA-seq with, it is meaningless. Well, not meaningless, it means you have controlled for technical noise but not biological noise.

pval vs. padj: This is the perspective from a biologist with very little statistical understand, but thought I might be able to add something. Fools all speak the same language. If you have 10,000 genes and you do pval cutoff of 0.2, while each one of those genes has an 20% chance of being a false positive, you will also get on average 2,000 false positives in your data set. So for example, if you got 2,200 differentially expressed genes, on average only 200 of then would be real.

Where as a padj(FDR) cutoff of 0.2 means on average 20% of the genes in you list are false positives.

If this isn't quite correct someone with a better understanding of statistics please chime in.

[billstevens]

About qPCR validation, my DESeq run calls many genes that have a differential expression difference as low as 20%, significantly differentially expressed. While I appreciate the power of DESeq, I need to verify this with qPCR.

I am just about to embark on this for the first time, obviously using MIQE (and help from a post-doc), but everyone in my lab has told me that unless it has at least 2 fold differential expression, I won't able to determine any differences. Is this the experience of other people on here? If so, do you not do qPCR validation for some of these genes that are closer in expression and simply rely on the sequencing data?

[ThePresident]

Also, replicates come into play too. With too few replicates, your statistical tests have limited power, so you have few statistically significant test results. That in turn means your FDR corrected p-values also will have very few (often none, if you had no replicates or only one or two) significant results, because your p-value distribution did not reflect any discrimination.

Fair well. Actually, I have two biological replicates for each of my conditions. It doesn't give me a huge confidence for my stats, but I assume it's enough to call for the most DE genes. I'll probably miss some of them by being too stringent but my goal is to get those that could have some biological impact. Even if I have a gene whose expression is statistically different from my reference condition, I would not consider it unless the fold change is worth something.

If I could I would have done 3 or more replicates for each of my conditions. But the budget was limiting so... you know the story. Now, I have to work with what I have and I'm trying to use some statistical tools to help me get through and avoid a huge amount of false positives that would cost even more in downstream validation.

But I understand that statistics can me misused and that we should always consider it as a tool in light of our experiment. We often use it straightforward, losing the big picture, and falling down for a p value < 0.05 because that is enough for most peer-review journals right? Not many are going to ask for a detailed statistical analysis, so in many labs (including mine) the main goal is to get a star* (statistical significance) above your histogram.

Anyway, my English is not at the top but I hope you understood what I wanted to say.

Thanks again

[mbblack]

Honestly, I don't have any certitudes concerning the outcome of my experience. I have my control and my test condition. Some genes could be upregulated, others downregulated but I expect that a majority would be unaltered.

From there, and considering your examples, I should expect some of the genes to come necessarily as false positives. So, a prudent way to proceed would be to consider a more stringent approach... using FDR < 0.05 I got 15 genes which is fair well plus I'm confident that I see some true changes in expression.

Thank you for your help

In any genomic differential gene expression experiment, one expects the considerable majority of genes will not be differentially expressed. That's just biology.

The reality also is you will always have some false positives and some false negatives in any large scale statistical analysis since you cannot either truly eliminate all errors (type I and type II), nor could you really know if you had should you try to do so. Using tools like FDR helps to control those types of errors, but it does not eliminate them.

Also, replicates come into play too. With too few replicates, your statistical tests have limited power, so you have few statistically significant test results. That in turn means your FDR corrected p-values also will have very few (often none, if you had no replicates or only one or two) significant results, because your p-value distribution did not reflect any discrimination.

If you actually have no replicates, then it really is pointless to even bother computing the statistics. In that worst case scenario, you'd do best by simply ranking genes by normalized expression or raw counts, and pick those with the greatest difference in observed values (and then validate them independently).

So you have to interpret your results in light of your experimental limitations, as well as what your goal from the analysis was, and adjust things as the situation calls for. The stats are just tools to guide you and add some rigor to your analysis.