DESeq filtering

[aoifemc]

I have a question about filtering prior to DE analysis using DESeq -is common practice to just use all raw count values for conditions (i.e. disease/control) or would it be better to ensure that every gene being tested for differential expression has at least say 5 or 10 reads in each of the replicates within the condition groups. I understand that DESeq can account for low read counts & I've found in my data that I get a larger number of genes that are significant when I filter as I guess there aren't as many tests being conducted however I'm unsure which i should go with.
I know this kind of question will depend on the data itself and there may be no right or wrong answers but I know very few people doing this kind of analysis with whom I can discuss ...and in my data I've found that not all of the genes that are significant with the unfiltered data are still significant when I filter(despite the inc in the number of genes reaching significance) so Im just wondering what people consider best practice ??
thanks for any advice

[dpryan]

Have a look at this paper.

[aoifemc]

Great-Thanks!!

[Simon Anders]

Note also that we have recently expanded the DESeq vignette with a section discussing such filtering.

[pinki999]

Can we carry out VST (variance stabilizing transformation) after the filtering step?

[rfilbert]

pre-filtering is a bad idea. Why do people do it? Because the software can't handle large data?

[dpryan]

pre-filtering is a bad idea. Why do people do it? Because the software can't handle large data?

There's no benefit to performing tests on genes that have no chance at showing a difference due to counts being too low. Had you bothered to read the paper I referenced, you would have known that.

[chadn737]

pre-filtering is a bad idea. Why do people do it? Because the software can't handle large data?

1) Specifically why is prefiltering "a bad idea"?

2) It has nothing to do with not being able to handle "large data". Detection power is reduced due to the number of genes tested, and is true for ALL software. Prefiltering is a way of addressing the issue by removing those genes that are unlikely to be differentially expressed and so reduce the overall number of tests performed.

[pinki999]

But, is it a good idea to filter before variance stabilizing?

[rfilbert]

why pre-filter at all? It is only an opportunity for false negatives. I think the only reason for filtering is software that can't handle the whole genome.

[rfilbert]

There's no benefit to performing tests on genes that have no chance at showing a difference due to counts being too low. Had you bothered to read the paper I referenced, you would have known that.

Clearly you have little background or access to a real statistician. Hello World! You must filter out low abundance transcripts - they are clearly not important!

[chadn737]

why pre-filter at all? It is only an opportunity for false negatives. I think the only reason for filtering is software that can't handle the whole genome.

Clearly you have little background or access to a real statistician. Hello World! You must filter out low abundance transcripts - they are clearly not important!

Clearly you don't either, other than the salesman at Partek.

Prefiltering reduces False Negatives. When one prefilters, you are usually removing genes with zero or very few counts. At very low count numbers the shot noise can dominate and all but the most significant changes will not be considered differentially expressed. As typically there will be hundreds, if not thousands of genes with zero to only a few reads, this has a huge affect on multiple testing correction and can lead to a large number of False Negatives. Removing these genes relaxes the multiple testing correction so that more of these genes pass the test.

Prefiltering can increase False Positives, but you said False Negatives, not False Positives. However, this can largely be mitigated if you filter using methods like those described in the PNAS paper linked to previously.

Every program for differential expression has no problem handling entire genomes. I have used DESeq and EdgeR on genomes twice the size of the human genome with ease. Try using them before you blindly criticize.

[chadn737]

But, is it a good idea to filter before variance stabilizing?

I'm not sure. What are you wanting to do with the variance stabilized data? As Filtering is usually done to increase detection of differential expression, there may not be any advantage in doing it for other purposes.

[rfilbert]

Clearly you don't either, other than the salesman at Partek.

Prefiltering reduces False Negatives. When one prefilters, you are usually removing genes with zero or very few counts. At very low count numbers the shot noise can dominate and all but the most significant changes will not be considered differentially expressed. As typically there will be hundreds, if not thousands of genes with zero to only a few reads, this has a huge affect on multiple testing correction and can lead to a large number of False Negatives. Removing these genes relaxes the multiple testing correction so that more of these genes pass the test.

Prefiltering can increase False Positives, but you said False Negatives, not False Positives. However, this can largely be mitigated if you filter using methods like those described in the PNAS paper linked to previously.

Every program for differential expression has no problem handling entire genomes. I have used DESeq and EdgeR on genomes twice the size of the human genome with ease. Try using them before you blindly criticize.

Um, if you filter out a gene that is truly differentially expressed, that is a false negative. Are you a statistician? Seems you are not.

[chadn737]

Um, if you filter out a gene that is truly differentially expressed, that is a false negative. Are you a statistician? Seems you are not.

I see you aren't getting it.

Genes with very few reads, of which there can be hundreds or thousands, are unlikely to be called as differentially expressed, unless the differences are very large.

On the other hand, because of multiple testing correction, many genes of higher expression will not pass the threshold of being considered differentially expressed.

Filtering will result in a handful of genes with low expression that are differentially expressed being discarded, but will allow for many more genes of higher expression to pass the threshold of being differentially expressed.

So overall MORE genes are called differentially expressed and the overall number of false negatives is decreased.

You should take a day or two to actually read some papers on the matter. Like the linked PNAS paper, the DESeq, EdgeR, etc papers and vignettes. These all explain exactly what they do so that nothing is hidden, unlike the black boxes that you are putting your data into.