[Simon Anders]

Yes, I did. Why?

[sdriscoll]

isn't this what they call trolling?

[chadn737]

I've been following this thread with interest as I always find Simon's answers to be enlightening. I would remind you all that this is not the first time rskr has picked fights with people here or the first time he has asserted that everything everyone else does is bunk.

[rskr]

You realise how disingenuous that reply is, I hope. He already directly addressed this above.

That is the paper he keeps citing as a justification for filtering genes prior to doing tests of significance to improve whatever results he is trying to get, and I don't think that particular paper really applies to whatever it is that he is doing since all the simulations were done with normally generated priors as if they were microarrays, and it isn't very good in the first place since it leads to somewhat absurd assumptions about filtering.

[rskr]

Not to belabor the point, but I don't see any reason why two classes with small variance and a small but significant difference in means, should be discarded in preference for classes with large variance and a large difference in means.

[sdriscoll]

Not to belabor the point, but I don't see any reason why two classes with small variance and a small but significant difference in means, should be discarded in preference for classes with large variance and a large difference in means.

I'm finally reading this paper now and the first question to you that popped into my head is - have you not heard of dimension reduction? What they are doing here is very similar to the reason one would apply PCA in other types of data. PCA doesn't work to reduce variables when the number of variables is greater than the number of samples so it can't be applied here but the concept is the same. That being the information is in the variance.

So do you mean to argue against all forms of dimension reduction in all applications or only when applied specifically to differential gene expression? I have to tell you - dimension reduction is of extreme usefulness in many fields. This paper simply appears to be proposing a way to perform a type of dimension reduction followed by statistical testing on a variable by variable basis when the number of variables is >> number of samples.

[rskr]

I'm finally reading this paper now and the first question to you that popped into my head is - have you not heard of dimension reduction? What they are doing here is very similar to the reason one would apply PCA in other types of data. PCA doesn't work to reduce variables when the number of variables is greater than the number of samples so it can't be applied here but the concept is the same. That being the information is in the variance.

So do you mean to argue against all forms of dimension reduction in all applications or only when applied specifically to differential gene expression? I have to tell you - dimension reduction is of extreme usefulness in many fields. This paper simply appears to be proposing a way to perform a type of dimension reduction followed by statistical testing on a variable by variable basis when the number of variables is >> number of samples.

PCA does have a similar problem, there are of course other ways to choose orthogonal basis, and for certain applications these may be valid. I am pointing out that in this case, filtering by variance introduces a powerful bias, against the very genes researchers are interested.

[sdriscoll]

So what's your evidence that the genes filtered out are those that researchers are interested in? I think this warrants a case study rather than a theoretical war.

[rskr]

So what's your evidence that the genes filtered out are those that researchers are interested in? I think this warrants a case study rather than a theoretical war.

It doesn't seem very complicated, that researchers would prefer separable genes with smaller variances, to separable genes with larger variances. It's not very hard to model for the normal case to see that given the same variance and p-value, one will overlap and one won't. I think most researchers would be more comfortable with the one that doesn't overlap for doing things like making medical decisions.

[sdriscoll]

I do wonder why low-expressed genes tend to get the short end of the stick in filtering. They aren't irrelevant...they are obviously expressed for some reason unless we want to argue that we've got extra genes expressed doing nothing at all. One explanation I tend to use is that when their count values are low and additionally their coverage is very low across all samples it's hard to say whether what we're seeing is noise or real evidence that the gene is present but expressed very low. We can look at it as a technical limitation of the sequencing run - we just didn't get enough reads to test those genes.

Back to the filtering I do wonder one thing. Low count features tend to have very high coefficients of variation but very low variance values. Highly expressed genes tend to have very small coefficients of variation but very high variance values. Is this maybe why Simon says they use the means instead of the variances in their adaptation of what's outlined in the paper? I fail to see how a linear cutoff could be applied when there's such a clear non-linear trend between count level and variance.