A stats question about DESeq

[a11msp]

It may be a slightly outdated question now that GLM-based testing is preferred to the exact test, but I'm just very curious...

In the 2010 Genome Biology paper (http://genomebiology.com/2010/11/10/r106), formula 14 describes how in the negative binomial test, variance of condition A is estimated under the null hypothesis of the equality of per-condition means:

σˆ2_{A} = ∑{all_j∈A} [ s{j}*q0+s{j}ˆ2 * v{A}(q0) ],

where:
∑{all_j∈A} is the sum across all samples in condition A,
s{j} are scaling factors for each of these samples,
q0 is the pooled mean estimate,
v{A} is the raw variance estimate for condition A given the mean.

However, looking at the DESeq code, I'm not sure v{A} is actually estimated based on q0. Rather, it seems to me it is estimated based on q{A} - the observed mean of counts for condition A.

Am I getting it wrong or there have been some changes that I'm not aware of?

Thanks very much!

[Simon Anders]

If I am still read my old code correctly, then the variance-mean relation for condition A is fitted by a local regression of the per-gene variances onto the mean of the counts only from condition A (in the code for the estimateDispersion method, 'if' clause for 'per-condition': bmv$baseMean, where bmv is calculated from only the columns for the current condition), but to get the fitted value, this function is then evaluated for the overall mean (in the code: overall_basemeans). This is consistent with what we wrote in the paper, I think.

However, in DESeq2, we now dropped the "per-condition" mode, because it was a bit difficult to reconcile with our general improvements to the dispersion estimation.

[a11msp]

Thanks a lot for your reply Simon!

[a11msp]

Sorry, one more question... Am I right then that for sharing mode == "maximum" and method == "per-condition", the maximum is computed between the fitted SCV given the overall mean and the observed SCV given the mean for a given condition?

Many thanks!

[Simon Anders]

Just looked through my code again. Yes, this is what I've put there. The reason, If I remember correctly, had something to do with the handling of zeroes.

[a11msp]

OK, thanks!