DESeq2 - proper normalization for clustering?

[rutgers2015]

Hello all,

While I know that there is no "right" way to perform clustering, I am wondering whether the rlog normalization in DESeq shrinks the data enough to be used for heirarchical clustering and subsequent analysis of time series expression profiles using cutree().

I am wondering because with rlog, I still get a pretty wide range of values, and my clusters end up not being as "tight" as I want them to be. Would doing something like median centering help, or should I use a different normalization than rlog, like standardization so that all of my rows have mean = 0, sd = 1.

[Michael Love]

You might also try the variance stabilizing transformation, but it could be that the within group variance is just large in your dataset.

Note that centering alone will not affect the distances. We do not recommend scaling the rows to have constant variance, because we do not want the rows with noisy, low counts to contribute equally as the rows with high counts, where we believe there to be more informative signal.

[Wolfgang Huber]

Filtering out likely uninformative variables (e.g. with low overall variance after rlog or VST) can also improve clustering. As in the best case, they only add noise and uniformly increase all distances (Central Limit Theorem) and in other cases they may disproportionally pick up subtle underlying confounders (e.g. "batch effects"). One can also think of this as variable weighting.