Spike in dataset to compare DESeq vs cuffdiff

[narges]

Hi all,

I wanted to ask if you have any idea to be fair as much as possible when comparing two RNA-seq analysis methods like DESeq and cuffdiff. I know one ideal option is using spikein datasets, or alternatively a simulated dataset but then I have no clue about if there are any publicly available spikein dataset for RNA-seq.
Any help or idea would be appreciated.

[Wolfgang Huber]

Dear Narges

simulation is great to check how well your software does (or approximates) what your theory says, but it does not help with the question whether your theory (or: model) agrees with reality.

As for real data, two criteria that seem to make sense are specificity (how many or few false positives do you find) and sensitivity (how many true positives do you find). As for specificity, this is in fact quite easy with real data, just do the same comparison that you would like to do, but in a "mock" fashion with all samples actually being biological replicates. For sensitivity, you need a "ground truth" of truly differentially expressed genes. These may be hard to come by - but you can use prior biological knowledge, independent experiments, etc. Also, this ground truth does not need to be strictly true for the purpose of method ranking, it is sufficient if it is enriched for truth, and the false genes in there are random (the concept of 'pseudo-ROC' by Richard Bourgon).

Best wishes
Wolfgang