number of cores in DEXSeq using parallel library

[alittleboy]

I am using DEXSeq and one of the steps is to estimateDispersions. It offers the parallel computing functionality via the parallel R package. In particular, I can run:

estimateDispersions(pExons, nCores=3, quiet=TRUE)

I think here, nCores=3 indicates that 4 CPU's are available but we only use 3 to avoid any disadvantageous overhead. My question is, because I am using a cluster, and the command:

$ less /proc/cpuinfo

displays different processors information (for a total of 15 processors), and each processor has 4 CPU cores. In this case, what shall I specify for the "nCores=" argument? Still 3? Thank you!

[dpryan]

If you're looking at the number of "processor" lines in /proc/cpuinfo, that will typically be the number of cores (or hyperthreads, depending on the cpu) available. For example, my desktop computer has a single processor with 6 cores (each with hyperthreading, so sort of 12). /proc/cpuinfo, then, reports 12 processors. The nodes on our cluster have two processors like this, so they'll report 24 "processor" lines in /proc/cpuinfo.

It's usually most convenient to just think about number of available cores, not processors. If you use, say, 12 cores, they'll likely just be distributed over all of the physical processors.

[alittleboy]

If you're looking at the number of "processor" lines in /proc/cpuinfo, that will typically be the number of cores (or hyperthreads, depending on the cpu) available. For example, my desktop computer has a single processor with 6 cores (each with hyperthreading, so sort of 12). /proc/cpuinfo, then, reports 12 processors. The nodes on our cluster have two processors like this, so they'll report 24 "processor" lines in /proc/cpuinfo.

It's usually most convenient to just think about number of available cores, not processors. If you use, say, 12 cores, they'll likely just be distributed over all of the physical processors.

Hi @dpryan:

Thanks for the information! So in my situation, since I have 15 processors, each having 4 cores, then I can specify nCores=45 so that each processor can be distributed with 3 cores?

[dpryan]

On average, at least (there's no guarantee that things will be equally distributed).

I should note that having 15 actual CPUs on one system is unusual (I've never heard of it, at least). I suspect that you actually have a system with 16 cores (the counting starts at 0), which would be more common. Also, even with an infinite number of cores, performance won't always increase with increasing number of allocated cores. So, if you'll be doing this a lot then just run a few tests to find out what's fastest.

[alittleboy]

On average, at least (there's no guarantee that things will be equally distributed).

I should note that having 15 actual CPUs on one system is unusual (I've never heard of it, at least). I suspect that you actually have a system with 16 cores (the counting starts at 0), which would be more common. Also, even with an infinite number of cores, performance won't always increase with increasing number of allocated cores. So, if you'll be doing this a lot then just run a few tests to find out what's fastest.

Hi @dpryan:

Yes, you're right, there are 16 CPUs on the system... I set nCores=4 and so far the program runs well.

I am a little bit surprise that for a simple two-group comparison (control vs. treatment), with >55,000 "genes" (some rows are like ENSG1+ENSG2+...), it might take more than 2 days to get the estimateDispersions results... I guess for the differential exon testing part, it may take that long as well... I also tried to include another covariate in the model, and after 4 hours, there is not a single dot (meaning 100 genes processed) shown on screen (for disp. estimation)!

Is it common for exon-level inference to take such a long time?

Thanks ;-)

[dpryan]

Yes, a lot of the DEXseq steps can take quite a long time to complete since there are so many exons to calculate. You might try increasing nCores to 12 or 16.

[alittleboy]

Yes, a lot of the DEXseq steps can take quite a long time to complete since there are so many exons to calculate. You might try increasing nCores to 12 or 16.

Hi @dpryan:

Thanks! I realize that it seems to be faster than expected: the speed for each 100 genes slot differs, and I notice that it takes much longer for the 1st point to appear ;-) The dispersion estimation took ~9 hours and DEU <2 hours.

[jialu]

Hi @dpryan:

Thanks! I realize that it seems to be faster than expected: the speed for each 100 genes slot differs, and I notice that it takes much longer for the 1st point to appear ;-) The dispersion estimation took ~9 hours and DEU <2 hours.

Hi alittleboy,

Could you see any dot when you used multiple cores? I can see dots when I use only one core but there's no dot when I use multiple cores (after one day). Is that normal?

Thank you,

-Jia

[dwb0211]

Hi alittleboy,

Could you see any dot when you used multiple cores? I can see dots when I use only one core but there's no dot when I use multiple cores (after one day). Is that normal?

Thank you,

-Jia

The same as you when I set nCores=8.