overlapInCore using up all RAM

[skerker]

I have been running RS_HGAP_Assembly.2 in SMRT portal and the job has been failing at the overlapInCore step. I was able to solve the problem by changing the nproc from 64 down to 24 so it would not use up all the available RAM on our system.

Each overlapInCore job was trying to use ~2.5GB of RAM, and the system only has 128GB. With 63 processes trying to run, the job failed. In order to optimize performance of the CA step - how do I determine how much RAM is used by each overlapInCore process?

For example, should I increase ovlThreads, while decreasing ovlConcurrency. I have 64 cores and 128GB RAM, so I'm trying to use as much available computing power as possible.

If anyone can provide insight on this, I'd appreciate it!

Best,
Jeff

From my runCA.spec file:


ovlThreads=integer (default=2)
The number of compute threads to use per overlap job.

ovlConcurrency=integer (default=1)
When not using SGE, the number of concurrent overlap jobs to run at the same time.

My current job is running with


ovlThreads = 1

ovlConcurrency = 24

[rhall]

Optimizing the overlapper isn't particularly straightforward, the best explanation I have found is in the CA documentation:

http://sourceforge.net/apps/mediawiki/wgs-assembler/index.php?title=RunCA

in the "OVL Overlapper" section.

[skerker]

Thanks for the reply. I haven't found much other information either. I guess I'll just have to play around with ovlThreads=integer (default=2) and ovlConcurrency=integer (default=1) and see what happens.

If threads = 2 and Concurrency = 12 is this the same as threads = 1 and concurrency = 24? That is the question I really was getting at.

Maybe it has something to do with how many threads can be processed on each CPU /core. All of this gets into the details of our actual hardware, which is beyond my expertise.

Jeff

[rhall]

I believe, threads 1 and concurrency 24 will use approximately double the memory of threads 2 concurrency 12, if all other parameters are the same. As each concurrent job will allocate its own memory. Really optimizing the OVL requires altering the hash parameters, but that is data and system specific, and will likely involve some trial and error.

[skerker]

Thank you - I appreciate the help and advice.

Jeff