ALLPATHS LG failed while running RemoveDodgyReads

[leshin]

I was trying to run the latest version of ALLPATHS-LG on our cluster for several times. But my jobs always failed at the same point that is running the RemoveDodgyReads job. The error message I got (shown as below) from RemoveDodgyReads.out file seems showing that my job ran into some memory issues. But our machine has about 500GB memory and the overall memory usage of my job before it failed was only 198GB, so the memory size should not be an issue. I really have no idea how to solve this problem and hope someone can provide some advice. Thanks a lot.

--wls

****************************************************************

Call to new failed, memory usage before call = 198027084k.

Here is the output of top:

Aborting.

Wed Dec 12 08:39:58 2012. Abort. Stopping.

Generating a backtrace...

Dump of stack:

0. CRD::exit(int), in Exit.cc:49
1. our_new_handler(), in RunTime.cc:577
2. Mempool:reAllocate(unsigned long), in Mempool.cc:92
3. Mempool:reAllocate(unsigned long, unsigned long), in Mempool.h:59
4. MempoolAllocator<unsigned char>:reAllocate( ... ), in Mempool.h:243
5. MasterVec<SerfVec<unsigned char> >:reAlloc( ... ), in MasterVec.h:296

[crusoe]

Are you sure that no other jobs or processes were running at the same time?

On some clusters you may need to specify how much memory your job needs. Perhaps there is a low default maximum memory for your cluster. Ask your admins and check other errors logs for more details.

Or just re-run it and see if it happens again.

[christinawu2008]

I have the problem, Can I please know how you ended up?

[leshin]

After some profiling by using the collectl, I found that the last program running before my allpaths-lg job terminated is FindErrors. And the reason was FindErrors tried to allocate more memory than it is available on the machine for loading some files. So I reran my job on another machine that has 2TB memory and it worked. Hope this will help.

ls