PacBio quiver memory usage

[colindaven]

Dear all,

I have created a pure pacbio assembly for a medium sized genome (~700mb) with PBcR and have now attempted a quiver polishing step.

The assembly is very impressive, yet the first step of quiver has led to a C++ exception. Monitoring shows the general memory usage to be over 500Gb which lead to the exception on a 512GB server.

Is this normal behaviour ? About 4.5m reads were used.

Can I iteratively use parts of the baxQuiverInput.fofn ?

Thanks!
Colin



[ERROR] BlasrService returned a non-zero exit status. [INFO] 2015-04-29T13:07:38 [blasr] started.
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
Aborted

[1]+ Exit 1 pbalign --nproc 32 --forQuiver baxQuiverInput.fofn asm.scf.fasta

[rhall]

To quiver correct such a large assembly the processing will have to be split up. Do you have access to a full SMRT Analysis install, the BAM_Resequncing_Beta.1 protocol deals with this if SMRT Analysis has been setup on a cluster.

Otherwise you will have to split up the computation manually. https://github.com/PacificBioscience...-using-pbalign

Once you have the megered cmp.h5 from pbalign you will probably want to split it up again by reference contig to create multiple small jobs for Quiver.

[colindaven]

Dear rhall,

thanks for your assistance! I am doing it the manual way which isn't too tricky.

The split command is helpful here and might be useful in the PacBio wiki:
split -d -l 3 --additional-suffix=.fofn ../baxQuiverInput.fofn baxQuiver_

Are there any ballpark figures for how much memory pbalign requires per SMRT cell ?
I'm trying 30 SMRT cells on the 512GB server initially.

Regards,
Colin

[GenoMax]

@colin: Unrelated query.

That looks like a nice big data set that is pure PacBio. Are all 30 cells similar in terms of yield etc (same library being sequenced)? Are you able to tell us anything about average stats (read lengths, size of data) etc? How many chromosomes/ploidy for this genome?

[colindaven]

I haven't looked at yield per SMRT cell. The same library is being sequenced. Read lengths are critical, and are ~10.5 kbp median before filtering and ~8.5kbp median after filtering to subreads.

[rhall]

Are there any ballpark figures for how much memory pbalign requires per SMRT cell ?
I'm trying 30 SMRT cells on the 512GB server initially.

Per cell memory will be predominantly dependent on the reference size (human can be aligned on a 16Gb machine). The memory scaling with number of SMRT cells is a known bug and will likely be fixed soon. When pbalign is used in SMRT pipe workflows this isn't encountered as jobs are split up by SMRT cell in order to allow parallel execution across cluster nodes.