Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • PacBio quiver memory usage

    Dear all,

    I have created a pure pacbio assembly for a medium sized genome (~700mb) with PBcR and have now attempted a quiver polishing step.

    The assembly is very impressive, yet the first step of quiver has led to a C++ exception. Monitoring shows the general memory usage to be over 500Gb which lead to the exception on a 512GB server.

    Is this normal behaviour ? About 4.5m reads were used.

    Can I iteratively use parts of the baxQuiverInput.fofn ?

    Thanks!
    Colin



    [ERROR] BlasrService returned a non-zero exit status. [INFO] 2015-04-29T13:07:38 [blasr] started.
    terminate called after throwing an instance of 'std::bad_alloc'
    what(): std::bad_alloc
    Aborted

    [1]+ Exit 1 pbalign --nproc 32 --forQuiver baxQuiverInput.fofn asm.scf.fasta

  • #2
    To quiver correct such a large assembly the processing will have to be split up. Do you have access to a full SMRT Analysis install, the BAM_Resequncing_Beta.1 protocol deals with this if SMRT Analysis has been setup on a cluster.

    Otherwise you will have to split up the computation manually. https://github.com/PacificBioscience...-using-pbalign

    Once you have the megered cmp.h5 from pbalign you will probably want to split it up again by reference contig to create multiple small jobs for Quiver.
    Last edited by rhall; 04-30-2015, 10:36 AM. Reason: added link

    Comment


    • #3
      Dear rhall,

      thanks for your assistance! I am doing it the manual way which isn't too tricky.

      The split command is helpful here and might be useful in the PacBio wiki:
      split -d -l 3 --additional-suffix=.fofn ../baxQuiverInput.fofn baxQuiver_

      Are there any ballpark figures for how much memory pbalign requires per SMRT cell ?
      I'm trying 30 SMRT cells on the 512GB server initially.

      Regards,
      Colin

      Comment


      • #4
        @colin: Unrelated query.

        That looks like a nice big data set that is pure PacBio. Are all 30 cells similar in terms of yield etc (same library being sequenced)? Are you able to tell us anything about average stats (read lengths, size of data) etc? How many chromosomes/ploidy for this genome?

        Comment


        • #5
          I haven't looked at yield per SMRT cell. The same library is being sequenced. Read lengths are critical, and are ~10.5 kbp median before filtering and ~8.5kbp median after filtering to subreads.

          Comment


          • #6
            Are there any ballpark figures for how much memory pbalign requires per SMRT cell ?
            I'm trying 30 SMRT cells on the 512GB server initially.
            Per cell memory will be predominantly dependent on the reference size (human can be aligned on a 16Gb machine). The memory scaling with number of SMRT cells is a known bug and will likely be fixed soon. When pbalign is used in SMRT pipe workflows this isn't encountered as jobs are split up by SMRT cell in order to allow parallel execution across cluster nodes.

            Comment

            Latest Articles

            Collapse

            • seqadmin
              Recent Advances in Sequencing Analysis Tools
              by seqadmin


              The sequencing world is rapidly changing due to declining costs, enhanced accuracies, and the advent of newer, cutting-edge instruments. Equally important to these developments are improvements in sequencing analysis, a process that converts vast amounts of raw data into a comprehensible and meaningful form. This complex task requires expertise and the right analysis tools. In this article, we highlight the progress and innovation in sequencing analysis by reviewing several of the...
              05-06-2024, 07:48 AM
            • seqadmin
              Essential Discoveries and Tools in Epitranscriptomics
              by seqadmin




              The field of epigenetics has traditionally concentrated more on DNA and how changes like methylation and phosphorylation of histones impact gene expression and regulation. However, our increased understanding of RNA modifications and their importance in cellular processes has led to a rise in epitranscriptomics research. “Epitranscriptomics brings together the concepts of epigenetics and gene expression,” explained Adrien Leger, PhD, Principal Research Scientist...
              04-22-2024, 07:01 AM

            ad_right_rmr

            Collapse

            News

            Collapse

            Topics Statistics Last Post
            Started by seqadmin, Yesterday, 06:57 AM
            0 responses
            12 views
            0 likes
            Last Post seqadmin  
            Started by seqadmin, 05-06-2024, 07:17 AM
            0 responses
            16 views
            0 likes
            Last Post seqadmin  
            Started by seqadmin, 05-02-2024, 08:06 AM
            0 responses
            19 views
            0 likes
            Last Post seqadmin  
            Started by seqadmin, 04-30-2024, 12:17 PM
            0 responses
            24 views
            0 likes
            Last Post seqadmin  
            Working...
            X