Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • BBDuk Out of Memory

    Hello! I am running BBDuk from BBTools to remove reads that map to a mammal host genome (2.3G). I am running the bbduk.sh script on just 1 sample (an R1 and R2; each 12G). Initially, I kept getting an "java.lang.OutOfMemoryError: Java heap space" or "java.lang.OutOfMemoryError: GC overhead limit exceeded" errors, so I started requesting 100G+ of memory for my slurm jobs. The jobs run until the time limit I set is reached (3 days), and they do not produce any output files or even temporary files. The last job I ran (1 node, 15 cores, 300G of memory) is below:

    bbduk.sh in1=R1.fastq in2=R2.fastq out1=clean.R1.fastq out2=clean.R2.fastq outm1=matched1.fq outm2=matched2.fq ref=CC.genomic.fna stats=stats.txt k=31 prealloc hdist=1 rskip=4 -Xmx255g

    The output I received:
    Executing jgi.BBDuk
    Version 38.87

    Initial size set to 2142000000
    36.158 seconds.
    Initial:
    Memory: max=273804m, total=273804m, free=85629m, used=188175m

    slurmstepd: error: *** JOB CANCELLED DUE TO TIME LIMIT ***

    ---------------------
    I have also tried adding the rskip=4 and qskip=4 flags to no avail. I ran the same code on just the first 1000 lines of my fastq file and also received the "out of memory" issue.

    Any idea for what might be going on? Why is this task taking so long and requires so much memory for 1 sample? What I can do to diagnose the issue and actually be able to successfully run my code? I have yet to successfully run the BBDuk code on my data.

  • #2
    You should use a different program for this purpose. Look up "bbsplit.sh". BBDuk is not the right program for this application.

    Comment


    • #3
      Thank you for your reply. That is what I was suspecting. I will look into other tools and/or bbsplit!!

      Comment


      • #4
        Originally posted by connieR View Post
        Thank you for your reply. That is what I was suspecting. I will look into other tools and/or bbsplit!!
        See: http://seqanswers.com/forums/showthread.php?t=41288

        Comment

        Latest Articles

        Collapse

        • seqadmin
          Exploring the Dynamics of the Tumor Microenvironment
          by seqadmin




          The complexity of cancer is clearly demonstrated in the diverse ecosystem of the tumor microenvironment (TME). The TME is made up of numerous cell types and its development begins with the changes that happen during oncogenesis. “Genomic mutations, copy number changes, epigenetic alterations, and alternative gene expression occur to varying degrees within the affected tumor cells,” explained Andrea O’Hara, Ph.D., Strategic Technical Specialist at Azenta. “As...
          07-08-2024, 03:19 PM
        • seqadmin
          Exploring Human Diversity Through Large-Scale Omics
          by seqadmin


          In 2003, researchers from the Human Genome Project (HGP) announced the most comprehensive genome to date1. Although the genome wasn’t fully completed until nearly 20 years later2, numerous large-scale projects, such as the International HapMap Project and 1000 Genomes Project, continued the HGP's work, capturing extensive variation and genomic diversity within humans. Recently, newer initiatives have significantly increased in scale and expanded beyond genomics, offering a more detailed...
          06-25-2024, 06:43 AM

        ad_right_rmr

        Collapse

        News

        Collapse

        Topics Statistics Last Post
        Started by seqadmin, Today, 11:09 AM
        0 responses
        15 views
        0 likes
        Last Post seqadmin  
        Started by seqadmin, 07-19-2024, 07:20 AM
        0 responses
        147 views
        0 likes
        Last Post seqadmin  
        Started by seqadmin, 07-16-2024, 05:49 AM
        0 responses
        121 views
        0 likes
        Last Post seqadmin  
        Started by seqadmin, 07-15-2024, 06:53 AM
        0 responses
        111 views
        0 likes
        Last Post seqadmin  
        Working...
        X