Header Leaderboard Ad

Collapse

seq_crumbs memory issues

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • seq_crumbs memory issues

    Hey all,

    I've been loving the seq_crumbs package for trimming and filtering my Illumina data, and was excited to download the latest update that can handle paired end data. Unfortunately, when I tried to use the tools I noticed that the memory requirements had increased dramatically. I expected a modest increase in processing paired end data compared to single end files, but it was jumping from 8G for single end to 80G for pe. Has anyone else encountered this issue or am I doing something totally wrong?

    Also, does anyone else know of tools for quality filtering and complexity filtering that maintain read pairs? I'm aware of trimmomatic and trim_galore for quality trimming, but I didn't think they do filtering based on the quality/complexity of the read overall.

    Thanks!

  • #2
    Originally posted by DrWorm View Post
    Hey all,

    I've been loving the seq_crumbs package for trimming and filtering my Illumina data, and was excited to download the latest update that can handle paired end data. Unfortunately, when I tried to use the tools I noticed that the memory requirements had increased dramatically. I expected a modest increase in processing paired end data compared to single end files, but it was jumping from 8G for single end to 80G for pe. Has anyone else encountered this issue or am I doing something totally wrong?

    Also, does anyone else know of tools for quality filtering and complexity filtering that maintain read pairs? I'm aware of trimmomatic and trim_galore for quality trimming, but I didn't think they do filtering based on the quality/complexity of the read overall.

    Thanks!
    Hi DrWorm,

    I am one of the developers of the seq_crumbs package. We're interested in the memory hogging issue.
    Versions prior to 0.1.8 had memory related bug that we solved in the 0.1.8 version.

    seq_crumb uses python's standar library for multiprocessing and seems that versions of python <=2.7.3 are affected by a bug related with memory consumption.

    With python 2.7.4 in the latest ubuntu the memory use stays stable but with latest debian and python 2.7.3 memory use grows untill the process is finished.

    Which version of python are you using?
    p.

    Comment


    • #3
      I believe we're using Python 2.6.5. I'll request an update and see if that makes a difference.

      Thanks for the advice!!

      Comment

      Latest Articles

      Collapse

      • seqadmin
        A Brief Overview and Common Challenges in Single-cell Sequencing Analysis
        by seqadmin


        ​​​​​​The introduction of single-cell sequencing has advanced the ability to study cell-to-cell heterogeneity. Its use has improved our understanding of somatic mutations1, cell lineages2, cellular diversity and regulation3, and development in multicellular organisms4. Single-cell sequencing encompasses hundreds of techniques with different approaches to studying the genomes, transcriptomes, epigenomes, and other omics of individual cells. The analysis of single-cell sequencing data i...

        01-24-2023, 01:19 PM
      • seqadmin
        Introduction to Single-Cell Sequencing
        by seqadmin
        Single-cell sequencing is a technique used to investigate the genome, transcriptome, epigenome, and other omics of individual cells using high-throughput sequencing. This technology has provided many scientific breakthroughs and continues to be applied across many fields, including microbiology, oncology, immunology, neurobiology, precision medicine, and stem cell research.

        The advancement of single-cell sequencing began in 2009 when Tang et al. investigated the single-cell transcriptomes
        ...
        01-09-2023, 03:10 PM

      ad_right_rmr

      Collapse
      Working...
      X