Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • HTSeq 0.6.1 Maximum alignment buffer size exceeded error

    Hi! I'm usg HTSeq v 0.6.1 to count reads mapping to features (the htseq-count script). I'm getting a

    Code:
    18700000 SAM alignment record pairs processed.
    Error occured when processing SAM input (record #40455805 in file /scratch/me/accepted_hits.bam):
      Maximum alignment buffer size exceeded while pairing SAM alignments.
      [Exception type: ValueError, raised in __init__.py:671]
    My accepted_hits.bam file is generated by mapping with STAR and then coordinate-sorted and converted to BAM using samtools.

    I am using the htseq command
    Code:
    htseq-count --format=bam --order=pos --mode=union --idattr=gene_id --stranded=reverse --type=exon} $WORKDIR/accepted_hits.bam gencode.v19.gtf
    What is wrong and how do I fix it? (Apart from running htseq on the sam file)?

    Thanks in advance,
    Darya

  • #2
    The "--order=pos" option is new and may need a bit of fine-tuning. Perhaps the buffer is a bit small.

    For now, maybe sort the file be read name (with "samtools sort -n") and then use "--order=name".

    Comment


    • #3
      I can confirm this happening:

      Code:
      ...
      37400000 SAM alignment record pairs processed.
      37500000 SAM alignment record pairs processed.
      Error occured when processing SAM input (line 78096327):
        Maximum alignment buffer size exceeded while pairing SAM alignments.
        [Exception type: ValueError, raised in __init__.py:671

      Comment


      • #4
        Same here with
        a) BAM ordered by position
        b) STAR alignments

        Comment


        • #5
          Have hit the same problem with STAR aligner, and positional ordering.

          12200000 SAM alignment record pairs processed.
          Error occured when processing SAM input (record #27408270 in file alignments/S6.bam):
          Maximum alignment buffer size exceeded while pairing SAM alignments.
          [Exception type: ValueError, raised in __init__.py:671]
          Last edited by Bukowski; 03-19-2014, 08:51 AM.

          Comment


          • #6
            I get the same "Maximum alignment buffer size exceeded" whether running my bam file as position or name sorted. It gets farther with the name sorted file than the position sorted file (output size of 1 Gb vs. 600+ Mb respectively) but still fails. My input bam is 11Gb of around 142 million reads.

            Comment


            • #7
              When you used the name-sorted file, have you remembered to omit the "-r pos"?

              Comment


              • #8
                Working now

                Originally posted by Simon Anders View Post
                When you used the name-sorted file, have you remembered to omit the "-r pos"?
                Excellent catch. I reran without that flag and it worked just fine.

                Thank you!

                Comment


                • #9
                  As Simon reads these I just wanted to flag up that I have run into the same issue with a position sorted alignment from STAR as well.

                  I was so excited to see this feature in 0.6 as the position to name based ordering is an annoyingly long step in my workflow! Fingers crossed for a fix soon...
                  @sidderb

                  Comment


                  • #10
                    I get the same with reads aligned with star, and position sorted using samtools.

                    Code:
                    tools/samtools view /home/daniel.klevebring/cust001/tests/fullRNA/L8760/rnaseq/L8760T/L8760T.star.50.stranded.PE.bam 
                    | htseq-count  --stranded=reverse --mode=intersection-nonempty  --order=pos - /mnt/hds/proj/cust001/autoseq_genome/genes/genes_fixed.gtf > /home/daniel.kleveb
                    ring/cust001/tests/fullRNA/L8760/rnaseq/L8760T/L8760T.star.50.stranded.PE.htseq-count.txt
                    
                    ...
                    18900000 SAM alignment record pairs processed.
                    Error occured when processing SAM input (line 40808946):
                      Maximum alignment buffer size exceeded while pairing SAM alignments.
                      [Exception type: ValueError, raised in __init__.py:671] 
                    
                    $ htseq-count |grep version
                    Public License v3. Part of the 'HTSeq' framework, version 0.6.1p1.
                    I don't always get it, and it runs with larger files without a problem. What's the best fix for this? Is there one?

                    Comment


                    • #11
                      It appears that this problem is not addressed in 0.6.1p2 either.

                      It would be extremely nice if there were a way to use htseq-count on positionally-sorted data (which is the default Tophat output). Maybe there could be an option that htseq-count could use to just increase the buffer size? I have 48 GB of RAM on this machine, so it is not possible that htseq-count could actually consume it all on one file.

                      Comment


                      • #12
                        I get this problem as well using tophat2 aligned BAM files, position sorted. I will revert to name sorting.

                        Comment


                        • #13
                          Is there a way to increase the buffer size?

                          I got this error with a pos-sorted.bam file (3.2GB) created with tophat2

                          Code:
                          [xxx@dev-intel14-phi 20131017mRNA_gfCC]$ htseq-count --format=bam --stranded=no --order=pos \
                          > ~/RTSF/hiseq/20131017mRNA_gfCC/ms12cc2/accepted_hits.bam \
                          > ~/RTSF/hiseq/20131017mRNA_gfCC/genes.gtf \
                          > > ~/RTSF/hiseq/20131017mRNA_gfCC/htseq/ms12cc2.txt
                          100000 GFF lines processed.
                          ...
                          16100000 SAM alignment record pairs processed.
                          16200000 SAM alignment record pairs processed.
                          Error occured when processing SAM input (record #35593187 in file /mnt/home/xxx/RTSF/hiseq/20131017mRNA_gfCC/ms12cc2/accepted_hits.bam):
                            Maximum alignment buffer size exceeded while pairing SAM alignments.
                            [Exception type: ValueError, raised in __init__.py:671]
                          [xxx@dev-intel14-phi 20131017mRNA_gfCC]$ Write failed: Broken pipe

                          Comment


                          • #14
                            Two options:

                            You can either re-sort your files lexographically/alphabetically and run htseq-counts with the lexigraphic order (in other words, NOT --order=pos)

                            or

                            You can switch to the much faster and generally more user friendly (easier options, easier output file format) program "featureCounts". Part of the "subread" package. http://subread.sourceforge.net/

                            I would switch to subread. htseq-counts is incredibly slow and very poor at handling large files without crashing.


                            [QUOTE=dc3000;161213]Is there a way to increase the buffer size?

                            I got this error with a pos-sorted.bam file (3.2GB) created with tophat2

                            [CODE]
                            [xxx@dev-intel14-phi 20131017mRNA_gfCC]$ htseq-count --format=bam --stranded=no --order=pos \

                            Comment


                            • #15
                              P.S. subread only takes about 10 minutes to install. It doesn't have any complicated dependencies. Once again: http://subread.sourceforge.net/

                              Comment

                              Latest Articles

                              Collapse

                              • seqadmin
                                Exploring the Dynamics of the Tumor Microenvironment
                                by seqadmin




                                The complexity of cancer is clearly demonstrated in the diverse ecosystem of the tumor microenvironment (TME). The TME is made up of numerous cell types and its development begins with the changes that happen during oncogenesis. “Genomic mutations, copy number changes, epigenetic alterations, and alternative gene expression occur to varying degrees within the affected tumor cells,” explained Andrea O’Hara, Ph.D., Strategic Technical Specialist at Azenta. “As...
                                07-08-2024, 03:19 PM
                              • seqadmin
                                Exploring Human Diversity Through Large-Scale Omics
                                by seqadmin


                                In 2003, researchers from the Human Genome Project (HGP) announced the most comprehensive genome to date1. Although the genome wasn’t fully completed until nearly 20 years later2, numerous large-scale projects, such as the International HapMap Project and 1000 Genomes Project, continued the HGP's work, capturing extensive variation and genomic diversity within humans. Recently, newer initiatives have significantly increased in scale and expanded beyond genomics, offering a more detailed...
                                06-25-2024, 06:43 AM

                              ad_right_rmr

                              Collapse

                              News

                              Collapse

                              Topics Statistics Last Post
                              Started by seqadmin, Yesterday, 06:53 AM
                              0 responses
                              12 views
                              0 likes
                              Last Post seqadmin  
                              Started by seqadmin, 07-10-2024, 07:30 AM
                              0 responses
                              34 views
                              0 likes
                              Last Post seqadmin  
                              Started by seqadmin, 07-03-2024, 09:45 AM
                              0 responses
                              204 views
                              0 likes
                              Last Post seqadmin  
                              Started by seqadmin, 07-03-2024, 08:54 AM
                              0 responses
                              213 views
                              0 likes
                              Last Post seqadmin  
                              Working...
                              X