Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Picard error

    Hi,

    I'm having trouble understanding some of the params with Picard- well not really understanding but I'm getting this error here:

    ...
    INFO 2010-10-26 22:45:32 MarkDuplicates Read 40000000 records. Tracking 2669417 as yet unmatched pairs. 69542 records in RAM. Last sequence index: 2
    INFO 2010-10-26 22:45:47 MarkDuplicates Read 41000000 records. Tracking 2725267 as yet unmatched pairs. 47482 records in RAM. Last sequence index: 2
    INFO 2010-10-26 22:46:15 MarkDuplicates Read 42000000 records. Tracking 2779116 as yet unmatched pairs. 25059 records in RAM. Last sequence index: 2
    [Tue Oct 26 22:46:36 CDT 2010] net.sf.picard.sam.MarkDuplicates done.
    Runtime.totalMemory()=778698752
    Exception in thread "main" net.sf.picard.PicardException: Exception writing ReadEnds to file.
    at net.sf.picard.sam.ReadEndsCodec.encode(ReadEndsCodec.java:74)
    at net.sf.picard.sam.ReadEndsCodec.encode(ReadEndsCodec.java:32)
    at net.sf.samtools.util.SortingCollection.spillToDisk(SortingCollection.java:185)
    at net.sf.samtools.util.SortingCollection.add(SortingCollection.java:140)
    at net.sf.picard.sam.MarkDuplicates.buildSortedReadEndLists(MarkDuplicates.java:305)
    at net.sf.picard.sam.MarkDuplicates.doWork(MarkDuplicates.java:109)
    at net.sf.picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:150)
    at net.sf.picard.sam.MarkDuplicates.main(MarkDuplicates.java:93)
    Caused by: java.io.IOException: No space left on device
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:260)
    at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
    at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
    at java.io.DataOutputStream.flush(DataOutputStream.java:106)
    at net.sf.picard.sam.ReadEndsCodec.encode(ReadEndsCodec.java:71)
    ... 7 more
    ...

    Has anyone ever gotten this writing read ends to file error?

  • #2
    Caused by: java.io.IOException: No space left on device
    No more space in the hard drive ?
    Francois Sabot, PhD

    Be realistic. Demand the Impossible.
    www.wikiposon.org

    Comment


    • #3
      I thought that was what it would be, but is this a simple matter of modifying MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP to greater than the number of mapped reads i have in the .bam file?

      Comment


      • #4
        Originally posted by JohnK View Post
        I thought that was what it would be, but is this a simple matter of modifying MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP to greater than the number of mapped reads i have in the .bam file?
        No. Your problem is *writing* to the disk. Not reading from it.

        Comment


        • #5
          I would check whether or not your drive is full, write-protected and/or that you have permissions to write to the drive/directory in question. It mostly looks like the drive is full and you'd have to delete files off your drive.

          Comment


          • #6
            Thanks, ladies/guys. My admin-dude found the issue for me and it was along the lines of what you were both saying in a way. I posted the reason on another thread.

            Comment


            • #7
              Picard Exception

              Any ideas for why it can't create a ReadEnds File?
              Code:
              Runtime.totalMemory()=166526976
              Exception in thread "main" net.sf.picard.PicardException: Error creating temporary ReadEnds file
              	at net.sf.picard.sam.DiskReadEndsMap.getOutputStreamForSequence(DiskReadEndsMap.java:175)
              	at net.sf.picard.sam.DiskReadEndsMap.put(DiskReadEndsMap.java:147)
              	at net.sf.picard.sam.MarkDuplicates.buildSortedReadEndLists(MarkDuplicates.java:278)
              	at net.sf.picard.sam.MarkDuplicates.doWork(MarkDuplicates.java:109)
              	at net.sf.picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:160)
              	at net.sf.picard.sam.MarkDuplicates.main(MarkDuplicates.java:93)
              Caused by: java.io.FileNotFoundException: /media/1B3B8E0F52CC3359/mRNAsnp/DREM.4603601404094682558.tmp/32279.read_ends (Too many open files)
              	at java.io.FileOutputStream.open(Native Method)
              	at java.io.FileOutputStream.<init>(FileOutputStream.java:209)
              	at java.io.FileOutputStream.<init>(FileOutputStream.java:160)
              	at net.sf.picard.sam.DiskReadEndsMap.getOutputStreamForSequence(DiskReadEndsMap.java:170)
              	... 5 more
              it says there are too many open files, but I should have plenty
              Code:
               cat /proc/sys/fs/file-max
              197316

              Comment


              • #8
                From http://lj4newbies.blogspot.com/2007/...pen-files.html (first google results for Java Too Many open Files)

                This is because too many file descriptors're opened by tomcat. File descriptor can be limited in both system level and shell level.

                To check maximum number of fd in system type 'cat /proc/sys/fs/file-max'. In my case it is 65536(someone said it should set to 200000). Tomcat error when try to open socket number 272 so I think 65536 is ok for me for now. Anyway if u want to set it add 'fs.file-max = 200000' to /etc/sysctl.conf
                So, the descriptors seem to be too numerous
                Francois Sabot, PhD

                Be realistic. Demand the Impossible.
                www.wikiposon.org

                Comment


                • #9
                  Your friendly sysadmin will help you here. 200k may be not enough FDs. It depens on your environment. Are you sharing the box among many users?
                  -drd

                  Comment


                  • #10
                    It's just me.

                    Comment


                    • #11
                      Thank you all!

                      But it seems i dont have the right to change the /etc/sysctl.conf, any other solutions?

                      thanks.

                      Comment


                      • #12
                        Hi everyone,

                        I have the same problem when trying to run SamSort.jar for a 33GB BAM, I asked my sys-admin to expand the file-max limit to 8,000,000 and.. I still got the same error. So there maybe something else...

                        Thanks.

                        Comment


                        • #13
                          tatinhawk: Given that this thread is around 9 months old and has talked about at least two different type of errors, what "same error" are you referring to?

                          Comment


                          • #14
                            Originally posted by tatinhawk View Post
                            Hi everyone,

                            I have the same problem when trying to run SamSort.jar for a 33GB BAM, I asked my sys-admin to expand the file-max limit to 8,000,000 and.. I still got the same error. So there maybe something else...

                            Thanks.
                            sorry ... I forgot to mention that the error I have is the " Exception in thread "main" net.sf.samtools.util.RuntimeIOException: java.io.FileNotFoundException: /temp/sortingcollection.1760628862445939889.tmp (Too many open files)"

                            Comment


                            • #15
                              On my university cluster the max number of open files is set to 1024, and I was getting the same error reported above. Setting the following picard flag to something less than that max seems to resolve the problem.

                              MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=[some number lower than the output of `ulimit -n`]

                              Comment

                              Latest Articles

                              Collapse

                              • seqadmin
                                Exploring the Dynamics of the Tumor Microenvironment
                                by seqadmin




                                The complexity of cancer is clearly demonstrated in the diverse ecosystem of the tumor microenvironment (TME). The TME is made up of numerous cell types and its development begins with the changes that happen during oncogenesis. “Genomic mutations, copy number changes, epigenetic alterations, and alternative gene expression occur to varying degrees within the affected tumor cells,” explained Andrea O’Hara, Ph.D., Strategic Technical Specialist at Azenta. “As...
                                07-08-2024, 03:19 PM
                              • seqadmin
                                Exploring Human Diversity Through Large-Scale Omics
                                by seqadmin


                                In 2003, researchers from the Human Genome Project (HGP) announced the most comprehensive genome to date1. Although the genome wasn’t fully completed until nearly 20 years later2, numerous large-scale projects, such as the International HapMap Project and 1000 Genomes Project, continued the HGP's work, capturing extensive variation and genomic diversity within humans. Recently, newer initiatives have significantly increased in scale and expanded beyond genomics, offering a more detailed...
                                06-25-2024, 06:43 AM

                              ad_right_rmr

                              Collapse

                              News

                              Collapse

                              Topics Statistics Last Post
                              Started by seqadmin, Yesterday, 05:49 AM
                              0 responses
                              15 views
                              0 likes
                              Last Post seqadmin  
                              Started by seqadmin, 07-15-2024, 06:53 AM
                              0 responses
                              26 views
                              0 likes
                              Last Post seqadmin  
                              Started by seqadmin, 07-10-2024, 07:30 AM
                              0 responses
                              37 views
                              0 likes
                              Last Post seqadmin  
                              Started by seqadmin, 07-03-2024, 09:45 AM
                              0 responses
                              204 views
                              0 likes
                              Last Post seqadmin  
                              Working...
                              X