Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Picard error

    Hi,

    I'm having trouble understanding some of the params with Picard- well not really understanding but I'm getting this error here:

    ...
    INFO 2010-10-26 22:45:32 MarkDuplicates Read 40000000 records. Tracking 2669417 as yet unmatched pairs. 69542 records in RAM. Last sequence index: 2
    INFO 2010-10-26 22:45:47 MarkDuplicates Read 41000000 records. Tracking 2725267 as yet unmatched pairs. 47482 records in RAM. Last sequence index: 2
    INFO 2010-10-26 22:46:15 MarkDuplicates Read 42000000 records. Tracking 2779116 as yet unmatched pairs. 25059 records in RAM. Last sequence index: 2
    [Tue Oct 26 22:46:36 CDT 2010] net.sf.picard.sam.MarkDuplicates done.
    Runtime.totalMemory()=778698752
    Exception in thread "main" net.sf.picard.PicardException: Exception writing ReadEnds to file.
    at net.sf.picard.sam.ReadEndsCodec.encode(ReadEndsCodec.java:74)
    at net.sf.picard.sam.ReadEndsCodec.encode(ReadEndsCodec.java:32)
    at net.sf.samtools.util.SortingCollection.spillToDisk(SortingCollection.java:185)
    at net.sf.samtools.util.SortingCollection.add(SortingCollection.java:140)
    at net.sf.picard.sam.MarkDuplicates.buildSortedReadEndLists(MarkDuplicates.java:305)
    at net.sf.picard.sam.MarkDuplicates.doWork(MarkDuplicates.java:109)
    at net.sf.picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:150)
    at net.sf.picard.sam.MarkDuplicates.main(MarkDuplicates.java:93)
    Caused by: java.io.IOException: No space left on device
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:260)
    at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
    at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
    at java.io.DataOutputStream.flush(DataOutputStream.java:106)
    at net.sf.picard.sam.ReadEndsCodec.encode(ReadEndsCodec.java:71)
    ... 7 more
    ...

    Has anyone ever gotten this writing read ends to file error?

  • #2
    Caused by: java.io.IOException: No space left on device
    No more space in the hard drive ?
    Francois Sabot, PhD

    Be realistic. Demand the Impossible.
    www.wikiposon.org

    Comment


    • #3
      I thought that was what it would be, but is this a simple matter of modifying MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP to greater than the number of mapped reads i have in the .bam file?

      Comment


      • #4
        Originally posted by JohnK View Post
        I thought that was what it would be, but is this a simple matter of modifying MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP to greater than the number of mapped reads i have in the .bam file?
        No. Your problem is *writing* to the disk. Not reading from it.

        Comment


        • #5
          I would check whether or not your drive is full, write-protected and/or that you have permissions to write to the drive/directory in question. It mostly looks like the drive is full and you'd have to delete files off your drive.

          Comment


          • #6
            Thanks, ladies/guys. My admin-dude found the issue for me and it was along the lines of what you were both saying in a way. I posted the reason on another thread.

            Comment


            • #7
              Picard Exception

              Any ideas for why it can't create a ReadEnds File?
              Code:
              Runtime.totalMemory()=166526976
              Exception in thread "main" net.sf.picard.PicardException: Error creating temporary ReadEnds file
              	at net.sf.picard.sam.DiskReadEndsMap.getOutputStreamForSequence(DiskReadEndsMap.java:175)
              	at net.sf.picard.sam.DiskReadEndsMap.put(DiskReadEndsMap.java:147)
              	at net.sf.picard.sam.MarkDuplicates.buildSortedReadEndLists(MarkDuplicates.java:278)
              	at net.sf.picard.sam.MarkDuplicates.doWork(MarkDuplicates.java:109)
              	at net.sf.picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:160)
              	at net.sf.picard.sam.MarkDuplicates.main(MarkDuplicates.java:93)
              Caused by: java.io.FileNotFoundException: /media/1B3B8E0F52CC3359/mRNAsnp/DREM.4603601404094682558.tmp/32279.read_ends (Too many open files)
              	at java.io.FileOutputStream.open(Native Method)
              	at java.io.FileOutputStream.<init>(FileOutputStream.java:209)
              	at java.io.FileOutputStream.<init>(FileOutputStream.java:160)
              	at net.sf.picard.sam.DiskReadEndsMap.getOutputStreamForSequence(DiskReadEndsMap.java:170)
              	... 5 more
              it says there are too many open files, but I should have plenty
              Code:
               cat /proc/sys/fs/file-max
              197316

              Comment


              • #8
                From http://lj4newbies.blogspot.com/2007/...pen-files.html (first google results for Java Too Many open Files)

                This is because too many file descriptors're opened by tomcat. File descriptor can be limited in both system level and shell level.

                To check maximum number of fd in system type 'cat /proc/sys/fs/file-max'. In my case it is 65536(someone said it should set to 200000). Tomcat error when try to open socket number 272 so I think 65536 is ok for me for now. Anyway if u want to set it add 'fs.file-max = 200000' to /etc/sysctl.conf
                So, the descriptors seem to be too numerous
                Francois Sabot, PhD

                Be realistic. Demand the Impossible.
                www.wikiposon.org

                Comment


                • #9
                  Your friendly sysadmin will help you here. 200k may be not enough FDs. It depens on your environment. Are you sharing the box among many users?
                  -drd

                  Comment


                  • #10
                    It's just me.

                    Comment


                    • #11
                      Thank you all!

                      But it seems i dont have the right to change the /etc/sysctl.conf, any other solutions?

                      thanks.

                      Comment


                      • #12
                        Hi everyone,

                        I have the same problem when trying to run SamSort.jar for a 33GB BAM, I asked my sys-admin to expand the file-max limit to 8,000,000 and.. I still got the same error. So there maybe something else...

                        Thanks.

                        Comment


                        • #13
                          tatinhawk: Given that this thread is around 9 months old and has talked about at least two different type of errors, what "same error" are you referring to?

                          Comment


                          • #14
                            Originally posted by tatinhawk View Post
                            Hi everyone,

                            I have the same problem when trying to run SamSort.jar for a 33GB BAM, I asked my sys-admin to expand the file-max limit to 8,000,000 and.. I still got the same error. So there maybe something else...

                            Thanks.
                            sorry ... I forgot to mention that the error I have is the " Exception in thread "main" net.sf.samtools.util.RuntimeIOException: java.io.FileNotFoundException: /temp/sortingcollection.1760628862445939889.tmp (Too many open files)"

                            Comment


                            • #15
                              On my university cluster the max number of open files is set to 1024, and I was getting the same error reported above. Setting the following picard flag to something less than that max seems to resolve the problem.

                              MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=[some number lower than the output of `ulimit -n`]

                              Comment

                              Latest Articles

                              Collapse

                              • seqadmin
                                Best Practices for Single-Cell Sequencing Analysis
                                by seqadmin



                                While isolating and preparing single cells for sequencing was historically the bottleneck, recent technological advancements have shifted the challenge to data analysis. This highlights the rapidly evolving nature of single-cell sequencing. The inherent complexity of single-cell analysis has intensified with the surge in data volume and the incorporation of diverse and more complex datasets. This article explores the challenges in analysis, examines common pitfalls, offers...
                                06-06-2024, 07:15 AM
                              • seqadmin
                                Latest Developments in Precision Medicine
                                by seqadmin



                                Technological advances have led to drastic improvements in the field of precision medicine, enabling more personalized approaches to treatment. This article explores four leading groups that are overcoming many of the challenges of genomic profiling and precision medicine through their innovative platforms and technologies.

                                Somatic Genomics
                                “We have such a tremendous amount of genetic diversity that exists within each of us, and not just between us as individuals,”...
                                05-24-2024, 01:16 PM

                              ad_right_rmr

                              Collapse

                              News

                              Collapse

                              Topics Statistics Last Post
                              Started by seqadmin, 06-14-2024, 07:24 AM
                              0 responses
                              12 views
                              0 likes
                              Last Post seqadmin  
                              Started by seqadmin, 06-13-2024, 08:58 AM
                              0 responses
                              13 views
                              0 likes
                              Last Post seqadmin  
                              Started by seqadmin, 06-12-2024, 02:20 PM
                              0 responses
                              17 views
                              0 likes
                              Last Post seqadmin  
                              Started by seqadmin, 06-07-2024, 06:58 AM
                              0 responses
                              185 views
                              0 likes
                              Last Post seqadmin  
                              Working...
                              X