Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • qh2012
    replied
    Hi Cliff. Did you find a solution to your problem? I'm getting the same error.

    Leave a comment:


  • jflowers
    replied
    Regarding the java.io.FileNotFoundException: (Too many open files), there are a number of things to do. Picard sorting seems to open so many files that it can overload default limits on unix systems. To deal with this, you can either (1) ask you sys admin to increase the allowable number of open files using ulimit -n or (2) for some picard programs you can increase the value of the MAX_RECORDS_IN_RAM command-line parameter which instructs Picard to store more records in fewer files and reduce the number of open files. (watch out for increased memory usage though). If your using markdups and see this error you can 3) use the command-line para
    meter MAX_FILE_HANDLES_FOR_READ_ENDS_MAP. By reducing this number, you reduce the number of concurrently open files.

    On our hpc, ulimit -n == 4096, so I use MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=4000

    Leave a comment:


  • rahilsethi
    replied
    I did not use bwa. I used SHRiMP 2.2.2 for mapping. samtools 1.8 for converting sam to bam and Picard version: 1.74 for combining mapped bam files.

    Leave a comment:


  • MolecularToast
    replied
    If anyone is still listening to this thread - what versions of bwa is everyone using who is getting this error?

    Leave a comment:


  • rahilsethi
    replied
    Picard java exception

    Hi I got similar error using Picard MergeSamFiles.jar
    Wondering if anyone found an answer for it

    Exception in thread "main" net.sf.samtools.util.RuntimeIOException: java.io.FileNotFoundException: /scratch/temp/sortingcollection.4193001834577553277.tmp (Too many open files)
    at net.sf.samtools.util.SortingCollection$FileRecordIterator.<init>(SortingCollection.java:445)
    at net.sf.samtools.util.SortingCollection$MergingIterator.<init>(SortingCollection.java:384)
    at net.sf.samtools.util.SortingCollection.iterator(SortingCollection.java:254)
    at net.sf.samtools.util.SortingCollection.iterator(SortingCollection.java:43)
    at net.sf.samtools.SAMFileWriterImpl.close(SAMFileWriterImpl.java:190)
    at net.sf.samtools.AsyncSAMFileWriter.synchronouslyClose(AsyncSAMFileWriter.java:42)
    at net.sf.samtools.util.AbstractAsyncWriter.close(AbstractAsyncWriter.java:78)
    at net.sf.picard.sam.MergeSamFiles.doWork(MergeSamFiles.java:154)
    at net.sf.picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:177)
    at net.sf.picard.sam.MergeSamFiles.main(MergeSamFiles.java:79)
    Caused by: java.io.FileNotFoundException: /scratch/temp/sortingcollection.4193001834577553277.tmp (Too many open files)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(Unknown Source)
    at net.sf.samtools.util.SortingCollection$FileRecordIterator.<init>(SortingCollection.java:439)
    ... 9 more
    Even in my case the space available in /scratch is around 40T

    Leave a comment:


  • Jeremy37
    replied
    Similar error found when using SortSam

    Hi Cliff. Did you find a solution to your problem? I'm getting the same error, except for me it occurs when calling SortSam. Similar to your case we have lots of space available (62 TB).

    java -Xmx20g -Xms8g -jar SortSam.jar MAX_RECORDS_IN_RAM=2000000 VALIDATION_STRINGENCY=LENIENT CREATE_INDEX=true SORT_ORDER=coordinate INPUT=sequence.sam OUTPUT=sequence.sorted.bam


    Runtime.totalMemory()=17176592384
    Exception in thread "main" net.sf.samtools.util.RuntimeIOException: Write error; BinaryCodec in writemode; streamed file (filename not available)
    at net.sf.samtools.util.BinaryCodec.writeBytes(BinaryCodec.java:199)
    at net.sf.samtools.util.BinaryCodec.writeBytes(BinaryCodec.java:189)
    at net.sf.samtools.util.BinaryCodec.writeString(BinaryCodec.java:285)
    at net.sf.samtools.BinaryTagCodec.writeTag(BinaryTagCodec.java:168)
    at net.sf.samtools.BAMRecordCodec.encode(BAMRecordCodec.java:144)
    at net.sf.samtools.BAMRecordCodec.encode(BAMRecordCodec.java:37)
    at net.sf.samtools.util.SortingCollection.spillToDisk(SortingCollection.java:201)
    at net.sf.samtools.util.SortingCollection.add(SortingCollection.java:140)
    at net.sf.samtools.SAMFileWriterImpl.addAlignment(SAMFileWriterImpl.java:157)
    at net.sf.picard.sam.SortSam.doWork(SortSam.java:67)
    at net.sf.picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:157)
    at net.sf.picard.cmdline.CommandLineProgram.instanceMainWithExit(CommandLineProgram.java:117)
    at net.sf.picard.sam.SortSam.main(SortSam.java:79)
    Caused by: java.io.IOException: No space left on device
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:297)
    at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
    at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
    at net.sf.samtools.util.BinaryCodec.writeBytes(BinaryCodec.java:197)
    ... 12 more



    Originally posted by cliff View Post
    Hi, Nirav99

    Thanks! I tried your command and still failed..

    Leave a comment:


  • cliff
    replied
    Hi, Nirav99

    Thanks! I tried your command and still failed..

    Leave a comment:


  • nirav99
    replied
    Hi Cliff,

    TMP_DIR is a command line parameter for Picard tools.

    So, the way to do it would be

    java -Xmx4G -jar FixMateInformation.jar I=input.bam O=output.bam TMP_DIR=/home/temp VALIDATION_STRINGENCY=LENIENT

    Leave a comment:


  • cliff
    replied
    I specified a temporary directory which has enough space by

    java -Xmx4g -Djava.io.tmpdir=/home/temp -jar

    it still failed...

    Leave a comment:


  • drio
    replied
    The scratching happens in /tmp/<user_name>, so the problem was most likely /tmp. Use TMP_DIR to specify a different temporary dir.

    Leave a comment:


  • nilshomer
    replied
    Could it be that the temporary path is not pointing towards your 15TB?

    Leave a comment:


  • cliff
    replied
    thanks for getting back to me. I actually have 15TB left..

    Leave a comment:


  • nirav99
    replied
    Hi Cliff,

    Please check the disk space where you are running this. The exception indicates

    "Caused by: java.io.IOException: No space left on device"

    Leave a comment:


  • cliff
    replied
    hi, nirav99

    I got the same error "Mate negative strand flag does not match read negative strand flag of mate" and I tried FixMateInformation as:

    java -Xmx2g -jar FixMateInformation.jar INPUT=test.bam OUTPUT=test_fixed.bam VALIDATION_STRINGENCY=SILENT

    but I didn't get the output file and got this error:
    Exception in thread "main" net.sf.samtools.util.RuntimeIOException: Write error; BinaryCodec in writemode; streamed file (filename not available)
    at net.sf.samtools.util.BinaryCodec.writeBytes(BinaryCodec.java:199)
    at net.sf.samtools.util.BinaryCodec.writeBytes(BinaryCodec.java:189)
    at net.sf.samtools.BAMRecordCodec.encode(BAMRecordCodec.java:120)
    at net.sf.samtools.BAMRecordCodec.encode(BAMRecordCodec.java:37)
    at net.sf.samtools.util.SortingCollection.spillToDisk(SortingCollection.java:185)
    at net.sf.samtools.util.SortingCollection.add(SortingCollection.java:140)
    at net.sf.picard.sam.FixMateInformation.doWork(FixMateInformation.java:145)
    at net.sf.picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:156)
    at net.sf.picard.cmdline.CommandLineProgram.instanceMainWithExit(CommandLineProgram.java:117)
    at net.sf.picard.sam.FixMateInformation.main(FixMateInformation.java:74)
    Caused by: java.io.IOException: No space left on device
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:260)
    at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
    at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
    at net.sf.samtools.util.BinaryCodec.writeBytes(BinaryCodec.java:197)


    ....

    Leave a comment:


  • nirav99
    replied
    Picard's command line utility FixMateInformation fixes this error.

    Leave a comment:

Latest Articles

Collapse

  • seqadmin
    Current Approaches to Protein Sequencing
    by seqadmin


    Proteins are often described as the workhorses of the cell, and identifying their sequences is key to understanding their role in biological processes and disease. Currently, the most common technique used to determine protein sequences is mass spectrometry. While still a valuable tool, mass spectrometry faces several limitations and requires a highly experienced scientist familiar with the equipment to operate it. Additionally, other proteomic methods, like affinity assays, are constrained...
    04-04-2024, 04:25 PM
  • seqadmin
    Strategies for Sequencing Challenging Samples
    by seqadmin


    Despite advancements in sequencing platforms and related sample preparation technologies, certain sample types continue to present significant challenges that can compromise sequencing results. Pedro Echave, Senior Manager of the Global Business Segment at Revvity, explained that the success of a sequencing experiment ultimately depends on the amount and integrity of the nucleic acid template (RNA or DNA) obtained from a sample. “The better the quality of the nucleic acid isolated...
    03-22-2024, 06:39 AM

ad_right_rmr

Collapse

News

Collapse

Topics Statistics Last Post
Started by seqadmin, 04-11-2024, 12:08 PM
0 responses
32 views
0 likes
Last Post seqadmin  
Started by seqadmin, 04-10-2024, 10:19 PM
0 responses
37 views
0 likes
Last Post seqadmin  
Started by seqadmin, 04-10-2024, 09:21 AM
0 responses
31 views
0 likes
Last Post seqadmin  
Started by seqadmin, 04-04-2024, 09:00 AM
0 responses
53 views
0 likes
Last Post seqadmin  
Working...
X