[qh2012]

Hi Cliff. Did you find a solution to your problem? I'm getting the same error.

[jflowers]

Regarding the java.io.FileNotFoundException: (Too many open files), there are a number of things to do. Picard sorting seems to open so many files that it can overload default limits on unix systems. To deal with this, you can either (1) ask you sys admin to increase the allowable number of open files using ulimit -n or (2) for some picard programs you can increase the value of the MAX_RECORDS_IN_RAM command-line parameter which instructs Picard to store more records in fewer files and reduce the number of open files. (watch out for increased memory usage though). If your using markdups and see this error you can 3) use the command-line para
meter MAX_FILE_HANDLES_FOR_READ_ENDS_MAP. By reducing this number, you reduce the number of concurrently open files.

On our hpc, ulimit -n == 4096, so I use MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=4000

[rahilsethi]

I did not use bwa. I used SHRiMP 2.2.2 for mapping. samtools 1.8 for converting sam to bam and Picard version: 1.74 for combining mapped bam files.

[MolecularToast]

If anyone is still listening to this thread - what versions of bwa is everyone using who is getting this error?

[rahilsethi]

Picard java exception

Hi I got similar error using Picard MergeSamFiles.jar
Wondering if anyone found an answer for it

Exception in thread "main" net.sf.samtools.util.RuntimeIOException: java.io.FileNotFoundException: /scratch/temp/sortingcollection.4193001834577553277.tmp (Too many open files)
at net.sf.samtools.util.SortingCollection$FileRecordIterator.<init>(SortingCollection.java:445)
at net.sf.samtools.util.SortingCollection$MergingIterator.<init>(SortingCollection.java:384)
at net.sf.samtools.util.SortingCollection.iterator(SortingCollection.java:254)
at net.sf.samtools.util.SortingCollection.iterator(SortingCollection.java:43)
at net.sf.samtools.SAMFileWriterImpl.close(SAMFileWriterImpl.java:190)
at net.sf.samtools.AsyncSAMFileWriter.synchronouslyClose(AsyncSAMFileWriter.java:42)
at net.sf.samtools.util.AbstractAsyncWriter.close(AbstractAsyncWriter.java:78)
at net.sf.picard.sam.MergeSamFiles.doWork(MergeSamFiles.java:154)
at net.sf.picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:177)
at net.sf.picard.sam.MergeSamFiles.main(MergeSamFiles.java:79)
Caused by: java.io.FileNotFoundException: /scratch/temp/sortingcollection.4193001834577553277.tmp (Too many open files)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(Unknown Source)
at net.sf.samtools.util.SortingCollection$FileRecordIterator.<init>(SortingCollection.java:439)
... 9 more
Even in my case the space available in /scratch is around 40T

[Jeremy37]

Similar error found when using SortSam

Hi Cliff. Did you find a solution to your problem? I'm getting the same error, except for me it occurs when calling SortSam. Similar to your case we have lots of space available (62 TB).

java -Xmx20g -Xms8g -jar SortSam.jar MAX_RECORDS_IN_RAM=2000000 VALIDATION_STRINGENCY=LENIENT CREATE_INDEX=true SORT_ORDER=coordinate INPUT=sequence.sam OUTPUT=sequence.sorted.bam


Runtime.totalMemory()=17176592384
Exception in thread "main" net.sf.samtools.util.RuntimeIOException: Write error; BinaryCodec in writemode; streamed file (filename not available)
at net.sf.samtools.util.BinaryCodec.writeBytes(BinaryCodec.java:199)
at net.sf.samtools.util.BinaryCodec.writeBytes(BinaryCodec.java:189)
at net.sf.samtools.util.BinaryCodec.writeString(BinaryCodec.java:285)
at net.sf.samtools.BinaryTagCodec.writeTag(BinaryTagCodec.java:168)
at net.sf.samtools.BAMRecordCodec.encode(BAMRecordCodec.java:144)
at net.sf.samtools.BAMRecordCodec.encode(BAMRecordCodec.java:37)
at net.sf.samtools.util.SortingCollection.spillToDisk(SortingCollection.java:201)
at net.sf.samtools.util.SortingCollection.add(SortingCollection.java:140)
at net.sf.samtools.SAMFileWriterImpl.addAlignment(SAMFileWriterImpl.java:157)
at net.sf.picard.sam.SortSam.doWork(SortSam.java:67)
at net.sf.picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:157)
at net.sf.picard.cmdline.CommandLineProgram.instanceMainWithExit(CommandLineProgram.java:117)
at net.sf.picard.sam.SortSam.main(SortSam.java:79)
Caused by: java.io.IOException: No space left on device
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:297)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
at net.sf.samtools.util.BinaryCodec.writeBytes(BinaryCodec.java:197)
... 12 more

Hi, Nirav99

Thanks! I tried your command and still failed..

[cliff]

Hi, Nirav99

Thanks! I tried your command and still failed..

[nirav99]

Hi Cliff,

TMP_DIR is a command line parameter for Picard tools.

So, the way to do it would be

java -Xmx4G -jar FixMateInformation.jar I=input.bam O=output.bam TMP_DIR=/home/temp VALIDATION_STRINGENCY=LENIENT

[cliff]

I specified a temporary directory which has enough space by

java -Xmx4g -Djava.io.tmpdir=/home/temp -jar

it still failed...

[drio]

The scratching happens in /tmp/<user_name>, so the problem was most likely /tmp. Use TMP_DIR to specify a different temporary dir.

[nilshomer]

Could it be that the temporary path is not pointing towards your 15TB?

[cliff]

thanks for getting back to me. I actually have 15TB left..

[nirav99]

Hi Cliff,

Please check the disk space where you are running this. The exception indicates

"Caused by: java.io.IOException: No space left on device"

[cliff]

hi, nirav99

I got the same error "Mate negative strand flag does not match read negative strand flag of mate" and I tried FixMateInformation as:

java -Xmx2g -jar FixMateInformation.jar INPUT=test.bam OUTPUT=test_fixed.bam VALIDATION_STRINGENCY=SILENT

but I didn't get the output file and got this error:
Exception in thread "main" net.sf.samtools.util.RuntimeIOException: Write error; BinaryCodec in writemode; streamed file (filename not available)
at net.sf.samtools.util.BinaryCodec.writeBytes(BinaryCodec.java:199)
at net.sf.samtools.util.BinaryCodec.writeBytes(BinaryCodec.java:189)
at net.sf.samtools.BAMRecordCodec.encode(BAMRecordCodec.java:120)
at net.sf.samtools.BAMRecordCodec.encode(BAMRecordCodec.java:37)
at net.sf.samtools.util.SortingCollection.spillToDisk(SortingCollection.java:185)
at net.sf.samtools.util.SortingCollection.add(SortingCollection.java:140)
at net.sf.picard.sam.FixMateInformation.doWork(FixMateInformation.java:145)
at net.sf.picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:156)
at net.sf.picard.cmdline.CommandLineProgram.instanceMainWithExit(CommandLineProgram.java:117)
at net.sf.picard.sam.FixMateInformation.main(FixMateInformation.java:74)
Caused by: java.io.IOException: No space left on device
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:260)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109)
at net.sf.samtools.util.BinaryCodec.writeBytes(BinaryCodec.java:197)


....

[nirav99]

Picard's command line utility FixMateInformation fixes this error.