what if samtools sort failed while merging from temp files..

[acoada]

I was sorting a hug BAM file using

samtools sort -@ 5 -m 4G  <in.bam> <out.prefix>

While merging from temp files

 [bam_sort_core] merging from 630 files...

it was killed 'cos lack of memory...

So I checked the temp files and found that I could not simply use samtools merge to merge those temp files 'cos they were not in order

What shall I do if I just don't wanna start over again...*

[Brian Bushnell]

FYI, the "-@ 5" and "-m 4G" specify 4GB per thread for 5 threads, so 20GB total. You need to reduce the number of threads or the memory limit if you were trying to stay under 4GB.

[acoada]

FYI, the "-@ 5" and "-m 4G" specify 4GB per thread for 5 threads, so 20GB total. You need to reduce the number of threads or the memory limit if you were trying to stay under 4GB.

Yep, the total memory should under 20Gb, yet it got 25Gb while merging, so it was killed.

I just wanna know is there any way to tell samtools to redo the merge( just the merge), 'cos the temp files are already sorted in some way( fast sort? I'm not sure) and I don't want to waste the time to redo the sort.

Thank you~

[Brian Bushnell]

No, it probably died sorting the temp files. The split and merge parts are both low-memory, only the sort is high-memory.

[acoada]

No, it probably died sorting the temp files. The split and merge parts are both low-memory, only the sort is high-memory.

samtools has told me that

  [bam_sort_core] merging from 630 files...

thus I am sure the all the temp files have been sorted and
samtools was killed while doing merging work ...

The point is, is there any way to redo the merging only, 'cos sorting was really time consuming...

[Brian Bushnell]

I imagine that the merge and sort are done simultaneously. Anyway - if you want results you can trust, of course you can't use temp files from a program that crashed with a nondeterministic out-of-memory error; they could be in any condition.

Note - I could be wrong; I can't remember the order of samtools' sort messages since I always ignore them. But there's no reason for it to need 25GB RAM while merging unless it is sorting at the same time, or writes are being internally buffered; and in either case, the temp files could be corrupt.

[acoada]

I imagine that the merge and sort are done simultaneously. Anyway - if you want results you can trust, of course you can't use temp files from a program that crashed with a nondeterministic out-of-memory error; they could be in any condition.

Note - I could be wrong; I can't remember the order of samtools' sort messages since I always ignore them. But there's no reason for it to need 25GB RAM while merging unless it is sorting at the same time, or writes are being internally buffered; and in either case, the temp files could be corrupt.

You're right. Samtools do sort while merging temp files, because there isn't ordinal order between temp files( each temp file contains all the chromosome... ) .