samtools sort [bam_merge_core] [bam_sort_core] errors?

[nimrod337]

Okay, I've run out of ideas. I cannot get this BAM file to sort. I have a reference sequence in FASTA format that has 2 'Chromosomes' that are actually just scaffolds from a genome that isn't finished yet. The first time I ran my illumina reads through the workflow (BWA>SAM>BAM>sort>index... etc) Then when I tried to view it, it failed because there were replications in the BAM headers. I discovered that these were my fault from the python script I wrote to pull out the FASTA scaffolds that matched the gene I was looking for in the pile of scaffolds. Anyway: I fixed the script, and got the new reference, that is the same as the old one, but without the repetitive '>header' accidentally added 11 times. I ran BWA, no problem. I converted the SAM to BAM with samtools, no problem. But now I am trying to sort and index the BAM so I can look at it an whatnot, and the command I am running is:

samtools sort M6scaff_BWA.bam M6scaff_BWA.bam.sorted

the error I get is:

[bam_sort_core] merging from 142 files...
[W::bam_hdr_read] EOF marker is absent. The input is probably truncated.
[E::hts_open] fail to open file 'M6scaff_BWA.sorted.0118.bam'
[bam_merge_core] fail to open file M6scaff_BWA.sorted.0118.bam

I have looked around the interwebs for a solution, and most of the people that have this error are using the -f argument in samtool sort, and when they remove it, it's fixed. I am not using the -f argument, so I have no recourse. Can someone please suggest something to try? I am banging my head against a wall for a whole weekend now, and I just don't know what to do.

Thanks so much

[GenoMax]

Does the header on your BAM look right (when you do samtools view on it)? Are you running out of resources anywhere (/tmp, disk, memory etc) in this process? What is the size of the BAM file? What version of samtools are you using?

[nimrod337]

samtools view -H looks like this:

@SQ SN:scaffold_829 LN:219473
@SQ SN:scaffold_4279 LN:55059
@PG ID:bwa PN:bwa VN:0.7.5a-r405

(the scaffold_###) is the fasta header in my reference

How could I check if I am running out of resources? Does anything like that get printed in the log file somewhere? I re-ran the sort with -m 8GB to ensure that it wouldn't run out of RAM (16 on the machine) but I got this error:

[E::hts_open] fail to open file 'M6scaff_BWA.bam.sorted.0001.bam'

The BAM is 25G

samtools v 1.2

Thanks for the help. Anything helps at this point.

Edit: I can see the temporary M6scaff_BWA.bam.sorted.####.bam files in the folder, Why then, would it have trouble opening it?

[GenoMax]

That header looks good. Is it possible for you to try the older version of samtools (0.1.19) for the sort? If you are certain ample disk space is available then this may be worth a try.

[nimrod337]

Unfortunately, I started on the older samtools, and upgraded to 1.2 to see if that fixed the problem. It did not. I have cleared up some disk space, and rebooted to clear the RAM. Cross your fingers!!