I am mad because of samtools sort command

[Lv Ray]

I got .sam files from Bowtie2.
now I want to merge these two files.First ,I run:
samtools view -bSh ERR1.sam >ERR1.bam
samtools view -bSh ERR2.sam >ERR2.bam

and,I got the bam file.(they should have the head)
However,I run the next:
samtools sort ERR1.bam ERR1.sorted.bam (here,I got the sorted file,lucky)
samtools sort ERR2.bam ERR2.sorted.bam
about the ERR2.bam, I didn't get the sorted file, this was the output:

[bam_header_read] invalid BAM binary header (this is not a BAM file).
[bam_sort_core] truncated file. Continue anyway.
Segmentation fault (core dumped)

why?Just because the ERR2.sam is too big(about 66G)?

[Lv Ray]

supplement:
I run command: samtools view ERR2.bam |less -S
and I got this:
[bam_header_read] invalid BAM binary header (this is not a BAM file).
[main_samview] fail to read the header from "ERR173170_paired.bam".

[biocomputer]

If the file is too big for sorting you could split the .sam file on chromosome, sort each, recombine, then convert to .bam.

[Lv Ray]

If the file is too big for sorting you could split the .sam file on chromosome, sort each, recombine, then convert to .bam.

You mean that i got the fault in producing the sam file?but my sam file is okay!

[Lv Ray]

like your mean, maybe I need to split my big sam file

[dpryan]

66 gigs isn't too big to sort, the original BAM file was corrupt, likely due to running out of space or a hardware problem. Make sure you have enough space and then remake the BAM file.

[Lv Ray]

The space is enough, how about the memory?

[dpryan]

The whole SAM file isn't loaded into memory, it's processed line by line (and compressed in blocks).

[dpryan]

A transient hardware error is the most likely cause of this sort of thing.

[arundurvasula]

I recently got an error like that because I switched my reads with my reference sequence while mapping. I.e. my alignment was of my reference to my reads. Maybe that's your problem?

Here's a (correct) bash function that I used to map reads to a reference and only grab the mapped reads from the sam. Hopefully this can help guide you:

map () {
	bwa index -a bwtsw $refseq
	bwa bwasw $refseq ../temp/$1/sampled_reads.fasta > ../temp/$1/alignment.sam
	samtools view -bS -F 4 ../temp/$1/alignment.sam > ../temp/$1/mapped.alignment.bam
	samtools sort ../temp/$1/mapped.alignment.bam ../results/$1/mapped.sorted.alignment
	samtools index ../results/$1/mapped.sorted.alignment.bam
}

[Lv Ray]

Thank you a lot for sparing your beautiful bash.
However ,it seems that your bash is suitable for unpaired alignment. I thought that because the command samtools view -bS -F 4 ../temp/$1/alignment.sam > ../temp/$1/mapped.alignment.bam ,you discard the unmapped reads. But how to set the parameter -F In paired alignment reads (.sam) ?
Thanks all .

[dpryan]

-F 4 will remove unmapped reads in either case. If you want to remove those reads with an unmapped mate then just filter according to that bit in the flag.

[Lv Ray]

About the parameter -F of samtools view

-F 4 will remove unmapped reads in either case. If you want to remove those reads with an unmapped mate then just filter according to that bit in the flag.

Hi, dpryan. Thank you for your answer. Now I have the similar quetions about -F, . I appreciate and hope you can help me.
1) Should I remove the unmapped reads (but its mates mapped) or the unmapped mates(but its reads mapped)
2) about the paired reads, if I remove all them above ,should I use -F 12? However , it seems that there's no the value of 12. How about the 77 or 141.

[dpryan]

1) It depends on what you want to do with the results.
2) -F 12 is correct. There don't have to be any flags with that value since this is a bit comparison.

[Lv Ray]

Thank you, dpryan.
Today I tested it ,and the result is Consistent with your answer！
Thanks, again.