Suggested update to SAM format specification for unmapped reads

[andrewm]

I would like to suggest a minor but important update/clarification for the SAM format specification regarding unmapped reads.

In the sam spec http://samtools.github.io/hts-specs/SAMv1.pdf, it says the following about the flag:

0x10 SEQ being reverse complemented

and further down in the notes:

Bit 0x4 is the only reliable place to tell whether the read is unmapped. If 0x4 is set, no assumptions can be made about RNAME, POS, CIGAR, MAPQ, bits 0x2, 0x10, 0x100 and 0x800, and the bit 0x20 of the previous read in the template.

Thus for an unmapped read, we dont know if the reverse complement flag is valid. This has important implications for extracting the original read data from a bam file, since we dont know whether unmapped reads are represented as-is in the bam, or have been reverse complemented. It would be fairly simple to add the requirement that 0x10 is always valid, regardless of 0x4.

If I understand the bwa code correctly, the reverse complement flag is always set for a reverse complemented read (though confusingly, that flag takes on the same value of its mate if its mate is mapped, reverse complementing the read unnecessarily). If we updated the spec, bwa would already be within spec. I am not sure about other aligners.

In the communities I work with bam files are ubiquitous for storing sequencing data, and with good reason since they are smaller than compressed fastqs. For people relying on bams as a storage format, and wishing to use unmapped reads to predict, for instance, structural variants, it is important the bams allows for consistent extraction of the read sequences as they came off the sequencer.

Posting this to get feedback from the community, and hopefully change the spec. if I have misunderstood the spec, let me now.

Andrew

[dpryan]

You'd be better off posting this on the samtool-devel email list. Very few people who read SEQanswers have any influence on the SAM spec.

I suspect that the impetus for the "0x4 set means ignore most other things" footnote is to provide flexibility to aligners (e.g., you might find the best alignment in one step and then mark things as unmapped in a different one). While I like where you're coming from with the proposal, I fear that any change that will make current standards-conformant applications standards-nonconformant won't be viewed as well as desired.

[Brian Bushnell]

It would make a lot more sense if the standard simply didn't ever reorient the bases or qualities. There are a lot of non-ideal decisions woven into the SAM spec, but fortunately it is extensible, so you can have a custom tag indicating whether the sequence was reverse-complemented, for example. In my view, since the spec states that 0x10 is irrelevant if 0x4 is set, a read should never be presented as a reverse-complement when 0x4 is set, because:
A) There is no reason to do so;
and
B) There is no way to indicate the orientation in that circumstance.

Thus I don't think that reverse-complementing unmapped reads is currently correct behavior regardless of the 0x10 bit, and changing the spec would yield not only a lot of noncompliant software, but also a lot of noncompliant existing sam/bam files, which defeats the purpose of having a standard.

Also, I don't think bam is a good format to use for storing raw fastqs, even if the compression is slightly better than gzipped fastq; it makes actually using the data much more difficult.

[dpryan]

I disagree in part. It can be quite convenient to have everything oriented 5'->3' on the + strand. This allows for simple implementation of pileups, for example.

[andrewm]

It would make a lot more sense if the standard simply didn't ever reorient the bases or qualities.

Agreed. It requires very little computational effort to reorient the reads during pileup generation. It also takes very little effort to reorient during fastq recovery, providing the flag is correct.

I don't think bam is a good format to use for storing raw fastqs, even if the compression is slightly better than gzipped fastq; it makes actually using the data much more difficult.

You may be correct, but many sequencing centres do this anyway.

I fear that any change that will make current standards-conformant applications standards-nonconformant won't be viewed as well as desired.

changing the spec would yield not only a lot of noncompliant software, but also a lot of noncompliant existing sam/bam files, which defeats the purpose of having a standard.

The purpose of having a standard is to enforce consistency for the benefit of the community of users. If it is to our benefit to be able to consistently extract reads from a bam, we should change the standard so that is possible.

True, existing bams would be non-conformant, but their non-conformance could easily be identified by looking at the software version used to generate them, which is usually stored in the header.

I will try samtool-devel email list as suggested.

[maubp]

samtools-devel thread http://sourceforge.net/p/samtools/ma...sage/33023739/