Question on DNA Formatting

[AKatawazi]

Hello, this is my first post here and I am wondering if I could receive some assistance. I am on the ftp://ftp.ncbi.nih.gov/genomes/ ftp site and downloaded some genomes to review. I am trying to develop a more advanced compression algo for dna but noticed some interesting things about the files. Some letters are capped while others are not. There also seems to be a carriage return every so many characters. Is there a reason for this? Also I am assuming the large sections of N represent no data? I am trying to get the files as tiny as possible but am wondering if preserving caps or carriage returns is necessary for the tools that are being used. Thanks in advance.

[Brian Bushnell]

Sounds exciting!

Lower-case letters mean "masked", usually implying they are repetitive, or low-confidence; the exact meaning is application-specific. Typically, programs will either ignore lower-case letters (convert them to N) or make them upper-case and use them like all of the other upper-case letters. This information must be preserved; however, it's not relevant to most applications. An official genome of an organism is all upper-case; the ones with lower-case letters are processed in a specific way for some specific application.

Also, raw reads, which are much more interesting from a compression standpoint (since they amount to hundreds of thousands of times as much data as genomes) do not have lower-case letters. Ultimately, if you made a compression program that was case-insensitive, it would still be useful for that reason, though obviously less likely to catch on. I suggest you design it to handle all-upper-case ACGTN efficiently, and be capable of handling other things without regard to efficiency. Or have an option to convert lower-case to N, for example. There are also other degenerate bases to watch for.

However!

There are 2 other components - names and quality scores. Genomes normally don't have quality scores, but reads do (see the fastq format). And names can essentially contain anything other than newlines. So, compression of fasta files (only names and sequence) is dominated by the sequence, while compression of fastq files is usually dominated by qualities and names.

As for the number of letters before a newline - that's legacy stuff, probably for Fortran and fixed-width consoles. In fasta format, lines may be any length and newlines are irrelevant; they are typically wrapped at 70 characters. If you input a genome with 70-character wrapping and output it with 100-character wrapping, that is still the same genome, and no correctly-written program will differentiate between them. Fastq is much more convenient because newlines actually have a meaning.

Oh, and "N" means unknown. If you only care about compressing actual genomes as tightly as possible, you can just handle capital ACGTN, but you still must handle the names (in fasta, that's everything from the ">" to the next newline).

[AKatawazi]

Thank you for the very detailed response Brian, my background is in computer science so I have been very much flying in the dark here. I think I could make the program just detect the usage of lower and upper and preserve the formatting without much issue, but the lower case and upper case is also going to significantly degrade compression, I'm going to have to come up with a good solution. At any rate my current progress is 92% compression, which is not the best and I am able to compress a 250MB chromosome in 9 seconds which is also not the best. Thanks again for the much needed help.