GATK re aligner step doubles file size?

[manducasexta]

Has anyone else seen the GATK IndelRealigner double file size? It happened on a set of 12 genomes I'm processing. After the next step the file sizes returned to their normal range. A quick look at the first reads in the files showed no difference; same reads in the same order with the same amount of metadata, a little of it changed. I'm indexing them now so I can sample a few other regions.

Thanks!

[adaptivegenome]

Sounds odd. The only reason I can think is if GATK LR is returning the BAM unsorted for some reason. A sorted BAM can be smaller than an unsorted BAM since it is easier to compress...

[dawe]

Is that the IndelRealigner? I found the same problem with TableRecalibration, but that happens because GATK retains old quality scores for each read. Solved with "--doNotWriteOriginalQuals" options.

[swNGS]

I had noticed that too... Is there any reason why anyone would want to retain the original quality information (aside from an OCD-esque obsessing with not discarding anything)

[manducasexta]

Hi Dawe --
While the quality scores are retained in the later files, those files are back down to a reasonable size. The following image shows the progression of file sizes in obsessive detail. There are 12 points per analysis step because I'm running 12 samples in parallel:



I'm wondering if I could have caused the problem by something atypical I did in preparing the interval file for the realignment: I needed to add read groups to the alignments, so I ran parallel jobs creating the interval file and adding read groups. Then I used the interval file to realign the reads in the file with the newly added read groups (one per file). I reasoned that read groups aren't relevant to realignment when there is one per file, but maybe I tripped over some unexpected consequences.

[manducasexta]

>Is there any reason why anyone would want to retain the original quality information

swNGS: it's useful if you plan to keep the bams as your sole archive for a sequencing project because if you need to realign in the future (e.g. to a new version of the genome or a new genome entirely because you've been aligning to a related organism since yours isn't sequenced yet), you can recreate fastq from the original quality scores, and not have artifacts in the quality scores from errors in the recalibration based on the first genome.

But obsessive data retention may play a part too.

[swNGS]

Manducasexta: hmmm you have a point there!
We were having a discussion in the lab recently about what data to keep/discard etc. I'm all for keeping the minimum required, and hadnt considered that you could regenerate the FASTQ from the bam.
What would be the path to achieve this? As I could theoretically discard the original fastqs....

[manducasexta]

swNGS: I haven't done it, so I don't have a method immediately at hand. But having verified that the the information is present in the SAM file, I'm confident that fastq containing the original quality scores could be generated from SAM using perl or (probably) some other tool for parsing SAM format. When using an aligner that can use BAM input, it would be sufficient to replace the recalibrated quality scores with the original scores in a copy of the file.