Header Leaderboard Ad


GATK re aligner step doubles file size?



No announcement yet.
  • Filter
  • Time
  • Show
Clear All
new posts

  • GATK re aligner step doubles file size?

    Has anyone else seen the GATK IndelRealigner double file size? It happened on a set of 12 genomes I'm processing. After the next step the file sizes returned to their normal range. A quick look at the first reads in the files showed no difference; same reads in the same order with the same amount of metadata, a little of it changed. I'm indexing them now so I can sample a few other regions.


  • #2
    Sounds odd. The only reason I can think is if GATK LR is returning the BAM unsorted for some reason. A sorted BAM can be smaller than an unsorted BAM since it is easier to compress...


    • #3
      Is that the IndelRealigner? I found the same problem with TableRecalibration, but that happens because GATK retains old quality scores for each read. Solved with "--doNotWriteOriginalQuals" options.


      • #4
        I had noticed that too... Is there any reason why anyone would want to retain the original quality information (aside from an OCD-esque obsessing with not discarding anything)


        • #5
          Hi Dawe --
          While the quality scores are retained in the later files, those files are back down to a reasonable size. The following image shows the progression of file sizes in obsessive detail. There are 12 points per analysis step because I'm running 12 samples in parallel:

          I'm wondering if I could have caused the problem by something atypical I did in preparing the interval file for the realignment: I needed to add read groups to the alignments, so I ran parallel jobs creating the interval file and adding read groups. Then I used the interval file to realign the reads in the file with the newly added read groups (one per file). I reasoned that read groups aren't relevant to realignment when there is one per file, but maybe I tripped over some unexpected consequences.


          • #6
            >Is there any reason why anyone would want to retain the original quality information

            swNGS: it's useful if you plan to keep the bams as your sole archive for a sequencing project because if you need to realign in the future (e.g. to a new version of the genome or a new genome entirely because you've been aligning to a related organism since yours isn't sequenced yet), you can recreate fastq from the original quality scores, and not have artifacts in the quality scores from errors in the recalibration based on the first genome.

            But obsessive data retention may play a part too.


            • #7
              Manducasexta: hmmm you have a point there!
              We were having a discussion in the lab recently about what data to keep/discard etc. I'm all for keeping the minimum required, and hadnt considered that you could regenerate the FASTQ from the bam.
              What would be the path to achieve this? As I could theoretically discard the original fastqs....


              • #8
                swNGS: I haven't done it, so I don't have a method immediately at hand. But having verified that the the information is present in the SAM file, I'm confident that fastq containing the original quality scores could be generated from SAM using perl or (probably) some other tool for parsing SAM format. When using an aligner that can use BAM input, it would be sufficient to replace the recalibrated quality scores with the original scores in a copy of the file.


                Latest Articles


                • seqadmin
                  A Brief Overview and Common Challenges in Single-cell Sequencing Analysis
                  by seqadmin

                  ​​​​​​The introduction of single-cell sequencing has advanced the ability to study cell-to-cell heterogeneity. Its use has improved our understanding of somatic mutations1, cell lineages2, cellular diversity and regulation3, and development in multicellular organisms4. Single-cell sequencing encompasses hundreds of techniques with different approaches to studying the genomes, transcriptomes, epigenomes, and other omics of individual cells. The analysis of single-cell sequencing data i...

                  01-24-2023, 01:19 PM
                • seqadmin
                  Introduction to Single-Cell Sequencing
                  by seqadmin
                  Single-cell sequencing is a technique used to investigate the genome, transcriptome, epigenome, and other omics of individual cells using high-throughput sequencing. This technology has provided many scientific breakthroughs and continues to be applied across many fields, including microbiology, oncology, immunology, neurobiology, precision medicine, and stem cell research.

                  The advancement of single-cell sequencing began in 2009 when Tang et al. investigated the single-cell transcriptomes
                  01-09-2023, 03:10 PM