FixMateInformation after GATK realignment

[MolecularToast]

I'm using GATK to analyze some paired-end Illumina exome data and have been running into a problem after realignment using IndelRealigner. After realignment, I have been trying to use Picard's FixMateInformation to take the realigned query-name-sorted bams and produce fixed, coordinate-sorted bams. This function creates a bunch of temp files and does so fairly quickly. However, the process of merging these temp files together takes an extraordinarily long time. Not sure what I'm doing wrong here.

Is it possible to simply use the FixMate and Sort functions in SamTools to perform the same task? This I've done and it works relatively quickly.

Lastly, one other quick question, after realignment using IndelRealigner, a one exome bam file which is only about 7.3 GB (following initial alignment with BWA) becomes approximately 32.8 GB. Is this supposed to happen?

Thanks, Toast

[dawe]

However, the process of merging these temp files together takes an extraordinarily long time. Not sure what I'm doing wrong here.

I thought it was normal :-) I had to process 6 bam files and it took slightly less than 3 weeks (from recalibration to final rmdup).

Lastly, one other quick question, after realignment using IndelRealigner, a one exome bam file which is only about 7.3 GB (following initial alignment with BWA) becomes approximately 32.8 GB. Is this supposed to happen?

Thanks, Toast

The BAM produced after the indelrealigner is huge, that is supposed to happen and I suspect it is related to the BAM file structure. Sorted files are smaller than unsorted ones. After InderRealigner you have the "less sorted" BAM file.

[Michael.James.Clark]

As far as FixMateInformation, I did not find it taking that excessively long. This was on only around 50,000,000 reads, but it took less than two hours. You can definitely try bumping up the MAX_RECORDS_IN_RAM parameter assuming you do have a lot of RAM to speed it up. My guess based on what you said is that you are spending tons of time reading and writing to disk (tmp files) which you can reduce by using more RAM. Remember to pass JVM more RAM when you run it.

As for the huge BAM file, is your final BAM file uncompressed? You can compress it further with multiple tools (does IndelRealigner have the option to output compressed BAM?).

[MolecularToast]

Thank you both for your replies. I tried increasing the MAX_RECORDS_IN_RAM once and all it seemed to do was create larger temp files that merged just as slowly. I'll try passing even more RAM to JVM and see if that speeds it up any but I'm nearly maxed out already.

I completely missed that before but, yes, it is an uncompressed BAM file and IndelRealigner does have an additional argument you can use to compress the output. That makes sense now.

Anyone know though whether SamTools FixMate and Sort functions can accomplish the same thing as Picards FixMateInformation?

Thanks, Toast

[NGSfan]

If you are using Fedora/Ubuntu linux *and* you have tons of RAM (128 or 256GB) you might want to use /dev/shm/ which is the temporary RAM drive as your tmp directory.

[MolecularToast]

Thank you - I did end up running it on a linux box with about 4x the comp power and 4x the RAM and it did speed it up (comparatively) quite a bit (at least within my tolerable range).

Thank you everyone for your suggestions. As near as I can tell the samtools alternative method did allow for downstream processing but I haven't compared the calls.

[hiberbear]

Thank you - I did end up running it on a linux box with about 4x the comp power and 4x the RAM and it did speed it up (comparatively) quite a bit (at least within my tolerable range).

Thank you everyone for your suggestions. As near as I can tell the samtools alternative method did allow for downstream processing but I haven't compared the calls.

Could you post your parameter for FixMateInformation? I have the same problems, it is really slowly.