cuffcompare issue with many input gtf files

[choy]

I am running cufflinks with many input GTF files (>30). It works great for a few input files, but when the number approaches 30 the program crashes. There is definitely enough total memory available on my machine, so this must be a different kind of memory issue.

Here is a trace of my output:

> cuffcompare input1.gtf input2.gtf ....
Warning: found 19753 transcripts with undetermined strand.
Warning: found 32445 transcripts with undetermined strand.
Warning: found 46460 transcripts with undetermined strand.
Warning: found 45651 transcripts with undetermined strand.
Warning: found 53621 transcripts with undetermined strand.
Warning: found 35459 transcripts with undetermined strand.
Warning: found 17761 transcripts with undetermined strand.
Warning: found 57655 transcripts with undetermined strand.
Warning: found 25062 transcripts with undetermined strand.
Warning: found 131285 transcripts with undetermined strand.
Warning: found 33114 transcripts with undetermined strand.
Warning: found 61036 transcripts with undetermined strand.
Warning: found 59248 transcripts with undetermined strand.
Warning: found 25495 transcripts with undetermined strand.
Warning: found 19366 transcripts with undetermined strand.
Warning: found 78199 transcripts with undetermined strand.
Warning: found 7142 transcripts with undetermined strand.
Warning: found 31922 transcripts with undetermined strand.
Warning: found 90403 transcripts with undetermined strand.
Warning: found 38985 transcripts with undetermined strand.
Warning: found 141074 transcripts with undetermined strand.
Warning: found 81031 transcripts with undetermined strand.
Warning: found 62373 transcripts with undetermined strand.
Warning: found 66286 transcripts with undetermined strand.
Warning: found 11822 transcripts with undetermined strand.
Warning: found 21012 transcripts with undetermined strand.
*** glibc detected *** /lab/sw/bioinfo/rnaseq/cufflinks/current/cuffcompare: double free or corruption (out): 0x0000000025bc4b20 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3824c71634]
/lib64/libc.so.6(cfree+0x8c)[0x3824c74c5c]
/lab/sw/bioinfo/rnaseq/cufflinks/current/cuffcompare[0x419849]
/lab/sw/bioinfo/rnaseq/cufflinks/current/cuffcompare[0x412088]
/lab/sw/bioinfo/rnaseq/cufflinks/current/cuffcompare[0x414be4]
/lab/sw/bioinfo/rnaseq/cufflinks/current/cuffcompare[0x40293f]
/lab/sw/bioinfo/rnaseq/cufflinks/current/cuffcompare[0x4103f0]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x3824c1d8b4]
/lab/sw/bioinfo/rnaseq/cufflinks/current/cuffcompare(__gxx_personality_v0+0x5a)[0x4015aa]
======= Memory map: ========
00400000-0043a000 r-xp 00000000 00:19 730763876 /lab/sw/bioinfo/rnaseq/cufflinks/cufflinks-0.7.0.Linux_x86_64/cuffcompare
00539000-0053a000 rw-p 00039000 00:19 730763876 /lab/sw/bioinfo/rnaseq/cufflinks/cufflinks-0.7.0.Linux_x86_64/cuffcompare
0053a000-00543000 rw-p 0053a000 00:00 0
1340f000-1bf084000 rw-p 1340f000 00:00 0
3824800000-382481a000 r-xp 00000000 68:03 1702403 /lib64/ld-2.5.so
3824a1a000-3824a1b000 r--p 0001a000 68:03 1702403 /lib64/ld-2.5.so
3824a1b000-3824a1c000 rw-p 0001b000 68:03 1702403 /lib64/ld-2.5.so
3824c00000-3824d4a000 r-xp 00000000 68:03 1702560 /lib64/libc-2.5.so
3824d4a000-3824f49000 ---p 0014a000 68:03 1702560 /lib64/libc-2.5.so
3824f49000-3824f4d000 r--p 00149000 68:03 1702560 /lib64/libc-2.5.so
3824f4d000-3824f4e000 rw-p 0014d000 68:03 1702560 /lib64/libc-2.5.so
3824f4e000-3824f53000 rw-p 3824f4e000 00:00 0
3825000000-3825082000 r-xp 00000000 68:03 1702604 /lib64/libm-2.5.so
3825082000-3825281000 ---p 00082000 68:03 1702604 /lib64/libm-2.5.so
3825281000-3825282000 r--p 00081000 68:03 1702604 /lib64/libm-2.5.so
3825282000-3825283000 rw-p 00082000 68:03 1702604 /lib64/libm-2.5.so
3825800000-3825815000 r-xp 00000000 68:03 1702589 /lib64/libpthread-2.5.so
3825815000-3825a14000 ---p 00015000 68:03 1702589 /lib64/libpthread-2.5.so
3825a14000-3825a15000 r--p 00014000 68:03 1702589 /lib64/libpthread-2.5.so
3825a15000-3825a16000 rw-p 00015000 68:03 1702589 /lib64/libpthread-2.5.so
3825a16000-3825a1a000 rw-p 3825a16000 00:00 0
382a400000-382a40d000 r-xp 00000000 68:03 1702453 /lib64/libgcc_s-4.1.2-20080102.so.1
382a40d000-382a60d000 ---p 0000d000 68:03 1702453 /lib64/libgcc_s-4.1.2-20080102.so.1
382a60d000-382a60e000 rw-p 0000d000 68:03 1702453 /lib64/libgcc_s-4.1.2-20080102.so.1
382a800000-382a8e6000 r-xp 00000000 68:03 2496934 /usr/lib64/libstdc++.so.6.0.8
382a8e6000-382aae5000 ---p 000e6000 68:03 2496934 /usr/lib64/libstdc++.so.6.0.8
382aae5000-382aaeb000 r--p 000e5000 68:03 2496934 /usr/lib64/libstdc++.so.6.0.8
382aaeb000-382aaee000 rw-p 000eb000 68:03 2496934 /usr/lib64/libstdc++.so.6.0.8
382aaee000-382ab00000 rw-p 382aaee000 00:00 0
2acef01be000-2acef01c0000 rw-p 2acef01be000 00:00 0
2acef01db000-2acef0215000 rw-p 2acef01db000 00:00 0
2acef4000000-2acef4021000 rw-p 2acef4000000 00:00 0
2acef4021000-2acef8000000 ---p 2acef4021000 00:00 0
7fffba8d4000-7fffba8ec000 rw-p 7fffba8d4000 00:00 0 [stack]
ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0 [vdso]
Aborted

[gpertea]

Indeed, cuffcompare has a hard coded fixed limit of 24 input query files -- which I admit was chosen arbitrarily, just because we never needed more than 4-5 in our experiments and I didn't imagine that people would need many more anyway.. Obviously I was wrong, and I am going to increase that limit (I still want to keep that number fixed instead of dynamic for reasons related to the efficiency of the data structures employed there).

So what do you think it would be a reasonable upper limit to use there instead? Please let me know so I can adjust that limit accordingly for the next release of cuffcompare.

If you want to modify that limit yourself feel free to edit the source code -- it's just a matter of changing one number in this line at the beginning of gtf_tracking.h:

#define MAX_QFILES 24

Just place a larger value there instead of 24 and recompile, that should do it.