Problem with normailzing data using jellyfish

[jcorn427]

Hi all, I have a rather large set of data that I want to assemble using trinity, but I'm trying to normalize it first using the normalize_by_kmer_coverage.pl script. However, I got the following error, any help would be awesome.

Can't merge hash with different size
Error, cmd: /home/jcornel3/tools/trinity/util/..//trinity-plugins/jellyfish/bin/jellyfish merge -o jellyfish.kmer.db mer_counts_0 mer_counts_1 ... mer_counts_98 mer_counts_99 died with ret 256 at /home/jcornel3/tools/trinity/util/normalize_by_kmer_coverage.pl line 652.

[yueluo]

What was the command line you used, and what other warning/error messages did you get?
Also, you could try trinity's mailing-list -- there you can most likely get help from the developers.

[jcorn427]

Ok, here's the command I used to start the normalization:

/home/jcornel3/tools/trinity/util/normalize_by_kmer_coverage.pl --seqType fq --JM 450G --max_cov 30 --left reads.ALL.1.fq --right reads.ALL.2.fq --pairs_together --PARALLEL_STATS --JELY_CPU 30

And, here is the full output, it's pretty long:

Converting input files. (both directions in parallel)left file exists, nothing to doright file exists, nothing to do-------------------------------------------
----------- Jellyfish  --------------------
-- (building a k-mer catalog from reads) --
-------------------------------------------

Done converting input files.CMD: /home/jcornel3/tools/trinity/util/..//trinity-plugins/jellyfish/bin/jellyfish count -t 30 -m 25 -s 13495821759  --both-strands  both.fa                                                             [3/3079]
CMD finished (16019 seconds)
CMD: /home/jcornel3/tools/trinity/util/..//trinity-plugins/jellyfish/bin/jellyfish merge -o jellyfish.kmer.db mer_counts_0 mer_counts_1 mer_counts_10 mer_counts_100 mer
_counts_101 mer_counts_102 mer_counts_103 mer_counts_104 mer_counts_105 mer_counts_106 mer_counts_107 mer_counts_108 mer_counts_109 mer_counts_11 mer_counts_110 mer_cou
nts_111 mer_counts_112 mer_counts_113 mer_counts_114 mer_counts_115 mer_counts_116 mer_counts_117 mer_counts_118 mer_counts_119 mer_counts_12 mer_counts_120 mer_counts_
121 mer_counts_122 mer_counts_123 mer_counts_124 mer_counts_125 mer_counts_126 mer_counts_127 mer_counts_128 mer_counts_129 mer_counts_13 mer_counts_130 mer_counts_131
mer_counts_132 mer_counts_133 mer_counts_134 mer_counts_135 mer_counts_136 mer_counts_137 mer_counts_138 mer_counts_139 mer_counts_14 mer_counts_140 mer_counts_141 mer_
counts_142 mer_counts_143 mer_counts_144 mer_counts_145 mer_counts_146 mer_counts_147 mer_counts_148 mer_counts_149 mer_counts_15 mer_counts_150 mer_counts_151 mer_coun
ts_152 mer_counts_153 mer_counts_154 mer_counts_155 mer_counts_156 mer_counts_157 mer_counts_158 mer_counts_159 mer_counts_16 mer_counts_160 mer_counts_161 mer_counts_1
62 mer_counts_163 mer_counts_164 mer_counts_165 mer_counts_166 mer_counts_167 mer_counts_168 mer_counts_169 mer_counts_17 mer_counts_170 mer_counts_171 mer_counts_172 m
er_counts_173 mer_counts_174 mer_counts_175 mer_counts_176 mer_counts_177 mer_counts_178 mer_counts_179 mer_counts_18 mer_counts_180 mer_counts_181 mer_counts_182 mer_c
ounts_183 mer_counts_184 mer_counts_185 mer_counts_186 mer_counts_187 mer_counts_188 mer_counts_189 mer_counts_19 mer_counts_190 mer_counts_191 mer_counts_192 mer_count
s_193 mer_counts_194 mer_counts_195 mer_counts_196 mer_counts_197 mer_counts_198 mer_counts_199 mer_counts_2 mer_counts_20 mer_counts_200 mer_counts_201 mer_counts_202
mer_counts_203 mer_counts_204 mer_counts_205 mer_counts_206 mer_counts_207 mer_counts_208 mer_counts_209 mer_counts_21 mer_counts_210 mer_counts_211 mer_counts_212 mer_
counts_213 mer_counts_214 mer_counts_215 mer_counts_216 mer_counts_217 mer_counts_218 mer_counts_219 mer_counts_22 mer_counts_220 mer_counts_221 mer_counts_222 mer_coun
ts_223 mer_counts_224 mer_counts_225 mer_counts_226 mer_counts_227 mer_counts_228 mer_counts_229 mer_counts_23 mer_counts_230 mer_counts_231 mer_counts_232 mer_counts_2
33 mer_counts_234 mer_counts_235 mer_counts_236 mer_counts_237 mer_counts_238 mer_counts_239 mer_counts_24 mer_counts_240 mer_counts_241 mer_counts_242 mer_counts_243 m
er_counts_244 mer_counts_245 mer_counts_246 mer_counts_247 mer_counts_248 mer_counts_249 mer_counts_25 mer_counts_250 mer_counts_251 mer_counts_252 mer_counts_253 mer_c
ounts_254 mer_counts_255 mer_counts_256 mer_counts_257 mer_counts_258 mer_counts_259 mer_counts_26 mer_counts_260 mer_counts_261 mer_counts_262 mer_counts_263 mer_count
s_264 mer_counts_265 mer_counts_266 mer_counts_267 mer_counts_268 mer_counts_269 mer_counts_27 mer_counts_270 mer_counts_271 mer_counts_272 mer_counts_273 mer_counts_27
4 mer_counts_275 mer_counts_276 mer_counts_277 mer_counts_278 mer_counts_279 mer_counts_28 mer_counts_280 mer_counts_281 mer_counts_282 mer_counts_283 mer_counts_284 me
r_counts_285 mer_counts_286 mer_counts_287 mer_counts_288 mer_counts_289 mer_counts_29 mer_counts_290 mer_counts_291 mer_counts_292 mer_counts_293 mer_counts_294 mer_co
unts_295 mer_counts_296 mer_counts_297 mer_counts_298 mer_counts_299 mer_counts_3 mer_counts_30 mer_counts_300 mer_counts_301 mer_counts_302 mer_counts_303 mer_counts_3
04 mer_counts_305 mer_counts_306 mer_counts_307 mer_counts_308 mer_counts_309 mer_counts_31 mer_counts_310 mer_counts_311 mer_counts_312 mer_counts_313 mer_counts_314 m
er_counts_315 mer_counts_316 mer_counts_317 mer_counts_318 mer_counts_319 mer_counts_32 mer_counts_320 mer_counts_321 mer_counts_322 mer_counts_323 mer_counts_324 mer_c
ounts_325 mer_counts_326 mer_counts_327 mer_counts_328 mer_counts_329 mer_counts_33 mer_counts_330 mer_counts_331 mer_counts_332 mer_counts_333 mer_counts_334 mer_count
s_335 mer_counts_336 mer_counts_337 mer_counts_338 mer_counts_339 mer_counts_34 mer_counts_35 mer_counts_36 mer_counts_37 mer_counts_38 mer_counts_39 mer_counts_4 mer_c
ounts_40 mer_counts_41 mer_counts_42 mer_counts_43 mer_counts_44 mer_counts_45 mer_counts_46 mer_counts_47 mer_counts_48 mer_counts_49 mer_counts_5 mer_counts_50 mer_co
unts_51 mer_counts_52 mer_counts_53 mer_counts_54 mer_counts_55 mer_counts_56 mer_counts_57 mer_counts_58 mer_counts_59 mer_counts_6 mer_counts_60 mer_counts_61 mer_cou
nts_62 mer_counts_63 mer_counts_64 mer_counts_65 mer_counts_66 mer_counts_67 mer_counts_68 mer_counts_69 mer_counts_7 mer_counts_70 mer_counts_71 mer_counts_72 mer_coun
ts_73 mer_counts_74 mer_counts_75 mer_counts_76 mer_counts_77 mer_counts_78 mer_counts_79 mer_counts_8 mer_counts_80 mer_counts_81 mer_counts_82 mer_counts_83 mer_count
s_84 mer_counts_85 mer_counts_86 mer_counts_87 mer_counts_88 mer_counts_89 mer_counts_9 mer_counts_90 mer_counts_91 mer_counts_92 mer_counts_93 mer_counts_94 mer_counts
_95 mer_counts_96 mer_counts_97 mer_counts_98 mer_counts_99
Can't merge hash with different size
Error, cmd: /home/jcornel3/tools/trinity/util/..//trinity-plugins/jellyfish/bin/jellyfish merge -o jellyfish.kmer.db mer_counts_0 mer_counts_1 mer_counts_10 mer_counts_
100 mer_counts_101 mer_counts_102 mer_counts_103 mer_counts_104 mer_counts_105 mer_counts_106 mer_counts_107 mer_counts_108 mer_counts_109 mer_counts_11 mer_counts_110
mer_counts_111 mer_counts_112 mer_counts_113 mer_counts_114 mer_counts_115 mer_counts_116 mer_counts_117 mer_counts_118 mer_counts_119 mer_counts_12 mer_counts_120 mer_
counts_121 mer_counts_122 mer_counts_123 mer_counts_124 mer_counts_125 mer_counts_126 mer_counts_127 mer_counts_128 mer_counts_129 mer_counts_13 mer_counts_130 mer_coun
ts_131 mer_counts_132 mer_counts_133 mer_counts_134 mer_counts_135 mer_counts_136 mer_counts_137 mer_counts_138 mer_counts_139 mer_counts_14 mer_counts_140 mer_counts_1
41 mer_counts_142 mer_counts_143 mer_counts_144 mer_counts_145 mer_counts_146 mer_counts_147 mer_counts_148 mer_counts_149 mer_counts_15 mer_counts_150 mer_counts_151 m
er_counts_152 mer_counts_153 mer_counts_154 mer_counts_155 mer_counts_156 mer_counts_157 mer_counts_158 mer_counts_159 mer_counts_16 mer_counts_160 mer_counts_161 mer_c
ounts_162 mer_counts_163 mer_counts_164 mer_counts_165 mer_counts_166 mer_counts_167 mer_counts_168 mer_counts_169 mer_counts_17 mer_counts_170 mer_counts_171 mer_count
s_172 mer_counts_173 mer_counts_174 mer_counts_175 mer_counts_176 mer_counts_177 mer_counts_178 mer_counts_179 mer_counts_18 mer_counts_180 mer_counts_181 mer_counts_18
2 mer_counts_183 mer_counts_184 mer_counts_185 mer_counts_186 mer_counts_187 mer_counts_188 mer_counts_189 mer_counts_19 mer_counts_190 mer_counts_191 mer_counts_192 me
r_counts_193 mer_counts_194 mer_counts_195 mer_counts_196 mer_counts_197 mer_counts_198 mer_counts_199 mer_counts_2 mer_counts_20 mer_counts_200 mer_counts_201 mer_coun
ts_202 mer_counts_203 mer_counts_204 mer_counts_205 mer_counts_206 mer_counts_207 mer_counts_208 mer_counts_209 mer_counts_21 mer_counts_210 mer_counts_211 mer_counts_2
12 mer_counts_213 mer_counts_214 mer_counts_215 mer_counts_216 mer_counts_217 mer_counts_218 mer_counts_219 mer_counts_22 mer_counts_220 mer_counts_221 mer_counts_222 m
er_counts_223 mer_counts_224 mer_counts_225 mer_counts_226 mer_counts_227 mer_counts_228 mer_counts_229 mer_counts_23 mer_counts_230 mer_counts_231 mer_counts_232 mer_c
ounts_233 mer_counts_234 mer_counts_235 mer_counts_236 mer_counts_237 mer_counts_238 mer_counts_239 mer_counts_24 mer_counts_240 mer_counts_241 mer_counts_242 mer_count
s_243 mer_counts_244 mer_counts_245 mer_counts_246 mer_counts_247 mer_counts_248 mer_counts_249 mer_counts_25 mer_counts_250 mer_counts_251 mer_counts_252 mer_counts_25
3 mer_counts_254 mer_counts_255 mer_counts_256 mer_counts_257 mer_counts_258 mer_counts_259 mer_counts_26 mer_counts_260 mer_counts_261 mer_counts_262 mer_counts_263 me
r_counts_264 mer_counts_265 mer_counts_266 mer_counts_267 mer_counts_268 mer_counts_269 mer_counts_27 mer_counts_270 mer_counts_271 mer_counts_272 mer_counts_273 mer_co
unts_274 mer_counts_275 mer_counts_276 mer_counts_277 mer_counts_278 mer_counts_279 mer_counts_28 mer_counts_280 mer_counts_281 mer_counts_282 mer_counts_283 mer_counts
_284 mer_counts_285 mer_counts_286 mer_counts_287 mer_counts_288 mer_counts_289 mer_counts_29 mer_counts_290 mer_counts_291 mer_counts_292 mer_counts_293 mer_counts_294
 mer_counts_295 mer_counts_296 mer_counts_297 mer_counts_298 mer_counts_299 mer_counts_3 mer_counts_30 mer_counts_300 mer_counts_301 mer_counts_302 mer_counts_303 mer_c
ounts_304 mer_counts_305 mer_counts_306 mer_counts_307 mer_counts_308 mer_counts_309 mer_counts_31 mer_counts_310 mer_counts_311 mer_counts_312 mer_counts_313 mer_count
s_314 mer_counts_315 mer_counts_316 mer_counts_317 mer_counts_318 mer_counts_319 mer_counts_32 mer_counts_320 mer_counts_321 mer_counts_322 mer_counts_323 mer_counts_32
4 mer_counts_325 mer_counts_326 mer_counts_327 mer_counts_328 mer_counts_329 mer_counts_33 mer_counts_330 mer_counts_331 mer_counts_332 mer_counts_333 mer_counts_334 me
r_counts_335 mer_counts_336 mer_counts_337 mer_counts_338 mer_counts_339 mer_counts_34 mer_counts_35 mer_counts_36 mer_counts_37 mer_counts_38 mer_counts_39 mer_counts_
4 mer_counts_40 mer_counts_41 mer_counts_42 mer_counts_43 mer_counts_44 mer_counts_45 mer_counts_46 mer_counts_47 mer_counts_48 mer_counts_49 mer_counts_5 mer_counts_50
 mer_counts_51 mer_counts_52 mer_counts_53 mer_counts_54 mer_counts_55 mer_counts_56 mer_counts_57 mer_counts_58 mer_counts_59 mer_counts_6 mer_counts_60 mer_counts_61
mer_counts_62 mer_counts_63 mer_counts_64 mer_counts_65 mer_counts_66 mer_counts_67 mer_counts_68 mer_counts_69 mer_counts_7 mer_counts_70 mer_counts_71 mer_counts_72 m
er_counts_73 mer_counts_74 mer_counts_75 mer_counts_76 mer_counts_77 mer_counts_78 mer_counts_79 mer_counts_8 mer_counts_80 mer_counts_81 mer_counts_82 mer_counts_83 me
r_counts_84 mer_counts_85 mer_counts_86 mer_counts_87 mer_counts_88 mer_counts_89 mer_counts_9 mer_counts_90 mer_counts_91 mer_counts_92 mer_counts_93 mer_counts_94 mer
_counts_95 mer_counts_96 mer_counts_97 mer_counts_98 mer_counts_99 died with ret 256 at /home/jcornel3/tools/trinity/util/normalize_by_kmer_coverage.pl line 652.

Also, thanks for the suggestion, I'll try the mailing list.

[jcorn427]

Also, here is the actual code from the script that is causing the process to die, hopefully that helps too.

sub process_cmd {
    my ($cmd) = @_;

    print "CMD: $cmd\n";

    my $start_time = time();
    my $ret = system($cmd);
    my $end_time = time();

    if ($ret) {
        die "Error, cmd: $cmd died with ret $ret";
    }

    print "CMD finished (" . ($end_time - $start_time) . " seconds)\n";

    return;
}

[yueluo]

There has been some feedback on the mailing list:

http://sourceforge.net/p/trinityrnaseq/mailman/message/32120686/

And I have to say I'm shocked by the size of data you are working with. Good luck!