MosaikJump not working properly

[rahul.m.dhodapkar]

Hello all,

I have been trying to create a mosaik jump database from a reference fasta file (hg18, NCBI build 36). I'm using the following command:

./MosaikJump -ia hg18_combined.dat -out hg18_combined.jmp -hs 15

it starts off running fine, but when it gets to 5% completed, the ETA starts climbing and the "number of hashes" also starts climbing rapidly. this goes until both values roll over to 0, and then the program freezes. help much appreciated.

-Rahul Dhodapkar

[KevinLam]

how much ram do you have?

MosaikJump -ia human_g1k_v37_chr_ucsc.fasta.dat -out human_g1k_v37_chr_ucsc.fasta_15 -hs 15 -mem 6

works fine for me but I ran into memory problems again when I tried to aligned.

[rahul.m.dhodapkar]

i have 10 GB of ram, is that insufficient for this task?

-Rahul Dhodapkar

[tinacai]

Hello all,

I have been trying to create a mosaikbuild vertion from a reference fasta file (hg18, NCBI build 36). I'm using the following command:
./Mosaikbuild -fr hg18.fa -oa hg18.dat ,warning
ERROR: Could not open FASTA file (/home/database/hg18/hg18.fa) when performing integrity check
but I input ./Mosaikbuild -fr chr1.fa -oa chr1.dat ,It can running now .my RAM is 32G,I want to kown if my RAM is low

[rahul.m.dhodapkar]

32 Gigabytes of RAM should be plenty to convert hg18.fa to hg18.dat

[tinacai]

mosaik-aligner/bin/MosaikBuild -fr /home/database/hg18/hg18.fa.gz -oa /home/share2/chenchong/hg18.dat
------------------------------------------------------------------------------
MosaikBuild 1.0.1388 2010-02-01
Michael Stromberg Marth Lab, Boston College Biology Department
------------------------------------------------------------------------------

- converting /home/database/hg18/hg18.fa.gz to a reference sequence archive.

- parsing reference sequences:
ref seqs: 49 (0.1205 ref seqs/s)

- writing reference sequences:
100%[==================================] 1.71 ref seqs/s in 28 s

- calculating MD5 checksums:
100%[==================================] 3.90 ref seqs/s in 12 s

- writing reference sequence index:
100%[==================================] 49.0 ref seqs/s in 1 s
ERROR: Unable to allocate enough memory (3087005219 bytes) to create the concatenated reference sequence.
my RAM is 32G,but the result warning Unable to allocate enough memory (3087005219 bytes) to create the concatenated reference sequence.

[rahul.m.dhodapkar]

wait, why do you have 49 ref seqs? Shouldn't there only be 25 (22 autosomes + X, Y, M)? That might be the source of the problem. What exactly is in your hg18.fa file?

[tinacai]

>chr10
>chr10_random
>chr11
>chr11_random
>chr12
>chr13
>chr13_random
>chr14
>chr15
>chr15_random
>chr16
>chr16_random
>chr17
>chr17_random
>chr18
>chr18_random
>chr19
>chr19_random
>chr1
>chr1_random
>chr20
>chr21
>chr21_random
>chr22
>chr22_h2_hap1
>chr22_random
>chr2
>chr2_random
>chr3
>chr3_random
>chr4
>chr4_random
>chr5
>chr5_h2_hap1
>chr5_random
>chr6_cox_hap1
>chr6
>chr6_qbl_hap2
>chr6_random
>chr7
>chr7_random
>chr8
>chr8_random
>chr9
>chr9_random
>chrM
>chrX
>chrX_random
>chrY

[tinacai]

this mine hg18.fa

[rahul.m.dhodapkar]

I'm not sure how useful anything that aligns to _random will be, since the _random .fa files contain random sequence that is on those chromosomes. They're bits that haven't been fitted in yet, so I would try removing those, and running MosaikBuild again and seeing where that puts you. It may well be that those additional sequences are eating up an enormous amount of memory. Let me know how that goes.

-Rahul Dhodapkar

[tinacai]

[chenchong@node03 chenchong]$ mosaik-aligner/bin/MosaikBuild -fr /home/share2/chenchong/hg18/hg18.fa -oa /home/share2/chenchong/hg18.dat
------------------------------------------------------------------------------
MosaikBuild 1.0.1388 2010-02-01
Michael Stromberg Marth Lab, Boston College Biology Department
------------------------------------------------------------------------------

- converting /home/share2/chenchong/hg18/hg18.fa to a reference sequence archive.

- parsing reference sequences:
ref seqs: 25 (0.1938 ref seqs/s)

- writing reference sequences:
100%[==================================] 0.8736 ref seqs/s in 28 s

- calculating MD5 checksums:
100%[==================================] 1.99 ref seqs/s in 12 s

- writing reference sequence index:
100%[==================================] 25.0 ref seqs/s in 1 s
ERROR: Unable to allocate enough memory (3080448052 bytes) to create the concatenated reference sequence.
I have been removed the _random ,but the resullt seem to similar

[rahul.m.dhodapkar]

tinacai,

do you have a portal you can use to monitor the memory usage of your node? How much memory is the process using when it fails? Is there a memory usage spike?

-Rahul Dhodapkar

[tinacai]

Rahul
I'm sure it is not problem for my memory, my RAM is 32G and only me is using it

[rahul.m.dhodapkar]

it's possible is that there is some sort of internal memory ceiling that prevents your process from using all of the RAM that you physically have available, which is why I'm asking if the memory usage is highest at the last step of the process

[tinacai]

hi,Rahul
I have run my program again and check my memory situation,I found my memory usage highest at the last step of the process is 9.5%