ABYSS never completes a run

[Haumich]

Hello Everyone,

I am currently trying to assemble mammalian paired-end DNASeq files from Illumina using ABYSS. The problem is that ABYSS gets stuck during the Assembly and never finishes (or at least it didn't during seven days).
The data consists of two 80GB fastq files that I fed to ABYSS 1.5.2. After days, the terminal still shows
0: Reading `/data/reads_1.fq'...
1: Reading `/data/reads_2.fq'...
It always stays at this prompt if I try to read multiple files. I can see that no data is actually read into the RAM.
When I merge those files to a single one, the program reads data to the RAM, makes it to "Finding adjacenct k-mer..." and remains there.
I can see that the processes are still running, but no files are written to the disk. I am working on a single node with 32 Processors and 720GB RAM. The server uses SLURM and openMPI version 1.8.5.
I have read that sometimes the eager limit of MPI is to small, so I set it to a higher value with a given formula, but that didn't solve the problem.
The command I use:

sbatch --job-name=abyss --partition=hive --nodes=1 --ntasks-per-node=32 --wrap "
abyss-pe k=51 n=10 name=test np=32 in='/data/reads_1.fq /data/reads_2.fq'
"

If someone had a similar problem and would like to share some thoughts, any help is aprreciated.

EDIT
Thanks pmiguel for the remark.
With the "v=-vv" option, I got additional output. It seems as if ABYSS really takes that long to load the reads. In the beginning, it takes about 35 seconds to read 100.000 reads. That increases to about 1 minute and so on. I have the feeling, its the server that is so slow.

[pmiguel]

I suggest adding "v=-vv" (verbose mode) to your abyss-pe parameters to get a better idea of what is hanging it up. As long as you stdout and stderr are being captured to a log file, you will be able to see what the program is up to.

Also, maybe you should update ABySS? The most recent release is 1.9.0
--
Phillip

[westerman]

With the "v=-vv" option, I got additional output. It seems as if ABYSS really takes that long to load the reads. In the beginning, it takes about 35 seconds to read 100.000 reads. That increases to about 1 minute and so on. I have the feeling, its the server that is so slow.

I think, but do not know, that it is ABySS that is slow. The more reads it has to work with the slower it goes. Much worse than linear speed. Could be a hashing issue (in which case normalized reads or sub-sampled reads might help) or a memory space issue. Phillip and I are currently working on a mammalian genome and might get some insights into ABySS's performance along the way.

BTW: The ABySS forum is now hosted on Biostars.

https://www.biostars.org/t/abyss/

In particular you could look at:

https://www.biostars.org/p/150328/

[Haumich]

Hello,

thanks everyone for their suggestions. It seems that the problem was openMPI. At least, once I changed to mpich 3.1.3, the speed of the assembly increased greatly and the program finished in about a day.