[westerman]

Ray runs well when I use a single node, but when utilizing more than this I get an MPI exit code- like this
...
--------------------------------------------------------------------------
mpirun has exited due to process rank 4 with PID 25094 on
node c310 exiting without calling "finalize". This may
have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--------------------------------------------------------------------------

I am not a big Ray user but I will sometimes get the above problem and then when I do a re-run the problem goes away. I think that it has to do with my cluster's setup. I suggest trying a small run and put one job per node just to make sure that everything will work.

Not much help, I know, but the general idea is that the problem may be with your hardware setup and not with ray.

[bstamps]

Ray runs well when I use a single node, but when utilizing more than this I get an MPI exit code- like this

Ray:25109 terminated with signal 11 at PC=5718e0 SP=7fff9eb8a838. Backtrace:
/home/bstamps/Ray/Ray-v2.0.0/Ray(_ZNK14ReadAnnotation7getRankEv+0x0)[0x5718e0]
/home/bstamps/Ray/Ray-v2.0.0/Ray(_ZN40Adapter_RAY_MPI_TAG_REQUEST_VERTEX_READS4$
/home/bstamps/Ray/Ray-v2.0.0/Ray(_ZN18MessageTagExecutor11callHandlerEiP7Messag$
/home/bstamps/Ray/Ray-v2.0.0/Ray(_ZN11ComputeCore3runEv+0x3cc)[0x5985ec]
/home/bstamps/Ray/Ray-v2.0.0/Ray(_ZN7Machine5startEv+0x1d8d)[0x46906d]
/home/bstamps/Ray/Ray-v2.0.0/Ray(main+0x73)[0x464d73]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x2b3fbc934cdd]
/home/bstamps/Ray/Ray-v2.0.0/Ray[0x464c39]
--------------------------------------------------------------------------
mpirun has exited due to process rank 4 with PID 25094 on
node c310 exiting without calling "finalize". This may
have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--------------------------------------------------------------------------

Thoughts?

[jfpombert]

If you look at both outputs:

make[1]: mpicxx: Command not found

Make sure to add the directory containing the openmpi execs to your path. Should fix the problem.

[yaximik]

Guys,

I examine dthis tread from the very beginning but could not find answer for my problem. Sorry for silly question. I tried to install Ray 2.0.0 and failed on two machines, one SciLinux 5.5 and another RHEL 55, which are esentially the same. Here is the output:

[Code]
[yaximik@SciLinux55 Ray-v2.0.0]$ make PREFIX=ray-build
make[1]: Entering directory `/home/yaximik/Bioinformatics/Ray-v2.0.0/RayPlatform'
mpic++ -Wall -ansi -O3 -D MAXKMERLENGTH=32 -D RAY_VERSION=\"2.0.0\" -D RAYPLATFORM_VERSION=\"1.0.3\" -I. -c -o memory/ReusableMemoryStore.o memory/ReusableMemoryStore.cpp
make[1]: mpic++: Command not found
make[1]: *** [memory/ReusableMemoryStore.o] Error 127
make[1]: Leaving directory `/home/yaximik/Bioinformatics/Ray-v2.0.0/RayPlatform'
make[1]: Entering directory `/home/yaximik/Bioinformatics/Ray-v2.0.0/code'
mpic++ -Wall -ansi -O3 -D MAXKMERLENGTH=32 -D RAY_VERSION=\"2.0.0\" -I ../RayPlatform -I. -c -o application_core/ray_main.o application_core/ray_main.cpp
make[1]: mpic++: Command not found
make[1]: *** [application_core/ray_main.o] Error 127
make[1]: Leaving directory `/home/yaximik/Bioinformatics/Ray-v2.0.0/code'
mpic++ code/TheRayGenomeAssembler.a RayPlatform/libRayPlatform.a -o Ray
make: mpic++: Command not found
make: *** [Ray] Error 127
[yaximik@SciLinux55 Ray-v2.0.0]$
[Code]

Her is output from RHEL55

[code]
[[yaximik@G5NNJN1 Ray-v2.0.0]$ make PREFIX=ray-build
make[1]: Entering directory `/home/yaximik/Bioinformatics/Ray-v2.0.0/RayPlatform'
mpicxx -Wall -ansi -O3 -D MAXKMERLENGTH=32 -D RAY_VERSION=\"2.0.0\" -D RAYPLATFORM_VERSION=\"1.0.3\" -I. -c -o memory/ReusableMemoryStore.o memory/ReusableMemoryStore.cpp
make[1]: mpicxx: Command not found
make[1]: *** [memory/ReusableMemoryStore.o] Error 127
make[1]: Leaving directory `/home/yaximik/Bioinformatics/Ray-v2.0.0/RayPlatform'
make[1]: Entering directory `/home/yaximik/Bioinformatics/Ray-v2.0.0/code'
mpicxx -Wall -ansi -O3 -D MAXKMERLENGTH=32 -D RAY_VERSION=\"2.0.0\" -I ../RayPlatform -I. -c -o application_core/ray_main.o application_core/ray_main.cpp
make[1]: mpicxx: Command not found
make[1]: *** [application_core/ray_main.o] Error 127
make[1]: Leaving directory `/home/yaximik/Bioinformatics/Ray-v2.0.0/code'
mpicxx code/TheRayGenomeAssembler.a RayPlatform/libRayPlatform.a -o Ray
make: mpicxx: Command not found
make: *** [Ray] Error 127
[yaximik@G5NNJN1 Ray-v2.0.0]$
[code]

Essentially tghe same. I have

openmpiwrappers-openmpi-1-4.el5.x86_64
openmpi-1.4.-4.el5.x86_64
openmpi-devel-1.4-4. el5-x86_64
openmpi-libs-1.4-4.el5 x86_64

installed. Both machines are 64 bit, one is 2 processor, 8 GB RAM, another is 16 processor 96GB RAM. Please help as II'd like to try Ray 2.0.0 on my project.

[bstamps]

Problem solved- I had forgotten to set my mpi version on the cluster using mpi-selector.

[jfpombert]

You need to install the openmpi package. For example, if you are using Fedora do a 'yum install openmpi openmpi-devel'. If the packages are already installed, make sure that they are in your path (you can add them to your .bash_profile). If you are trying to run Ray from a remote 'screen' job, make sure you source your .bash_profile too.

[bstamps]

I'm attempting to run Ray on our local cluster and after building I get an error:

Ray: error while loading shared libraries: libmpi_cxx.so.0: cannot open shared object file: No such file or directory

Thoughts?

[seb567]

sheepyuan: what do you mean by "The same to u！" ?

If you refer to the post of taliato, I don't think it is a good idea to disable shared memory
as it is the fastest way to do message passing between processes on the same machine.

Also, Open-MPI 1.4.2 is very old. The current stable release of Open-MPI is 1.6.1.

A lot of improvements were added in Open-MPI since 1.4.2 !

And gcc 4.1.2 is very old too although I don't think this will change much.

The same to u！

[sheepyuan]

I compiled Ray with openmpi 1.4.2, gcc version 4.1.2 20080704 (Red Hat 4.1.2-44), x86_64 architecture and run it with "mpirun -mca btl ^sm". The data is 3 simulated Illumina libraries comprising 52x coverage of a 225MB chromosome: 40x 500bp PE 95nt reads (inward facing), 8x 5kb mate paired 36nt reads (outward facing), 4x 10kb mate paired 36nt reads(outward facing).

Using 128 cores (16 8-core nodes), it runs fine up until the "Extending seeds" step. After a while the printing of the dots seem to slow down to glacial speeds. I've let it sit for several days with no progress. Is this an open mpi problem, you think? Any ideas on getting around this problem?

The same to u！

[seb567]

Thanks, it worked out and we got a high quality assembly, indeed the total memory usage was not by Ray but by another process.
After switching to a non-virtual machine the behaviour stopped, so maybe it was the VM configuration.

Cool.

What was the faulty process ?

[VidJa]

Thanks, it worked out and we got a high quality assembly, indeed the total memory usage was not by Ray but by another process.
After switching to a non-virtual machine the behaviour stopped, so maybe it was the VM configuration.

[seb567]

Hello,

First, I don't see how it can use 180 GB for 13 GB of data.
From your log, it says '680960 KiB' for core 29.
And your command indicates that you are using 30 cores.

35 * 680 MB = 20400 MB or about 20 GB

Second, you also need to add -s in front of Paired-end/T_R2_unpaired_2.fastq.

Do you have any log with more details because the only error lines are


rank 29 in job 53 bamicsb_51206 caused collective abort of all ranks
exit status of rank 29: killed by signal 11

And rank 29 said that before dying:

Rank 29: assembler memory usage: 680960 KiB

If you compile with DEBUG=y ASSERT=y, you may get more information out of this, depending on your system.

Ray 2.0.0 fails sometimes after several days of execution with messages like:

...
Rank 15 reached 9900 vertices from seed 0, flow 1
Speed RAY_SLAVE_MODE_EXTENSION 247 units/second
Rank 15: assembler memory usage: 686056 KiB
Rank 2 reached 10400 vertices from seed 0, flow 1
Speed RAY_SLAVE_MODE_EXTENSION 265 units/second
Rank 2: assembler memory usage: 686948 KiB
Speed RAY_SLAVE_MODE_EXTENSION 305 units/second
Rank 29: assembler memory usage: 680960 KiB
rank 29 in job 53 bamicsb_51206 caused collective abort of all ranks
exit status of rank 29: killed by signal 11

hardware: 35 core virtual machine with 230 GB memory, Ubuntu 10.04, mpich2
commandline:
mpiexec -np 30 Ray -k 41 -p Paired-end/T_R1_val_1.fastq Paired-end/T_R2_val_2.fastq -s Paired-end/T_R1_unpaired_1.fastq Paired-end/T_R2_unpaired_2.fastq -o Ray_trimmed_PE_S_k41

Typical memory usage about 180 GB at the time of the crash.
Input about 13GB of paired end and single end quality clipped (q25) Illumina reads (100bp)

Is this a hardware issue (maybe faulty memory banks) or something with Ray or the input material. Ive seen this behavious with other runs as well since I switched to 2.0.0, but never when using Ray 1.7.

[VidJa]

Ray 2.0.0 fails sometimes after several days of execution with messages like:

...
Rank 15 reached 9900 vertices from seed 0, flow 1
Speed RAY_SLAVE_MODE_EXTENSION 247 units/second
Rank 15: assembler memory usage: 686056 KiB
Rank 2 reached 10400 vertices from seed 0, flow 1
Speed RAY_SLAVE_MODE_EXTENSION 265 units/second
Rank 2: assembler memory usage: 686948 KiB
Speed RAY_SLAVE_MODE_EXTENSION 305 units/second
Rank 29: assembler memory usage: 680960 KiB
rank 29 in job 53 bamicsb_51206 caused collective abort of all ranks
exit status of rank 29: killed by signal 11

hardware: 35 core virtual machine with 230 GB memory, Ubuntu 10.04, mpich2
commandline:
mpiexec -np 30 Ray -k 41 -p Paired-end/T_R1_val_1.fastq Paired-end/T_R2_val_2.fastq -s Paired-end/T_R1_unpaired_1.fastq Paired-end/T_R2_unpaired_2.fastq -o Ray_trimmed_PE_S_k41

Typical memory usage about 180 GB at the time of the crash.
Input about 13GB of paired end and single end quality clipped (q25) Illumina reads (100bp)

Is this a hardware issue (maybe faulty memory banks) or something with Ray or the input material. Ive seen this behavious with other runs as well since I switched to 2.0.0, but never when using Ray 1.7.

[seb567]

Thanks for the input. The network test took 4 minutes, 35 seconds, that's why I inquired.

Is that too long ?

[snowbear24]

No, it is not possible to skip network testing. However, this steps usually only requires
a few seconds.

Thanks for the input. The network test took 4 minutes, 35 seconds, that's why I inquired.