[caddymob]

next steps...

error polling HP CQ with status WORK REQUEST FLUSHED ERROR

Seems to be an Open-MPI issue.

Maybe .. our system was under high load and the time and the infinband links on the LUSTRE file system may have been saturated, compounding the problem. Nevertheless I have been warned not to try again until I fix the swap issue or risk losing my cluster account. IT is really mad at me for this

SWAP: 881G Also, it is a bad idea to have 1 terabyte of swap memory, really.

Agreed, but this is something Ray did, I do not, to my knowledge have a way to specify how much swap to use. What can I do to prevent this?

How many MPI ranks do you have ?
What is the Ray command you used ?
How much physical memory is available on your cluster for your job ?

I was using 256 cores -- our system has 8-core nodes, each with 24GB of RAM.

My command:

###Parameterized PBS Script ####
#PBS -S /bin/bash
#PBS -N LUNDE.Ray4
#PBS -l nodes=256
#PBS -l walltime=25:00:00
#PBS -q normal
#PBS -j oe
#PBS -o LUNDE.Ray4.o
#PBS -M ---redacted---
#PBS -m abe

wd=/scratch/myfiles/LUNDE_ASSEMBLE/
cd $wd

use intel-openmpi-1.4.2
use Ray-1.2.0

mpirun -np 256 Ray -p $wd\Lunde_1.fastq $wd\Lunde_2.fastq -o Lunde-contigs

How many files / reads do you have ?

I have 2 FASTQ files, one for the forward read, one for the reverse. I have 140,174,250 paired reads, 105mers. Would it be better to split these up into smaller chunks or does that matter?

Thanks for your help!

[pallo]

Have you compiled other biotools with the intel compilar and compared performance against gcc?

hi drio,

Nope, I dont have any benchmarks...
Its just SOP here to try to compile with the "native" intel and then gcc if intel binaries fail.

[drio]

I dont think we need to debug the Intel compiled version, but if you want debug data, Ill provide as much as I can.

Have you compiled other biotools with the intel compilar and compared performance against gcc?

[seb567]

@pallo: sorry, I mixed caddymob with you.

I will do tests with the Intel compiler myself.

[pallo]

Hi seb567

Good news, looks like it works with gcc 4.4 + openmpi 1.4.2 (still ran out of memory though, but I will try a bigger set of nodes next time).

I dont think we need to debug the Intel compiled version, but if you want debug data, Ill provide as much as I can.

Which questions are you referring to? The ones you posted to caddymob?

Most of this is in my post: the data is ca 500M reads of illumina PE data on 16 lanes plus 3 runs of 454 (total 16x2 + 3 = 35 files). The nodes are 20x8core so Im running mpirun 160. Each node has 24G mem + 24G swap.

cheers

[seb567]

Hi pallo,

Can you provide additional information (see my questions above) ?

(You can send me an email too.)


Otherwise it is difficult for me to rule out the problem.


Thank you !

[pallo]

Hi,

Ray 1.2.0 crashes here, and I mean segfaults:

ray.err:

Loaded openMPI 1.4.2, compiled with intel11.1 (found in /opt/openmpi/1.4.2intel11.1/)
[q241:26410] *** Process received signal ***
[q241:26410] Signal: Segmentation fault (11)
[q241:26410] Signal code: Address not mapped (1)
[q241:26410] Failing at address: 0x9
[q241:26410] [ 0] /lib64/libpthread.so.0 [0x2b51e6679b10]
[q241:26410] [ 1] /bubo/home/h12/pallol/glob/sandbox/Ray-1.2.0/code/Ray(_ZN6Vertex15addOutgoingEdgeEmiP1\
1MyAllocator+
[q241:26410] [ 2] /bubo/home/h12/pallol/glob/sandbox/Ray-1.2.0/code/Ray(_ZN16MessageProcessor23call_TAG_\
OUT_EDGES_DAT
[q241:26410] [ 3] /bubo/home/h12/pallol/glob/sandbox/Ray-1.2.0/code/Ray(_ZN16MessageProcessor14processMe\
ssageEP7Messa
[q241:26410] [ 4] /bubo/home/h12/pallol/glob/sandbox/Ray-1.2.0/code/Ray(_ZN7Machine5startEv+0x1464) [0x4\
31c04]
[q241:26410] [ 5] /bubo/home/h12/pallol/glob/sandbox/Ray-1.2.0/code/Ray(main+0x89) [0x45a739]
[q241:26410] [ 6] /lib64/libc.so.6(__libc_start_main+0xf4) [0x2b51e68a3994]
[q241:26410] [ 7] /bubo/home/h12/pallol/glob/sandbox/Ray-1.2.0/code/Ray(__gxx_personality_v0+0xe9) [0x41\
7289]
[q241:26410] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 144 with PID 26410 on node q241 exited on signal 11 (Segmentation fault\
).
--------------------------------------------------------------------------
slurmd[q75]: *** JOB 220623 CANCELLED AT 2010-12-16T01:07:13 DUE TO NODE FAILURE ***
mpirun: abort is already in progress...hit ctrl-c again to forcibly terminate

I have 16 lanes of illumina data (about 500M 90bp PE reads).
I call Ray simply with:

mpirun -np 160 ~/sandbox/Ray-1.2.0/code/Ray -p l_1_1.fastq l_1_2.fastq -p l_2_1.fastq...

cheers
pallo

[seb567]

Hi !



error polling HP CQ with status WORK REQUEST FLUSHED ERROR

Seems to be an Open-MPI issue.



SWAP: 881G


Also, it is a bad idea to have 1 terabyte of swap memory, really.





How many MPI ranks do you have ?




How much physical memory is available on your cluster for your job ?




How many files / reads do you have ?



What is the Ray command you used ?



I wrote a post on my blog about Ray and large datasets:

I scale Ray on 512 cores and 1536 gigabytes of distributed memory on a human genome. Still doing some tests though before Christmas.

http://dskernel.blogspot.com/2010/12/ray-and-large-datasets.html

[caddymob]

I made a lot of people angry with Ray today....

Hi seb567,

So, I had Ray 1.2.0 installed on our cluster, complied with the same with Intel Compiler. The job starts to run, then is crashing.... Apparently today I took down 32 compute nodes according to the angry email I received from IT...

They sent me the following information about the job, I am told the problem is the huge amount of swap space I was using -- nearly 1TB!

Req[0]  TaskCount: 256  Partition: anon
Utilized Resources Per Task:  PROCS: 120.18  MEM: 2596M  SWAP: 881G
Avg Util Resources Per Task:  PROCS: 120.18
Max Util Resources Per Task:  PROCS: 237.11  MEM: 2596M  SWAP: 881G
Average Utilized Memory: 1641.30 MB
Average Utilized Procs: 48473.59
NodeSet=ONEOF:FEATURE:awesometown
NodeAccess: SINGLEJOB
NodeCount:  32

Here is the tail of my outfile:

Rank 230 is adding ingoing edges (reverse complement) 300001/1095111
[[18226,1],230][btl_openib_component.c:3224:handle_wc] from s54-5.local to: s54-15 error polling LP CQ with status RETRY EXCEEDED ERROR status number 12 for wr_id 246144128 opcode 0  vendor error 129 qp_idx 2
[s56-6.local:00534] 33 more processes have sent help message help-mpi-btl-openib.txt / pp retry exceeded
[s56-6.local:00534] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[[18226,1],232][btl_openib_component.c:3224:handle_wc] from s54-4.local to: s55-8 error polling HP CQ with status WORK REQUEST FLUSHED ERROR status number 5 for wr_id 13189504 opcode 32767  vendor error 244 qp_idx 0
[[18226,1],173][btl_openib_component.c:3224:handle_wc] from s54-12.local to: s55-6 error polling HP CQ with status WORK REQUEST FLUSHED ERROR status number 5 for wr_id 16690688 opcode 32767  vendor error 244 qp_idx 0
[[18226,1],237][btl_openib_component.c:3224:handle_wc] from s54-4.local to: s54-12 error polling LP CQ with status RETRY EXCEEDED ERROR status number 12 for wr_id 121666816 opcode 32767  vendor error 129 qp_idx 2
[s56-6.local:00534] 1 more process has sent help message help-mpi-btl-openib.txt / pp retry exceeded
[[18226,1],234][btl_openib_component.c:3224:handle_wc] from s54-4.local to: s56-3 error polling HP CQ with status WORK REQUEST FLUSHED ERROR status number 5 for wr_id 13992832 opcode 32767  vendor error 244 qp_idx 0
[[18226,1],175][btl_openib_component.c:3224:handle_wc] from s54-12.local to: s55-13 error polling LP CQ with status RETRY EXCEEDED ERROR status number 12 for wr_id 113500032 opcode 32767  vendor error 129 qp_idx 2
[s56-6.local:00534] 1 more process has sent help message help-mpi-btl-openib.txt / pp retry exceeded
[[18226,1],172][btl_openib_component.c:3224:handle_wc] from s54-12.local to: s55-13 error polling LP CQ with status RETRY EXCEEDED ERROR status number 12 for wr_id 117999232 opcode 32767  vendor error 129 qp_idx 2
[[18226,1],170][btl_openib_component.c:3224:handle_wc] from s54-12.local to: s55-13 error polling LP CQ with status RETRY EXCEEDED ERROR status number 12 for wr_id 59806336 opcode 1  vendor error 129 qp_idx 2
[s56-6.local:00534] 2 more processes have sent help message help-mpi-btl-openib.txt / pp retry exceeded
[[18226,1],238][btl_openib_component.c:3224:handle_wc] from s54-4.local to: s55-16 error polling HP CQ with status WORK REQUEST FLUSHED ERROR status number 5 for wr_id 17854848 opcode 128  vendor error 244 qp_idx 0
[[18226,1],215][btl_openib_component.c:3224:handle_wc] from s54-7.local to: s54-12 error polling LP CQ with status RETRY EXCEEDED ERROR status number 12 for wr_id 115086208 opcode 0  vendor error 129 qp_idx 2
[s56-6.local:00534] 1 more process has sent help message help-mpi-btl-openib.txt / pp retry exceeded
[[18226,1],197][btl_openib_component.c:3224:handle_wc] from s54-9.local to: s54-12 error polling LP CQ with status RETRY EXCEEDED ERROR status number 12 for wr_id 102530048 opcode 32767  vendor error 129 qp_idx 2
[s56-6.local:00534] 1 more process has sent help message help-mpi-btl-openib.txt / pp retry exceeded
[[18226,1],199][btl_openib_component.c:3224:handle_wc] from s54-9.local to: s54-12 error polling LP CQ with status RETRY EXCEEDED ERROR status number 12 for wr_id 61565696 opcode 32767  vendor error 129 qp_idx 2
[s56-6.local:00534] 1 more process has sent help message help-mpi-btl-openib.txt / pp retry exceeded
[[18226,1],171][btl_openib_component.c:3224:handle_wc] from s54-12.local to: s54-15 error polling HP CQ with status WORK REQUEST FLUSHED ERROR status number 5 for wr_id 9502080 opcode 128  vendor error 244 qp_idx 0
[[18226,1],196][btl_openib_component.c:3224:handle_wc] from s54-9.local to: s54-2 error polling HP CQ with status WORK REQUEST FLUSHED ERROR status number 5 for wr_id 13076352 opcode 128  vendor error 244 qp_idx 0
[[18226,1],169][btl_openib_component.c:3224:handle_wc] from s54-12.local to: s55-8 error polling HP CQ with status WORK REQUEST FLUSHED ERROR status number 5 for wr_id 21503744 opcode 128  vendor error 244 qp_idx 0
[[18226,1],174][btl_openib_component.c:3224:handle_wc] from s54-12.local to: s56-6 error polling HP CQ with status WORK REQUEST FLUSHED ERROR status number 5 for wr_id 11039872 opcode 128  vendor error 244 qp_idx 0
[[18226,1],194][btl_openib_component.c:3224:handle_wc] from s54-9.local to: s55-8 error polling HP CQ with status WORK REQUEST FLUSHED ERROR status number 5 for wr_id 12326400 opcode 32767  vendor error 244 qp_idx 0
=>> PBS: job killed: node 26 (s54-7) requested job terminate, 'EOF' (code 1099) - internal or network failure attempting to communicate with sister MOM's
mpirun: abort is already in progress...hit ctrl-c again to forcibly terminate

15 total processes killed (some possibly by mpirun during cleanup)

So, now IT is really angry with me, I'm kinda proud of myself, but I still have a genome to try to assemble. What do you suggest?

[seb567]

I had to have my IT department compile and install Ray on our cluster and did not notice they used the old version of Ray. Thanks for pointing this out. I have requested the 1.0.0 version to be installed, along with questions about the MPI version available.

I think you are fine with Open-MPI 1.4.2 compiled with Intel Compiler (use intel-openmpi-1.4.2).

Any idea when the new version will be available for mammalian genomes? Looking forward to it!

Before Friday, for sure.

I will keep you posted on my progress once I get the new version up and running.

Thank you for your updates !

[caddymob]

Many thanks seb567!

I had to have my IT department compile and install Ray on our cluster and did not notice they used the old version of Ray. Thanks for pointing this out. I have requested the 1.0.0 version to be installed, along with questions about the MPI version available.

Any idea when the new version will be available for mammalian genomes? Looking forward to it!

I will keep you posted on my progress once I get the new version up and running. Thanks again!

[seb567]

Really excited to try out Ray! I first tried to grab the example datasets, but the links are dead.. getting a 550 error, no such file..

NCBI moved their infrastructure from .fastq to .sra files.

My favorite toy dataset is SRA001125, Illumina data of E. coli K-12 MG1655.

Search SRA001125 and you'll find it.

So then I went ahead and tried to assemble my own genome of a very homozygus (>96%) mammalian genome sequenced on illumina with paired 105bp reads. Ray is failing and I do not understand why.

Many reasons can explain that.



Ray ran for just over 2 hours on 256 cores before dying. Here are my commands:

use intel-openmpi-1.4.2
use Ray-0.1.0

You use Ray 0.1.0 ! Try Ray 1.0.0, I assure you it has many fixes included.

v. 1.0.0 is the release with the most changes to date.

http://sourceforge.net/apps/mediawiki/denovoassembler/index.php?title=ChangeLog#v._1.0.0

mpirun -np 256 Ray -p $wd\Lunde_1.fq $wd\Lunde_2.fq -o Lunde-contigs

Do you have acces to a SMP machine with 256 processor cores ?!

If so, I envy you.

And the output I get:

Code:

Rank 0 welcomes you to the MPI_COMM_WORLD.
Rank 0: website -> http://denovoassembler.sf.net/
Rank 0: using Open-MPI 1.2.7

So basically, you use a bad mix of software: intel-openmpi-1.4.2 with Ray compiled against Open-MPI 1.2.7.

This will surely fail !

Rank 0 is running as UNIX process 4193 on s28-2.local (MPI version 2.0)

Last standard is MPI 2.2 from 2009. MPICH2 and Open-MPI 1.4.3 comply with MPI 2.2.

Ray works with MPI 2.0 too, I guess.

Rank 0: Ray 0.1.0 is running

As I said, 0.1.0 is defunct. Embrace the new 1.0.0.

The next release is coming soon.

Ray for large genomes is on its way !

My last test on human chromosome 1 (the largest) with one library of
length 200 and another of length 400 shows great success:


Rank 0: 69173 contigs/205904915 nucleotides

Rank 0 reports the elapsed time, Sun Nov 28 20:38:22 2010
---> Step: Collection of fusions
Elapsed time: 1 minutes, 16 seconds
Since beginning: 8 hours, 22 minutes, 4 seconds

Elapsed time for each step, Sun Nov 28 20:38:22 2010

Beginning of computation: 3 seconds
Distribution of sequence reads: 25 minutes, 3 seconds
Distribution of vertices: 1 minutes, 16 seconds
Calculation of coverage distribution: 1 seconds
Distribution of edges: 1 minutes, 30 seconds
Indexing of sequence reads: 2 seconds
Computation of seeds: 10 minutes, 39 seconds
Computation of library sizes: 4 minutes, 51 seconds
Extension of seeds: 7 hours, 33 minutes, 36 seconds
Computation of fusions: 3 minutes, 47 seconds
Collection of fusions: 1 minutes, 16 seconds
Completion of the assembly: 8 hours, 22 minutes, 4 seconds

Rank 0 wrote r4068-human.CoverageDistribution.txt
Rank 0 wrote r4068-human.Library0.txt
Rank 0 wrote r4068-human.Library1.txt
Rank 0 wrote r4068-human.fasta
Rank 0 wrote r4068-human.ReceivedMessages.txt

Is there something wrong with my configuration?

You configuration is erroneous in two independent ways.

1. You are using Ray 0.1.0, not Ray 1.0.0.

2. You are running a executable compiled against Open-MPI 1.2.7 with, I believe, Open-MPI 1.4.2.


Thank you for your interest in Ray !

"The Ray of light is coming to life, and the Ray of darkness is fading away."

-Seb

[caddymob]

processes aborted

Really excited to try out Ray! I first tried to grab the example datasets, but the links are dead.. getting a 550 error, no such file..

So then I went ahead and tried to assemble my own genome of a very homozygus (>96%) mammalian genome sequenced on illumina with paired 105bp reads. Ray is failing and I do not understand why.

Ray ran for just over 2 hours on 256 cores before dying. Here are my commands:

use intel-openmpi-1.4.2
use Ray-0.1.0

mpirun -np 256 Ray -p $wd\Lunde_1.fq $wd\Lunde_2.fq -o Lunde-contigs

And the output I get:

Rank 0 welcomes you to the MPI_COMM_WORLD.
Rank 0: website -> http://denovoassembler.sf.net/
Rank 0: using Open-MPI 1.2.7
Rank 0 is running as UNIX process 4193 on s28-2.local (MPI version 2.0)
.
.
.
Rank 243 is running as UNIX process 4231 on s26-4.local (MPI version 2.0)
Rank 0: I am the master among 256 ranks in the MPI_COMM_WORLD.

Rank 0: Ray 0.1.0 is running
Rank 0: operating system is Linux (during compilation)

LoadPairedEndReads
 Left sequences: /scratch/jcorneveaux/LUNDE_ASSEMBLE/Lunde_1.fq
 Right sequences: /scratch/jcorneveaux/LUNDE_ASSEMBLE/Lunde_2.fq
 Average length: auto
 Standard deviation: auto

k-mer size: 21
 --> Number of k-mers of size 21: 4398046511104
  *** Note: A lower k-mer size bounds the memory usage. ***


Rank 0 loads /scratch/jcorneveaux/LUNDE_ASSEMBLE/Lunde_1.fq.
Rank 0 has 140174250 sequences to distribute.
Rank 0 distributes sequences, 1/140174250
mpirun noticed that job rank 1 with PID 4194 on node s28-2 exited on signal 15 (Terminated). 
254 additional processes aborted (not shown)
1 process killed (possibly by Open MPI)Rank 0 welcomes you to the MPI_COMM_WORLD.
Rank 0: website -> http://denovoassembler.sf.net/
Rank 0: using Open-MPI 1.2.7
Rank 0 is running as UNIX process 4193 on s28-2.local (MPI version 2.0)

Is there something wrong with my configuration?

[jfpombert]

Ok, great, i'll just change the extensions.

Un gros merci!

JF

[seb567]

@jfpombert

I forgot to provide a fix.

quick fix:

ln -s 100420_s_7_1_seq_GKD-1.txt 100420_s_7_1_seq_GKD-1.txt.fastq

ln -s 100420_s_7_2_seq_GKD-1.txt 100420_s_7_2_seq_GKD-1.txt.fastq

mpirun -np 8 \
Ray \
-p \
100420_s_7_1_seq_GKD-1.txt.fastq \
100420_s_7_2_seq_GKD-1.txt.fastq \
-s \
FQH37LX05.sff \
-s \
FQH37LX06.sff \
-s \
FTX7HMM01.sff \
-s \
FU6LJ3H01.sff \
-s \
FWZEL0L06.sff \
-o \
test.txt

or using bzip2, you will save precious space:

bzip2<100420_s_7_1_seq_GKD-1.txt>100420_s_7_1_seq_GKD-1.txt.fastq.bz2

bzip2<100420_s_7_2_seq_GKD-1.txt>100420_s_7_2_seq_GKD-1.txt.fastq.bz2

Ray \
-p \
100420_s_7_1_seq_GKD-1.txt.fastq.bz2 \
100420_s_7_2_seq_GKD-1.txt.fastq.bz2 \
-s \
FQH37LX05.sff \
-s \
FQH37LX06.sff \
-s \
FTX7HMM01.sff \
-s \
FU6LJ3H01.sff \
-s \
FWZEL0L06.sff \
-o \
test.txt

Thank you for providing a detailed report of what you did.