[yaximik]

Oops The machine has 96 GB memory

[yaximik]

More help needed....

Hi,

I tried to run Ray (maxkmer 32) on 2 x quad core RHEl58 with hyper-threading enabled:

mpiexec -n 16 Ray <Ray.conf> and got the error:

........
Loader::load] File: /media/FantomHD/Data/MiSeq/SC/AdQ30/SC-MILLib1-Herc2s10cFr1Fr2run2R1AdQ30.fastq (please wait...)
[Loader::load] File: /media/FantomHD/Data/MiSeq/SC/AdQ30/SC-MILLib1-Herc2s10cFr1Fr2run2R1AdQ30.fastq (please wait...)
[Loader::load] File: /media/FantomHD/Data/MiSeq/SC/AdQ30/SCPfx3s25cFr3-150-200run1R1AdQ30.fastq (please wait...)
[Loader::load] File: /media/FantomHD/Data/MiSeq/SC/AdQ30/SCPfx3s25cFr3-150-200run1R1AdQ30.fastq (please wait...)
[Loader::load] File: /media/FantomHD/Data/MiSeq/SC/AdQ30/SCPfx3s25cFr3-150-200run2R1AdQ30.fastq (please wait...)
[Loader::load] File: /media/FantomHD/Data/MiSeq/SC/AdQ30/SCPfx3s25cFr3-150-200run2R1AdQ30.fastq (please wait...)
[Loader::load] File: /media/FantomHD/AssRefMap/SC/SCold/SColdAll.fasta (please wait...)
[Loader::load] File: /media/FantomHD/AssRefMap/SC/SCold/SColdAll.fasta (please wait...)
[Loader::load] File: /media/FantomHD/AssRefMap/SC/SCold/SCallSanger.fasta (please wait...)
[Loader::load] File: /media/FantomHD/AssRefMap/SC/SCold/SCallSanger.fasta (please wait...)
[Loader::load] File: /home/yaximik/AssRefMap/SC/minia/SCMiSeqAllFGMGPGIGclean_k27.contigs.fasta (please wait...)
[G5NNJN1:07040] *** Process received signal ***
[G5NNJN1:07040] Signal: Segmentation fault (11)
[G5NNJN1:07040] Signal code:  (128)
[G5NNJN1:07040] Failing at address: (nil)
--------------------------------------------------------------------------
mpiexec noticed that process rank 0 with PID 7040 on node G5NNJN1 exited on signal 11 (Segmentation fault).

The last file loaded was a file with fasta contigs from another assembler (minia). Does this mean contigs from other assemblers cannot be used in Ray?

[yaximik]

A bit confused about parameters - help...

Hi,
What is the meaning of averageOuterDistance and standardDeviation for paired end files? Is it just average read length in the dataset?
If so, then why it is not required for single read file?
If not, is it an average fragment length in the library? Such as surmised from BioAnalyzer trace, for example?
If so, then default autocalc may give very wrong estimate, could it? For example, one of my paired read runs was done with a library of 600 bp +/- 15%, but during assembly autocalc estimate was something 150 bp - how this can be so much off?

[seb567]

Thanks guys! Sorry for the great delay in my reply. I have been at sea.

We are working with invertebrate genomes of unknown size but we're after the mitochondrial genomes for this project and they've been shaking out OK on our 80 CPU cluster.

Best,
Kevin

Cool !

Me again. seb567, the link to the visualization tool you posted (http://genome.ulaval.ca/corbeillab/Ray-Cloud-Browser) is broken.

It's the IT at my institution that failed I guess. Anyway, I have set up DNS canonical names (CNAME), which are more robust.

All my Ray Cloud Browser deployments are in the cloud.

4 demos (these are canonical names to cloud instances):

E. coli on a t1.micro spot instance in Amazon EC2

Some microbiomes of a colleague on 1 t1.micro spot instance in Amazon EC2

E. coli on a small Linux Virtual Machine in Windows Azure

A vertebrate genome (American eel) on a Silver instance in IBM SmartCloud


In all these links, raytrek.com can be replaced by boisvert.info (example: browser.cloud.raytrek.com and browser.cloud.boisvert.info are the same instance).

[kmkocot]

Me again. seb567, the link to the visualization tool you posted (http://genome.ulaval.ca/corbeillab/Ray-Cloud-Browser) is broken.

[kmkocot]

Thanks guys! Sorry for the great delay in my reply. I have been at sea.

We are working with invertebrate genomes of unknown size but we're after the mitochondrial genomes for this project and they've been shaking out OK on our 80 CPU cluster.

Best,
Kevin

[gringer]

Assemblathon 2 results indicate that Ray is really good with gene content, but its scaffolder is way too conservative.

FWIW, I've been able to improve on Ray assemblies a little by running the scaffolds through AMOS' minimus2 (in the default all-vs-all mode). That was able to pick up a few more SNPs and merge contigs that were almost identical.

[seb567]

... as long as your computer cluster is up to the challenge. While Ray is quite memory efficient, you may have a bit of difficulty assembling a human genome using Ray on a small cluster or desktop.

Sure.

And it depends what is implied by "assembling a human genome".

Assemblathon 2 results indicate that Ray is really good with gene content, but its scaffolder is way too conservative.

Our group is mostly into bacterial genomes and human microbiomes.

See our recent paper: http://genomebiology.com/2012/13/12/R122/abstract


Thanks for the feedback !

-Sébastien

[gringer]

The library was made with a Nextera kit and sequenced using the new 2 X 250 reagent kits. The average size distribution of my library was around 500 bp but some smaller fragments were present. For those fragments, the read pairs will at least partially overalp. Does Ray have a problem when the two members of a pair of reads overlap? Should I treat the data as non paired end?

Ray will be fine with those.

... as long as your computer cluster is up to the challenge (which depends more on the target genome size than the number of input reads). While Ray is quite memory efficient, you may have a bit of difficulty assembling a human genome using Ray on a small cluster or desktop.

[seb567]

Hi, I have a question here.
Does Ray expect the sequence order in two pair-end files the same?

yes

I ask because the sequence order in my two pair-end fastq files happened to be different the last time. They are indeed pair files, just the sequences are in different order. And I ran these pair-end files with Ray, got output1. After I realized the sequence order are not the same, I sort the fastq files to make them same order. I then re-ran Ray, got output2. It seems the two run results are different.
Thank you.

Ray need both files to list sequences in the same order.

By default, most sequencing technologies do that by default, and the dominant sequencing technology is just like that too.

Thanks for the feedback !

-Sébastien

[seb567]

Hi all,

Quick question: I have a paired-end data from a MiSeq that I would like to assemble in Ray. The library was made with a Nextera kit and sequenced using the new 2 X 250 reagent kits. The average size distribution of my library was around 500 bp but some smaller fragments were present. For those fragments, the read pairs will at least partially overalp. Does Ray have a problem when the two members of a pair of reads overlap? Should I treat the data as non paired end?

Thanks!
Kevin

Hi,

Ray will be fine with those.

I suggest you run something like this:

mpiexec -n 16 Ray -k 71 -p file_R1.fastq.gz file_R2.fastq.gz -o MiSeq+Ray


Also, you can use Ray Cloud Browser too to visualize your assembly in your web browser.

Demo: http://genome.ulaval.ca/corbeillab/Ray-Cloud-Browser


p.s.: you'll need to compile with this:

make MAXKMERLENGTH=96 HAVE_LIBZ=y

---
-Sébastien

[cwzkevin]

Hi, I have a question here.
Does Ray expect the sequence order in two pair-end files the same?
I ask because the sequence order in my two pair-end fastq files happened to be different the last time. They are indeed pair files, just the sequences are in different order. And I ran these pair-end files with Ray, got output1. After I realized the sequence order are not the same, I sort the fastq files to make them same order. I then re-ran Ray, got output2. It seems the two run results are different.
Thank you.

[kmkocot]

Hi all,

Quick question: I have a paired-end data from a MiSeq that I would like to assemble in Ray. The library was made with a Nextera kit and sequenced using the new 2 X 250 reagent kits. The average size distribution of my library was around 500 bp but some smaller fragments were present. For those fragments, the read pairs will at least partially overalp. Does Ray have a problem when the two members of a pair of reads overlap? Should I treat the data as non paired end?

Thanks!
Kevin

[cwzkevin]

Thank you very much for the detail, I understand now. Great appreciate it!

Sorry, I am not a programmer. (I think a programmer should already know there could be distributed Makefiles instead of the one that I edited. ^_^)

Now, it is time to try it out.
Thanks!

[seb567]

Hi !

Thanks for the logs, that really helps understanding what's going
on.

You can build Ray with these options without any Makefile modification:

$ make clean
$ make MAXKMERLENGTH=96 HAVE_LIBZ=y HAVE_LIBBZ2=y PREFIX=bin
$ make install

$ mpiexec -n 1 bin/Ray -version
Ray version 2.1.0
License for Ray: GNU General Public License version 3
RayPlatform version: 1.1.0
License for RayPlatform: GNU Lesser General Public License version 3

MAXKMERLENGTH: 96 <=========== Here you go !
KMER_U64_ARRAY_SIZE: 3
Maximum coverage depth stored by CoverageDepth: 4294967295
MAXIMUM_MESSAGE_SIZE_IN_BYTES: 4000 bytes
FORCE_PACKING = n
ASSERT = n
HAVE_LIBZ = y <=========== Here you go !
HAVE_LIBBZ2 = y <=========== Here you go !
CONFIG_PROFILER_COLLECT = n
CONFIG_CLOCK_GETTIME = n
__linux__ = y
_MSC_VER = n
__GNUC__ = y
RAY_32_BITS = n
RAY_64_BITS = y
MPI standard version: MPI 2.1
MPI library: Open-MPI 1.5.4
Compiler: GNU gcc/g++ 4.7.2 20120921 (Red Hat 4.7.2-2)

Oh, I see. With below, it is good now.

$ make PREFIX=bin HAVE_LIBZ=y HAVE_LIBBZ2=y

Seems my below question is more related to common sense of linux/compiler instead of Ray:
Q: What is the difference between Method 1 and Method 2, shouldn't they be the same?
Method 1: I edited the Makefile, changed to "HAVE_LIBZ = y", "HAVE_LIBBZ2 = y", then $ make PREFIX=bin
Method 2: $ make PREFIX=bin HAVE_LIBZ=y HAVE_LIBBZ2=y
Thanks.

Now, if why editing the Makefile fails ?

There are many Makefile files actually (distributed Makefiles)

Ray-v2.1.0/Makefile
Ray-v2.1.0/code/Makefile
Ray-v2.1.0/code/*/Makefile (23)

When you provide the variables in the make command line, they will
be given to child processes because they are exported. However,
variables within a Makefile are not exported.

It fails because of this:

Ray-v2.1.0/code/plugin_SequencesLoader/Makefile:

SequencesLoader-$(HAVE_LIBBZ2) += plugin_SequencesLoader/BzReader.o
SequencesLoader-$(HAVE_LIBBZ2) += plugin_SequencesLoader/FastqBz2Loader.o
SequencesLoader-$(HAVE_LIBZ) += plugin_SequencesLoader/FastqGzLoader.o

These configuration options are used by the Makefiles, but also by the
C++ code. For example, HAVE_LIBZ is valued to y in all the Makefiles,
and the -D HAVE_LIBZ passed to gcc defines HAVE_LIBZ in all C++ files
too.

If you really want to edit the Makefile, you have to do it like this:

--- Ray-v2.1.0/Makefile 2012-10-30 18:29:34.000000000 -0400
+++ Ray-v2.1.0-copy/Makefile 2012-11-04 20:05:54.099217300 -0500
@@ -33,13 +33,13 @@
# needs libz
# set to no if you don't have libz
# y/n
-HAVE_LIBZ = n
+export HAVE_LIBZ = y

# support for .bz2 files
# needs libbz2
# set to no if you don't have libbz2
# y/n
-HAVE_LIBBZ2 = n
+export HAVE_LIBBZ2 = y

# use Intel's compiler
# the name of the Intel MPI C++ compiler is mpiicpc

If you know programming, you can send me a patch that fixes this bug
in the Makefile that would add 'export ' in front of build options.

If you have other questions regarding the Ray build system,
let me know.


Otherwise, I'll put this in my patchwork queue !

***
Cheers, Sébastien

Yes, I edited the Makefile. Changed was made as below:

MAXKMERLENGTH = 96
HAVE_LIBZ = y
HAVE_LIBBZ2 = y

My make command is just simple as

$ make PREFIX=bin

My system is

$ uname -mrs
Linux 2.6.18-308.8.2.el5 x86_64
$ lsb_release -a
LSB Version:    :core-4.0-amd64:core-4.0-ia32:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-ia32:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-ia32:printing-4.0-noarch
Distributor ID: RedHatEnterpriseServer
Description:    Red Hat Enterprise Linux Server release 5.8 (Tikanga)
Release:        5.8
Codename:       Tikanga

Here is the link for full output http://pastebin.com/Kf35v5SK

Thank you for your help.