[seb567]

Great work Seb!

Did you run 1.7 on the parrot data (Assemblathon 2) to compare with Ray 1.6.1?

Of course !

(I am very thankful to the Assemblathon 2 organizers and data suppliers.)


cat k31-Ray-Bird-20110921-1/OutputNumbers.txt

Contigs >= 100 nt
Number: 88826
Total length: 1169161521
Average: 13162
N50: 41098
Median: 3368
Largest: 465622

Contigs >= 500 nt
Number: 68550
Total length: 1164709611
Average: 16990
N50: 41306
Median: 6862
Largest: 465622

Scaffolds >= 100 nt
Number: 47279
Total length: 1270995781
Average: 26882
N50: 567125
Median: 725
Largest: 3236250

Scaffolds >= 500 nt
Number: 27408
Total length: 1266700501
Average: 46216
N50: 571612
Median: 2137
Largest: 3236250


A lot of contigs end simply because there is no coverage.


I will post something on my blog soon describing the requirements and the outcomes.

See https://github.com/sebhtml/assemblathon-2-ray for the full command line required.

[lletourn]

Great work Seb!

Did you run 1.7 on the parrot data (Assemblathon 2) to compare with Ray 1.6.1?

[seb567]

Ray 1.7

Dear assemblers,


Ray v1.7 is now available (and the Assemblathon 2 is over).


Summary of what changed:

* MANUAL_PAGE.txt replaces the PDF manual.
* Output files are written to the directory specified by -o (previously it was a file prefix)
* Round-robin reception of messages
* Bloom filter
* Illumina mate-pairs support
* Job checkpointing
* New scaffolding algorithm
* New assembly engine for the extension of seeds with mate-pairs (NovaEngine)
* Parallel file partitionning
* Network latency testing
* Compiles cleanly on 32-bit systems

All the changes:

https://sourceforge.net/mailarchive/forum.php?thread_name=C4217F2ABFA5D44BA8F3DCD899E5B0C8F267700754%40EXCH-MBX-F.ulaval.ca&forum_name=denovoassembler-users

http://github.com/sebhtml/ray

[seb567]

----------CoverageDistributionAnalysis.txt----------[/B]
k-mer length: 21
Lowest coverage observed: 1
MinimumCoverage: 131
PeakCoverage: 131
RepeatCoverage: 139
Number of k-mers with at least MinimumCoverage: 2305986 k-mers
Estimated genome length: 1152993 nucleotides
Percentage of vertices with coverage 1: 24.3365 %
DistributionFile: RayOutput.CoverageDistribution.txt


----------CoverageDistributionAnalysis.txt----------
k-mer length: 21
Lowest coverage observed: 1
MinimumCoverage: 31
PeakCoverage: 172
RepeatCoverage: 178
Number of k-mers with at least MinimumCoverage: 55438516 k-mers
Estimated genome length: 27719258 nucleotides
Percentage of vertices with coverage 1: 22.5815 %
DistributionFile: RayOutput.CoverageDistribution.txt

This means that if I had used a single processor (-np 1, mpi), would I have got the same output?.
In case I run Ray twice (using only one processor), changing the order of the reads for the second run (in the input file), does the output of Ray should be different between the two runs, as well?

Something is wrong with your coverage distributions.

Can you post PREFIX.CoverageDistribution.txt to http://pastebin.com ?


If you run Ray with -np 1 (1 compute core) on the same data twice, you will obtain the same
result.

When using more than 1 compute core, assemblies can change because of the order of the messages.

[kail]

How big is your compute cluster ?

Actually is a single machine, a Mac Pro 2 x 2.4 GHz Quad-Core Intel Xeon, with 64GB of RAM and 2TB of disk.

That won't work well with Ray. Furthermore, using both libraries simultaneously give Ray much more information that helps assemble the genome.

I tried to do that, but my "cluster" ran out of memory.

How many nucleotides were outputted by Ray ?

I really don’t know how many nucleotides were outputted by Ray for the assembly of Library 2… I guest 26311531.

Can you provide the content of CoverageDistributionAnalysis.txt files for both of them ?

Here are some outputs of Ray for both libraries:

Library 1

----------LibraryStatistics.txt ----------
File: Paired-Ends.fastq
NumberOfSequences: 95607712

Total: 95607712

NumberOfPairedLibraries: 1

LibraryNumber: 0
InputFormat: Interleaved,Paired
DetectionType: Manual
File: Paired-Ends.fastq
NumberOfSequences: 95607712
AverageOuterDistance: 500
StandardDeviation: 50

----------OutputNumbers.txt----------
Number of contigs: 573667
Total length of contigs: 88436065
Number of contigs >= 500 nt: 1550
Total length of contigs >= 500 nt: 936979
Number of scaffolds: 573666
Total length of scaffolds: 88436427
Number of scaffolds >= 500 nt: 1549
Total length of scaffolds >= 500: 937341

----------CoverageDistributionAnalysis.txt----------
k-mer length: 21
Lowest coverage observed: 1
MinimumCoverage: 131
PeakCoverage: 131
RepeatCoverage: 139
Number of k-mers with at least MinimumCoverage: 2305986 k-mers
Estimated genome length: 1152993 nucleotides
Percentage of vertices with coverage 1: 24.3365 %
DistributionFile: RayOutput.CoverageDistribution.txt

Library 2

----------LibraryStatistics.txt ----------
File: Mate-Pair.fastq
NumberOfSequences: 85198684

Total: 85198684

NumberOfPairedLibraries: 1

LibraryNumber: 0
InputFormat: Interleaved,Paired
DetectionType: Manual
File: Mate-Pair.fastq
NumberOfSequences: 85198684
AverageOuterDistance: 2200
StandardDeviation: 200

----------OutputNumbers.txt----------
Number of contigs: 28224
Total length of contigs: 26311531
Number of contigs >= 500 nt: 5609
Total length of contigs >= 500 nt: 23362817
Number of scaffolds: 27056
Total length of scaffolds: 27898655
Number of scaffolds >= 500 nt: 4441
Total length of scaffolds >= 500: 24949941

----------CoverageDistributionAnalysis.txt----------
k-mer length: 21
Lowest coverage observed: 1
MinimumCoverage: 31
PeakCoverage: 172
RepeatCoverage: 178
Number of k-mers with at least MinimumCoverage: 55438516 k-mers
Estimated genome length: 27719258 nucleotides
Percentage of vertices with coverage 1: 22.5815 %
DistributionFile: RayOutput.CoverageDistribution.txt

Ray will generate different assemblies with the same input. This is caused by the randomness of the order in which messages are sent during the computation.

But the assemblies are mostly equivalent.

This means that if I had used a single processor (-np 1, mpi), would I have got the same output?.
In case I run Ray twice (using only one processor), changing the order of the reads for the second run (in the input file), does the output of Ray should be different between the two runs, as well?

[seb567]

Dear all,

I have two libraries of genomic DNA sequenced with Illumina. I want to perform a de novo assembly using Ray (v 1.6.0).

Library 1 (is paired-ends, 500 ± 50)
95.607.712 reads (47.803.856 pairs)
573.667 contigs were built by Ray
It took 1 day, 24 minutes and 16 seconds to perform the assembly

Library 2 (is mate-pair, 2200 ± 200)
85.198.684 reads (42.599.342 pairs)
28.224 contigs were built by Ray
It took 3 hours, 7 minutes and 44 seconds to perform the assembly

My problem is that my cluster is not big enough to run Ray using the two libraries simultaneously.

How big is your compute cluster ?

So, what I did was run Ray in two steps, first, using only the Library 1 and then Library 2. Now, the idea is to run again Ray but using as input the generated contigs in the two previous steps (as single-end reads). Therefore my question is, can I do this?,

That won't work well with Ray. Furthermore, using both libraries simultaneously give Ray much more information that helps assemble the genome.

will work?

I don't think it will work well.

(All under the assumption that this way I will use less memory)

My second question is why it took only 3 hours the assembly of the Library 2?

How many nucleotides were outputted by Ray ?

Can you provide the content of CoverageDistributionAnalysis.txt files for both of them ?

There is something wrong with the assembly?

I don't know, you did not provide much information describing your assemblies aside from the number of contigs.

Should I be concerned?

You should definitely run Ray on all data at once.

You have < 100 M reads.

My last question is related with the previous one. In order to double-check the assembly of library 2, I run it again (I used the same parameters as before, defaults ones). The result in time was the same but the number of contigs varied slightly between runs (I have not checked the contigs sequences). So, Ray is deterministic software? Or every time it is used, will generate a different output…

Ray will generate different assemblies with the same input. This is caused by the randomness of the order in which messages are sent during the computation.

But the assemblies are mostly equivalent.

Thanks in advance.

***
Sébastien Boisvert

http://github.com/sebhtml/ray

[kail]

Dear all,

I have two libraries of genomic DNA sequenced with Illumina. I want to perform a de novo assembly using Ray (v 1.6.0).

Library 1 (is paired-ends, 500 ± 50)
95.607.712 reads (47.803.856 pairs)
573.667 contigs were built by Ray
It took 1 day, 24 minutes and 16 seconds to perform the assembly

Library 2 (is mate-pair, 2200 ± 200)
85.198.684 reads (42.599.342 pairs)
28.224 contigs were built by Ray
It took 3 hours, 7 minutes and 44 seconds to perform the assembly

My problem is that my cluster is not big enough to run Ray using the two libraries simultaneously. So, what I did was run Ray in two steps, first, using only the Library 1 and then Library 2. Now, the idea is to run again Ray but using as input the generated contigs in the two previous steps (as single-end reads). Therefore my question is, can I do this?, will work? (All under the assumption that this way I will use less memory)

My second question is why it took only 3 hours the assembly of the Library 2? There is something wrong with the assembly? Should I be concerned?

My last question is related with the previous one. In order to double-check the assembly of library 2, I run it again (I used the same parameters as before, defaults ones). The result in time was the same but the number of contigs varied slightly between runs (I have not checked the contigs sequences). So, Ray is deterministic software? Or every time it is used, will generate a different output…

Thanks in advance.

[seb567]

Thanks for the helpful responses, Sébastien, and for addressing this problem in the next version.
Meanwhile, are the files PREFIX.LibraryX.txt made automatically, please? I cannot see them, whether I set a mean and sd for insert size in the command line, or not.

These are automatically generated.

Can you send an email to denovoassembler-users AT lists.sourceforge.net with:

You Ray command.

Thank you.

[seb567]

Hi Sébastien,

I have a question, Ray appears to be automatically setting the minimum kmer coverage to 1 minus the peak coverage, is that true, and what's the reason for this? I apparently have absurd peak coverage of 626, even with k=61. Could this be why my assembly was poor the first time with k=31 above?

I'm thinking I'm going to cancel my job and drop that min coverage.

Can you post somewhere the content of CoverageDistribution.txt and send an email
with that information to denovoassembler-users AT lists.sourceforge.net.

[Wallysb01]

Hi Sébastien,

I have a question, Ray appears to be automatically setting the minimum kmer coverage to 1 minus the peak coverage, is that true, and what's the reason for this? I apparently have absurd peak coverage of 626, even with k=61. Could this be why my assembly was poor the first time with k=31 above?

I'm thinking I'm going to cancel my job and drop that min coverage.

[habm]

mate-pair libraries

Thanks for the helpful responses, Sébastien, and for addressing this problem in the next version.
Meanwhile, are the files PREFIX.LibraryX.txt made automatically, please? I cannot see them, whether I set a mean and sd for insert size in the command line, or not.

[Wallysb01]

So, peak coverage was 913.

Interestingly it estimated the genome size was ~1.8Mb, when it should be closer to ~1.8Gb.

I also specified the insert size of both the paired end libraries which were done with the Illumina TruSeq v3. So fair the insert sizes where both about 450 bps with 104 bp paired end reads. We may end up increasing the number of libraries we sequence to include larger inserts to improve the contigs and actually get scaffolds.

I was also planning back blast to filter out contigs aligning to human/bacterial genomes, but might not happen until I try a could of different sets of Kmers sizes.

[seb567]

Thanks for your help Sébastien, doing a fresh install of 1.6.1-rc3 with force packing off worked wonderfully. The starting sequence totaled about 250M pe 104bp reads and Ray finished in 18.5 hours on 96 nodes.

I ran the simple default condition, except for setting Kmers to 31, since I'd tried a few other programs with that size and wanted to be able to compare the assemblies, though no other program has finished yet.

The assembly was not particularly great however. N50=194, the total contig lengths only added up to 44% of the estimated genome size and the longest contig was 5383. Do you have any particular recommendations? We should have a coverage of almost 30x. Does that mean we should try larger Kmers? Would debug-bubbles and debug-seeds help?

Well, you have to do some post-assembly quality control.

First, check the PREFIX.CoverageDistributionAnalysis.txt You need a high peak coverage. (at least 20, higher is better)

Do you have paired information ?

If so, check PREFIX.LibraryStatistics.txt and assert that the averages and standard deviations are OK.

What data do you have ? Is it Illumina TruSeq v. 3 ? (those are very neat)

-debug-bubbles and -debug-seeds and -show-ending-context just display more information to you.

Sébastien

http://github.com/sebhtml

[Wallysb01]

Thanks for your help Sébastien, doing a fresh install of 1.6.1-rc3 with force packing off worked wonderfully. The starting sequence totaled about 250M pe 104bp reads and Ray finished in 18.5 hours on 96 nodes.

I ran the simple default condition, except for setting Kmers to 31, since I'd tried a few other programs with that size and wanted to be able to compare the assemblies, though no other program has finished yet.

The assembly was not particularly great however. N50=194, the total contig lengths only added up to 44% of the estimated genome size and the longest contig was 5383. Do you have any particular recommendations? We should have a coverage of almost 30x. Does that mean we should try larger Kmers? Would debug-bubbles and debug-seeds help?

[seb567]

I finally did another try with a new install and Ray measured the latency around 125 microseconds. Is that part of the reason its loading fairly slowly?

Also, I did the FORCE_PACKING=n option on Ray-1.6.3-rc3, and I still encountered a bus error. Any ideas on what else might be causing that?

125 is just fine.

Is this your system:

Saguaro 2

http://www.top500.org/system/details/9789

If so, the architecture is

Intel EM64T Xeon E54xx (Harpertown) 2830 MHz (11.32 GFlops)

http://en.wikipedia.org/wiki/Xeon#5400-series_.22Harpertown.22

Did you supply a 'make clean' before doing 'make FORCE_PACKING=n' if you had build Ray-1.6.3-rc3 with FORCE_PACKING=y previously ?

Otherwise, your build will be a mix of both.

If not, can you send an email on the Ray mailing list and include at which point you get your bus error ?

https://lists.sourceforge.net/lists/listinfo/denovoassembler-users

colosse.clumeq.ca also uses Intel Xeon processors, but they are Nehalem-EP, not Harpertown.

Thanks.

Sébastien