[arelouse]

Hi, a simple question: how do u achieve parallelism in Ray?

I took a quick look at this thread and also read the paper and the slides describing Ray, but ending with almost nothing.
Sorry if I missed some points! It would be appreciated if u can provide a little basic ideas behind Ray.
For example, it's easy (at least from the description) to understand AByss' distribution strategy.

[seb567]

I'm going to have a go at tackling the colour space problem. It might take me a while to get up to speed, because it's been a few years since I last worked on a large c++ project -- my head is currently geared towards the cotton-wool world of Java.

If, as appears, the reverse complement of a colour space sequence is its reverse, then it may be more space-efficient to store as colourspace, something like:

<first base><colour-space sequence><reverse-complement first base>

My guess at what needs to happen:
* store colour space reads as <first base>[0-3]+<rc first base>
* assemble by matching colour space [i.e. ignore first reads]
- don't convert to base space, because misreads in colour space cause the remainder of the read [when converted to base space] to be junk
- this would help greatly if reads were stored by Ray as colour space as above, but I expect that would be quite a disruptive change
* when reporting assembly, convert to base space
- only convert / report where the starting base in a segment is known, or can be inferred

There is also an equivalent csfastq format, which might be nice to implement.

My git fork (which happens to be my first attempt at a git fork, so apologies for badness) has started off with modifying the colour space decoder. I've replaced char* use in that file with strings, which may mean it's ended up more broken if constant strings are being thrown around.

Presently, Ray can assemble color-space reads to produce color-space contigs.

https://github.com/sebhtml/ray

The only color-space data I have is ecoli50x50

2010-05-19
http://solidsoftwaretools.com/gf/project/ecoli50x50/

An example of contig:

>contig-0 1037 nucleotides
233001233101131131320333012130033132102200122123303132133111
110110230111011310130100101130301003233013332033103101113122
132330033101100120331321333012101320332012311312322312213100
103202013101300233010123103313122103220031030113313031013000
211210100010110330322020130113033330021130101220112301210213
231002121313320013010032021330312020213230100300222033012233
002312023232010122302100111333023230330230331331331232333312
333030023220232210010010211202321100230332102220301232330132
330103313203212301203312313122213110121301330012333012312322
0221023133032310312002212103102303310200ly12332033002101310310
011100130101300312331000002131331230210122001030033230123330
103320133330330123302300332001131201221200233300321233021330
112211213321122101302022122110132111012023121113231201211201
321131122031112030000030100200133301230332133231220111331230
321221210222112030103031123321223120130321222311101103031330
101213203201011330110300201300211231223100202013210123012023
130003102300211303301003231330001201100013231032231222021223
03112102222232320

With these data and with a k-mer length = 21,

93.1947% of k-mers occur only once, owing to an enormous error rate presumably.

I (still) don't see a way to convert these colored strings back into biological (nucleotide) space.

To do the conversion, you need the first letter. But the first letter is not known. Furthermore, a contig does not necessarily starts right at the beginning of a read.


If you want to discuss about the code, please use the mailing list instead.

Sébastien

[seb567]

In order for RAY to recognize the pair halves belonging together, do they need to confirm to the fastq readID convention?

forward read: @READ_ID/1
reverse read: @READ_ID/2

In a single fastq file, or in two files?

One way to do this would be to
1)run newbler on the paired-read sff files with the -tr option, this results in the reads being split and output (in a single file) as

>READ_ID_left
sequence
>READ_ID_right
sequence

2) split the fasta/qual files according to _left and _right, if needed (small script, I guess)
3) change the _left to /1 and _right to /2 (e.g. using sed)
4) convert fasta+qual to fastq using your favorite tool

Correct?

In Ray, your paired sequences can be in two files (-p file1.fastq file2.fastq) or in one file (-i file.fastq).

For -p, files must contain the same number of sequences.

Example: file1 contains seq1/1, seq2/1 and file2 contains seq1/2, seq2/2.

For -i, the file must contain seq1/1, seq1/2, seq2/1, seq2/2,

The name of your sequences are irrelevant to Ray.

[flxlex]

To use 454 mate-pairs in an SFF file, you must extract them
and provide Ray with the 2 resulting fastq files.

Ray only supports 454 shotgun (single) reads.

In order for RAY to recognize the pair halves belonging together, do they need to confirm to the fastq readID convention?

forward read: @READ_ID/1
reverse read: @READ_ID/2

In a single fastq file, or in two files?

One way to do this would be to
1)run newbler on the paired-read sff files with the -tr option, this results in the reads being split and output (in a single file) as

>READ_ID_left
sequence
>READ_ID_right
sequence

2) split the fasta/qual files according to _left and _right, if needed (small script, I guess)
3) change the _left to /1 and _right to /2 (e.g. using sed)
4) convert fasta+qual to fastq using your favorite tool

Correct?

[gringer]

I'm going to have a go at tackling the colour space problem. It might take me a while to get up to speed, because it's been a few years since I last worked on a large c++ project -- my head is currently geared towards the cotton-wool world of Java.

If, as appears, the reverse complement of a colour space sequence is its reverse, then it may be more space-efficient to store as colourspace, something like:

<first base><colour-space sequence><reverse-complement first base>

My guess at what needs to happen:
* store colour space reads as <first base>[0-3]+<rc first base>
* assemble by matching colour space [i.e. ignore first reads]
- don't convert to base space, because misreads in colour space cause the remainder of the read [when converted to base space] to be junk
- this would help greatly if reads were stored by Ray as colour space as above, but I expect that would be quite a disruptive change
* when reporting assembly, convert to base space
- only convert / report where the starting base in a segment is known, or can be inferred

There is also an equivalent csfastq format, which might be nice to implement.

My git fork (which happens to be my first attempt at a git fork, so apologies for badness) has started off with modifying the colour space decoder. I've replaced char* use in that file with strings, which may mean it's ended up more broken if constant strings are being thrown around.

[seb567]

I just finished a run with Ray 1.6.0 on about 70 gigs of trimmed 454 reads data (singles+pairs) in .sff format. This time it finished without any errors and in just 4.5 hours (on 16 cores). I'm surprised that it finished this fast, but the resulting contig file was much smaller than expected (just 3.3MB). I didn't explicitly specify the insert sizes (I let Ray estimate those, not sure if it's reliable), so maybe I should do that next time.

Edit: Apparently I set the k-mer size too low (17). I did another run with k-mer size set to 31 which resulted in many more contigs (235MB).

To use 454 mate-pairs in an SFF file, you must extract them
and provide Ray with the 2 resulting fastq files.

Ray only supports 454 shotgun (single) reads.

[figure002]

I just finished a run with Ray 1.6.0 on about 70 gigs of trimmed 454 reads data (singles+pairs) in .sff format. This time it finished without any errors and in just 4.5 hours (on 16 cores). I'm surprised that it finished this fast, but the resulting contig file was much smaller than expected (just 3.3MB). I didn't explicitly specify the insert sizes (I let Ray estimate those, not sure if it's reliable), so maybe I should do that next time.

Edit: Apparently I set the k-mer size too low (17). I did another run with k-mer size set to 31 which resulted in many more contigs (235MB).

[seb567]

seb567,

This is the first time I assemble a genome, so, i thought that my set was big because it has MANY sequences, anyway...

How long does the assembly will take?, if i have the following two set:

Paired-Ends (500 +- 50)
47.803.856 pairs

Mate-pair (2200 +- 200)
42.599.342 pairs

PD: I'm using Ray 1.3.0

You should update to v1.6.0 !

http://denovoassembler.sf.net

You should try Ray on your MANY sequences !

[figure002]

Regardless, I guess it is correct to consider SFF files as containers, just like FASTA or FASTQ files.

Therefore, Ray will no longer try to match the key sequence. Instead, it will *simply* load all sequences in the SFF file and trim them using the clipping values therein.

See http://github.com/sebhtml/ray/commit/15826e290f1

Thanks Sébastien! I'll give version 1.6.0 a try.

[kail]

E. coli

ftp://ftp.ddbj.nig.ac.jp/ddbj_databa...65_1.fastq.bz2
ftp://ftp.ddbj.nig.ac.jp/ddbj_databa...65_2.fastq.bz2
ftp://ftp.ddbj.nig.ac.jp/ddbj_databa...66_1.fastq.bz2
ftp://ftp.ddbj.nig.ac.jp/ddbj_databa...66_2.fastq.bz2

If you search http://www.ncbi.nlm.nih.gov/sra, you can probably find a more up-to-date dataset. However, sra files take forever to convert...


When compiling Ray 1.6.0, be sure to turn off data structure packing because it will produce bus errors on Itanium processors I believe.

wget http://sourceforge.net/projects/deno...-1.6.0.tar.bz2
tar xjf Ray-1.6.0.tar.bz2
cd Ray-1.6.0
make PREFIX=build-ray-1.6.0 FORCE_PACKING=n
make install
ls build-ray-1.6.0/Ray


Ray does not write any file while running, except result files. For a list, see

http://denovoassembler.sourceforge.net/manual.html#SECTION00060000000000000000

Why do you say your dataset is too large ?

seb567,

This is the first time I assemble a genome, so, i thought that my set was big because it has MANY sequences, anyway...

How long does the assembly will take?, if i have the following two set:

Paired-Ends (500 +- 50)
47.803.856 pairs

Mate-pair (2200 +- 200)
42.599.342 pairs

PD: I'm using Ray 1.3.0

[lletourn]

Very cool seb. I'm anxious to try out the MAXKMERLENGTH!

[seb567]

Ray now supports arbitrary large k-mers (MAXKMERLENGTH)

= 1.6.0 =
2011-06-13

• Moved the code tree to subversion to git and from an in-house tree to a github tree -- see http://github.com/sebhtml/ray
• Fixed a compilation problem in Scaffolder.cpp. Thanks to Volker Winkelmann (University of Cologne).
• Changed CC to MPICXX and added lines to compile Ray with Intel's MPI implementation. Thanks to Volker Winkelmann (University of Cologne).
• Implemented a Kmer class for arbitrary long k-mers (MAXKMERLENGTH)
• Added pack and unpack methods to Kmer to abstract the communication of k-mers -- thanks to Élénie Godzaridis for the idea.
• Output contigs >= 100, not paths >= 100
• Detailed the warning for unmatched 454 prefix.
• Fixed a bug in the TLE entries in the AMOS file.
• The Makefile can now install Ray somewhere. (make PREFIX=prefix; make install)
• Structures are now packed by default. Set FORCE_PACKING=n to disable it.
• Created subdirectories for code.
• Ray now uses all sequences in an SFF file -- not just those matching the sequence key.
• Ray now estimates the genome length in RayOutput.CoverageDistributionAnalysis.txt.
• Fixed an integer overflow in CoverageDistribution when the number of k-mers occuring once is very large (for Assemblathon-2 datasets).
• Added exit code EXIT_NO_MORE_MEMORY=42 as suggested by Hannes Pouseele (applied-maths.com).
• Fixed the an access violation on Windows. Bug reported by Hannes Pouseele (applied-maths.com).
• Fixed compilation errors for Microsoft Visual C++ (xiosbase and stdexcept) Bug reported by Hannes Pouseele (applied-maths.com>)
• Ray compiles with Microsoft Visual Studio 10.0 without any change.

Website: http://denovoassembler.sourceforge.net/

[seb567]

Dear all,


I’m trying to test the installation of Ray (and openMPI) in my cluster. However, the set that I possess is too big (~90.403.198 paired reads).

So, can someone tell me were can I get a smaller set to test Ray? The idea will be to have a set that can run in 1 or 2 day… or less if possible

Cluster description:

Itanium II 64 processors 1.6 GHz machine with 128 GBRAM and Infiniband Voltaire 10Gbps interconnect switch.

Also, does Ray write to the disk while it is running? Where?


Thanks in advance for your help!



PD: There are 16 nodes each with four cores.

E. coli

ftp://ftp.ddbj.nig.ac.jp/ddbj_databa...65_1.fastq.bz2
ftp://ftp.ddbj.nig.ac.jp/ddbj_databa...65_2.fastq.bz2
ftp://ftp.ddbj.nig.ac.jp/ddbj_databa...66_1.fastq.bz2
ftp://ftp.ddbj.nig.ac.jp/ddbj_databa...66_2.fastq.bz2

If you search http://www.ncbi.nlm.nih.gov/sra, you can probably find a more up-to-date dataset. However, sra files take forever to convert...


When compiling Ray 1.6.0, be sure to turn off data structure packing because it will produce bus errors on Itanium processors I believe.

wget http://sourceforge.net/projects/deno...-1.6.0.tar.bz2
tar xjf Ray-1.6.0.tar.bz2
cd Ray-1.6.0
make PREFIX=build-ray-1.6.0 FORCE_PACKING=n
make install
ls build-ray-1.6.0/Ray


Ray does not write any file while running, except result files. For a list, see

http://denovoassembler.sourceforge.net/manual.html#SECTION00060000000000000000

Why do you say your dataset is too large ?

[kail]

Test Ray

Dear all,


I’m trying to test the installation of Ray (and openMPI) in my cluster. However, the set that I possess is too big (~90.403.198 paired reads).

So, can someone tell me were can I get a smaller set to test Ray? The idea will be to have a set that can run in 1 or 2 day… or less if possible

Cluster description:

Itanium II 64 processors 1.6 GHz machine with 128 GBRAM and Infiniband Voltaire 10Gbps interconnect switch.

Also, does Ray write to the disk while it is running? Where?


Thanks in advance for your help!



PD: There are 16 nodes each with four cores.

[SES]

You mean taking the initiative to write code changes to BioPython so that Bio/SeqIO/SffIO can change the key sequence, right ?

No. I was just sympathizing with kmcarr and referring specifically to the need for sff support in bioperl.