Using cd-hit to merge paired-end data then cluster

[SymbioNGS]

Hi Seqanswers community,

I have a paired-end dataset of Illumina MiSeq data where each F and R reads overlap along most of the merged fragment length (300 bp of ~320 bp fragments).

It is my understanding that cd-hit-otu can handle this type of data, but when I try to run the following command using the perl script for paired end data:

cd-hit-otu-all-pair.pl -i Alltrimfoward.fastq,Alltrimreverse.fastq -o
OTUcdhitonly -t 1.0,1.0 -p 20,21 -c 0.99 -m true -e 0.00000044

I get the following error messages:

running command
./cdhit-dup/cd-hit-dup -i OTUcdhitonly/fltd-c -o OTUcdhitonly/fltd-c.dup -d 0 -m false -f false > OTUcdhitonly/dup1.log
cd-hit-dup: cdhit-dup.cxx:141: int HashingDepth(int, int): Assertion `len >= min' failed.
sh: line 1: 9553 Aborted ./cdhit-dup/cd-hit-dup -i OTUcdhitonly/fltd-c -o OTUcdhitonly/fltd-c.dup -d 0 -m false -f false > OTUcdhitonly/dup1.log

running command
./cd-hit/clstr_sort_by.pl < OTUcdhitonly/fltd-c.dup.clstr > OTUcdhitonly/fltd-c.dup.clstr.sorted
sh: OTUcdhitonly/fltd-c.dup.clstr: No such file or directory

running command
./multi-contig-filter.pl OTUcdhitonly/fltd-c.dup.clstr.sorted > OTUcdhitonly/wrong-contigs.ids
can not open OTUcdhitonly/fltd-c.dup.clstr.sorted at ./multi-contig-filter.pl line 8.

running command
./fetch_fasta_exclude_ids.pl OTUcdhitonly/wrong-contigs.ids OTUcdhitonly/fltd-c > OTUcdhitonly/fltd-c1

running command
./cdhit-dup/cd-hit-dup -i OTUcdhitonly/fltd-c1 -o OTUcdhitonly/fltd-c1.dup -d 0 -m false -f true > OTUcdhitonly/dup2.log
cd-hit-dup: cdhit-dup.cxx:141: int HashingDepth(int, int): Assertion `len >= min' failed.
sh: line 1: 9561 Aborted ./cdhit-dup/cd-hit-dup -i OTUcdhitonly/fltd-c1 -o OTUcdhitonly/fltd-c1.dup -d 0 -m false -f true > OTUcdhitonly/dup2.log
can not open OTUcdhitonly/fltd-c1.dup.clstr at ./cd-hit-otu-all-pair.pl line 192.

There seems to be a number of possible things going wrong, here is a list of ideas:

1) the following file is not getting made, causing the next steps to throw errors:
fltd-c.dup.clstr

This could be due to path specification problems, but unlikely.

2) Maybe something is going wrong with the merging of the individual F and R reads into one paired-end read causing the dup.clstr file to not get made.

I tried to see if I could merge the reads manually using the auxiliary tools (http://weizhong-lab.ucsd.edu/cd-hit/...uxtools-manual), specifically cd-hit-lap. However, it was not to be found in the cdhit-dup folder, although I was able to find another auxiliary tool (read-linker).

3) Can cd-hit-otu handle merging pairs that have large overlaps? This information from the auxiliary tools suggests yes: "cd-hit-lap is program for extracting pairs of overlapping reads by clustering based on tail-head overlaps (with perfect matching)." But why isn't this script built into the package?

4) The first error message output might also indicate the problem and might be causing the issue: cd-hit-dup: cdhit-dup.cxx:141: int HashingDepth(int, int): Assertion `len >= min' failed.
I don't see anything about read length specifications in the input instructions, but I could be missing that in the perl script.

I have searched around cd-hit documentation and googled the error messages, but really not too much information comes up.

I have a workaround were I have used FLASh to merge my F and R reads into one single overlapping read, then passed that fastq file to Cd-hit using the perl script for single-end reads in cd-hit (cd-hit-otu-all-pair.pl). This works, but I'm not sure if it is legit so I want to compare this results with an output that cd-hit produces from merging to clustering.

Any help would be appreciated!

[SymbioNGS]

Solution

Hi All,
The reason the paired end perl script in cd-hit-otu was failing was because the name of each paired read in my Illumina generated raw data files (R1 and R2 for each sample) did not have the same identifier. The sequencer will name each uniquely. For example:
@M02288:41:000000000-A8MHN:1:1101:22104:1895 1:N:0:26 (in the forward reads file per sample)
and
@M02288:41:000000000-A8MHN:1:1101:22104:1895 2:N:0:26 (in the reverse reads file per sample)

You will need to rename all the reads in your reverse reads folder to be the same as those in your forward reads folder, ie, change the 2:N to 1: N. After I changed this and formatted the names of the reads to reflect the sample number, the paired end perl script ran perfectly.