problem with newbler and cap3

[bioben]

I used newbler to assemble >10 million 454 ESTs, and then used CD-HIT(http://www.bioinformatics.org/cd-hit/) to analyze the resulting 454Isotigs.fna file, basically to find out if the resulted isotigs have identical sequences. If you don't know, CD-HIT is a sequence clustering tool, putting similar sequences into clusters depending on what identity threshold you use. I used identity threshold at 100%, so that sequences that are 100% identical but with different lengths, e.g. one sequence is a sub-region of another, are put into one cluster. I found there are thousands of isotigs 100% identical to others (i.e. are contained in other sequences). I think this is not supposed to happen: the identical sequences should be output as one isotig.

I also observed the same problem when I am analyzing cap3 assembly. The resulting contigs and singlets files both contain identical sequences.

Is this normal or wrong? If we don't do the CD-HIT clustering, our final assembly will be overestimated ...