[steven]

Do you mean that two overlapping reads show discrepancy often or do you mean that PCR duplicates occur often? Sorry, I just want to clarify to be sure.

I could not say if it is because of PCR duplicates or the other reasons i mentioned, but my feeling is that coverage variation is substantial (see also the link to a dedicated thread in my previous post). I have no precise measurement of this variation though. Maybe a large scale estimation of this variability could be an interesting experiment -if not already performed? Note that Helicos data could be used as a negative control regarding the potential contribution of the PCR step.

[RockChalkJayhawk]

One other approach would be to convert the files to tab-delimited; sort on the sequence column & then you can compress duplicates by just comparing the current row to the previous one -- I believe UNIX sort can deal with files that are quite large & perhaps use on-disk storing of the intermediates during the sort.

What UNIX command do you use to compare multiple lines in the same file? The only way I would know to do iit is to sort, cut the sequence from the SAM file, uniq, then join that list with the SAM. There is an easier way than this, right?

[drio]

What UNIX command do you use to compare multiple lines in the same file? The only way I would know to do iit is to sort, cut the sequence from the SAM file, uniq, then join that list with the SAM. There is an easier way than this, right?

drio@ned:~/tmp **$ cat test.txt 
a:1
z:2
t:1
r:4
x:1
drio@ned:~/tmp **$ cat test.txt  | gsort -t: -k2,2 -u
a:1
z:2
r:4

[RockChalkJayhawk]

drio@ned:~/tmp **$ cat test.txt 
a:1
z:2
t:1
r:4
x:1
drio@ned:~/tmp **$ cat test.txt  | gsort -t: -k2,2 -u
a:1
z:2
r:4

Yep...that is much simpler. Now, I am filtering my SAM file to contain correctly matched pairs while removing duplicates by:

awk '($7=="=")' accepted_hits.sam |sort -k10,10 -u > reads.filtered