Count unique reads in a FASTQ file

[id0]

Is there a tool that will count the frequency of occurrences of each read from a FASTQ file?

FastQC calls this report "Overrepresented sequences", but it only shows the most repeated sequences. It would be nice to get them all. Also, it would be nice if it was command-line so it's easier to process in batch.

Better yet, is it possible to do this for paired-end reads? Instead of just counting sequences, counting unique sequence pairs?

[vivek_]

Just a guess without looking at your data:

grep "^@" *.fastq | awk '{gsub(/\/.*$/,""); print}' | sort | uniq -c | awk '{if($1 == 2) print $2}' > headers

Will print the unique headers to a file.

for seq in `cat headers`;do grep -A 3 $seq *.fastq;done

Will print the corresponding sequences.

I think the unix command might error out for large files.

[micans]

efficient command line tool for counting duplicated reads

tally will do what you ask. It compresses sequences in memory (approximately 3.5-fold, depending on read length), so it can handle fairly large files. It comes with reaper, and can be downloaded here: http://www.ebi.ac.uk/~stijn/reaper/s...per-12-205.tgz.