fastq.gz VS fastq

[lala2013]

Hi, I have a question about fastq and fastq.gz files. I understand that fastq.gz is the compressed version of fastq file. Can I combine all the R1.fastq.gz files and R2.fastq.gz separately before feeding it to fastQC ? Thanks.

[ECO]

FastQC can use .gz.

You probably want to run R1 and R2 separately for understanding the quality of your sequencing run/data, as they have distinctly different error profiles.

[lala2013]

Sorry I didn't make it clear. Is it possible to do the following and feed R1.fastq.gz to fastQC?
cat L001_R1.fastq.gz L002_R1.fastq.gz L003_R1.fastq.gz L004_R1.fastq.gz > R1.fastq.gz

[vivek_]

yes you can but if you can distribute jobs on a cluster, you could do each file separately much faster.

[winsettz]

Looks like fastqc will use one thread per file. Reading the -help for fastqc it states --threads "Specifies the number of files which can be processed simultaneously". Perhaps upping --threads to equal the number of fastq you have, then giving it all the fastq as inputs is the way to go? Going divide-and-conquer is what you need to do.

The problem with cat'ing all your fastq is that the statistics would be calculated as average of all reads: you might not catch an anomaly in one subset of reads, especially if it gets averaged out by the other three set of reads.

[fkrueger]

Sorry I didn't make it clear. Is it possible to do the following and feed R1.fastq.gz to fastQC?
cat L001_R1.fastq.gz L002_R1.fastq.gz L003_R1.fastq.gz L004_R1.fastq.gz > R1.fastq.gz

I agree that you should probably not merge the files prior to feeding them to FastQC because you run the risk of masking any potential lane specific effects. Apart from that, using cat doesn't work for gzipped files, but you can use zcat to do the same thing. The command should something like this:

zcat L001_R1.fastq.gz L002_R1.fastq.gz L003_R1.fastq.gz L004_R1.fastq.gz | gzip -c - > R1.fastq.gz

[vivek_]

I think cat does concatenate gzipped files. You don't need to gzip again either.

[fkrueger]

Oh didn't know that, cheers for that!

[kmcarr]

I think cat does concatenate gzipped files. You don't need to gzip again either.

I recall that FastQC didn't properly process the resultant file when simply 'cat'ed the gzipped files. The file which you get by concatenating gzipped file still contains some sort of division between the original compressed parts and FastQC would stop when it reached the end of the first block of reads. This was in and earlier version of FastQC and Simon may have addressed this in a later release.

O.K. I just went to the FastQC site and under the Change Log:

3-5-12: Version 0.10.1 released
Added a workround to allow the analysis of concatenated gzipped files(emphasis mine)
Fixed a bug when FastQC was installed in a path containing characters needing to be escaped in a URL
Added an option to specify the location of the java interpreter on the command line

This was added in the last version (0.10.1) so make sure you're using that version if you are concatenating gzipped files.