Converting fastq formats?

[beeman]

First post, so please be gentle

I've looked for the answer in faq's and related posts, but I'm still stumped.

I have a number of PE illumina datasets in fastq format, however I've recently downloaded a new dataset which appears to have a substantially different format for it's fastq header/identifiers. I'd like to use this new dataset in some established pipelines that don't seem to be happy with this different format.

Firstly, can someone please explain what may be the underlying cause of this? Could this arise due to different technologies or merely different revisions of the standard pipelines over time?

Secondly, is it possible to use any tools like Biopython to reformat the header/identifiers to a more acceptable format?


Example of old *working* fastq data compatible with our pipeline

--------
@SRR094905.1 HWI-BRUNOP20X:0617:2:1:12498:1196 length=75
TATAAGATAATGTTTAATAAAAAANNNNNNNNNNNNNNNAGNNNNNNNNNNNTNNNNTTTTGTNNNNNNNNGAGG
+SRR094905.1 HWI-BRUNOP20X:0617:2:1:12498:1196 length=75
FFFFDFFFFEFEFFFFFFFF####!!!!!!!!!!!!!!!##!!!!!!!!!!!#!!!!######!!!!!!!!####
@SRR094905.2 HWI-BRUNOP20X:0617:2:1:5633:1197 length=75
AATATTTTTTATGATGGTAGAATANNNNNNNNNNNNNNNTGNNNNNNNNNNNTNNNNTTTATTNNNNNNNNGAGT
+SRR094905.2 HWI-BRUNOP20X:0617:2:1:5633:1197 length=75
HHHHHHHHEEEEEHHHHGHH####!!!!!!!!!!!!!!!##!!!!!!!!!!!#!!!!######!!!!!!!!####
@SRR094905.3 HWI-BRUNOP20X:0617:2:1:6241:1197 length=75
GAGGGGGTTTTTATTGGTTTTAGGNNNNNNNNNNNNNNNTTNNNNNNNNNNNTNNNNATTTGTNNNNNNNNTTGT
+SRR094905.3 HWI-BRUNOP20X:0617:2:1:6241:1197 length=75
GFGGF=86=DFF;?@#########!!!!!!!!!!!!!!!##!!!!!!!!!!!#!!!!######!!!!!!!!####
@SRR094905.4 HWI-BRUNOP20X:0617:2:1:6877:1197 length=75
GGAGGTATTTTTTACGAGGTAGGGNNNNNNNNNNNNNNNTTNNNNNNNNNNNGNNNNTTAGGANTNNNNNNGTAG
+SRR094905.4 HWI-BRUNOP20X:0617:2:1:6877:1197 length=75
HHHHHGHHHHHGGGHHHHHG####!!!!!!!!!!!!!!!##!!!!!!!!!!!#!!!!######!#!!!!!!####
@SRR094905.5 HWI-BRUNOP20X:0617:2:1:8302:1197 length=75
TTTGTATTAGTTATTTTGGCGTGANNNNNNNNNNNNNNNTANNNNNNNNNNNGNNNNTTGGTTNNNNNNNNTTTT
+SRR094905.5 HWI-BRUNOP20X:0617:2:1:8302:1197 length=75
HHHHHHHHGHHHGHHHCCHHA@A#!!!!!!!!!!!!!!!##!!!!!!!!!!!#!!!!######!!!!!!!!####


Example of *new* dataset with different headers/identifiers
--------
@A80PGJABXX:8:1:1648:2147#0/1
AATAATAATTTATTTATATATTTATTTATTTATAAATTTATTTTTTAAA
+
ggggggfgggggggggggggggfggggggggdggadfbfegggggf_^a
@A80PGJABXX:8:1:1639:2201#0/1
ATGATGAATATTTTTTTGATTAAATTATGATTATATTTATGTATTTTGA
+
gggfgggggfgggbgggggggfgdgggggefgggggggegfeggefffa
@A80PGJABXX:8:1:1546:2220#0/1
TGTTATGTATTTATTAAGATAATATGATGAAAATTATAAAATATAATTA
+
fffff^^ffd^efdcdeeaecfefdecddfffNffeeadedfdffaffe
@A80PGJABXX:8:1:1579:2244#0/1
TAAGTTAGGGTGATTGTATATAATAATTTGTTTAAGGTGATAGTTATTA
+
effefedeee[dcddffcffgdfggggggggggeaddJd^dadcbbcfd
@A80PGJABXX:8:1:1753:2142#0/1
GAATTTTATAATATGGAATTTATTATATTTGAATAATGTAAGAGTTGAT
+
gfgggggeggggfggfgfgggdggdgegggfddggggfeaggdgfggeg


Thanks in advance for your patience!

[GenoMax]

Perhaps you missed the Wikipedia entry for FASTQ format.

http://en.wikipedia.org/wiki/FASTQ_format

Am curious as to why your pipelines are not happy with the change in sequence ID. Are you parsing the ID's?

[beeman]

Hi thanks for your reply.

Yes I have read the fastq wiki, but the two fastq files appear to be annotated in a different way, hence my confusion.

In the first fastq file the read ID is first (and increments for each read), followed by the instrument name "SRR094905.1 HWI-BRUNOP20X". For the second file, the first entry is the instrument name and is the same for every read, in line with the illumina format on the fastq wiki, rather than a unique read ID for each and every read.

Does this make sense?

I am unsure as to why there is such a significant difference in read annotation between these two datasets. Of course I can parse this information for each read and output to the required format, but I want to establish what is the "correct" and "proper" format that I should use, as despite the standard outlined in the wiki page there are obviously significant variations.

Cheers,

[GenoMax]

Hi thanks for your reply.

Yes I have read the fastq wiki, but the two fastq files appear to be annotated in a different way, hence my confusion.

In the first fastq file the read ID is first (and increments for each read), followed by the instrument name "SRR094905.1 HWI-BRUNOP20X". For the second file, the first entry is the instrument name and is the same for every read, in line with the illumina format on the fastq wiki, rather than a unique read ID for each and every read.

Does this make sense?

The first data set appears to have come from SRA. As noted in the Wikipedia article SRA identifier is added to each original ID line.

The second dataset you have seems to have originated from a sequencer.

I am unsure as to why there is such a significant difference in read annotation between these two datasets. Of course I can parse this information for each read and output to the required format, but I want to establish what is the "correct" and "proper" format that I should use, as despite the standard outlined in the wiki page there are obviously significant variations.

Cheers,

There is no standard as to what there needs to be there on the ID line. Minimal FASTQ format only needs the following items

@SEQ_ID
GATTTGGGGTTCAAAGCAGTATCGATCAAATAGTAAATCCATTTGTTCAACTCACAGTTT
+
!''*((((***+))%%%++)(%%%%).1***-+*''))**55CCF>>>>>>CCCCCCC65

[beeman]

Ok thanks,

[HESmith]

The "new" format is actually from an older version of the Illumina platform. Your pipeline may be unhappy b/c the quality scores are PHRED+64 encoded instead of the Sanger standard of PHRED+33.

[JackieBadger]

If so, you can converts between PHRED encryptions using the Fastq groomer tool in Galaxy portal

[beeman]

Thanks for your helpful replies