fastest way to 'parse' fasta or fastq?

[HenrivdGeest]

I am looking for a real (High performace computing / HPC) fast fasta or fastq parsing program. I just want the most simple statistics imaginable:
- number of reads
- total nr of bases.
Other stuff like average length/ATCG composition is nice, but not required.

I searched the software page, tried some packages, wrote my own parsers but they are all slow.
I am looking for something in C code, which can be super fast I hope.
I also tried this simple bash code:
" time grep -v '^>' ./test.fa | wc -m -l"

which is 'fast' ( 30 seconds to scan 1 GB fasta (file in memory)
My simple python script takes over a minute to scan this file. But I hope this can be done faster, or all in one script.


If you want to scan gigabytes of files, it would be nice to have a very fast parser.

Anyone who is aware of such program? Or, what do you think is the fastest program you know?

[aggp11]

Hi,

You could try the FASTQC package if you haven't already. It can take fastq/bam/sam files and gives most of the important statistics for a NGS run.

[husamia]

I suggest using native linux tools such as grep, sed, awk in multithreaded environment also 64 bit may be useful in some applications where it is supported. There is option of using CUDA with GPU to do super fast calculations.

[JackieBadger]

PRINSEQ and FASTQC

[Richard Finney]

If you're up for moding a couple of lines of code for your needs
this should do the trick ...

#include <stdio.h>
#include <string.h>
#include <ctype.h>
unsigned long int sum[5];
unsigned long int basecount;
unsigned long int readcount = 0;
char s[512];
int main()
{
    register int i,j;
    char ch;
    basecount = 0;
    memset(sum,0,sizeof(sum));
    while (gets(s))
    {
        if (s[0] == '>') continue; // skip fasta entry header
        readcount++;
        for (i=0;i<s[i];i++)
        {
            ch = toupper(s[i]);
            if (ch == 'A') { sum[0]++; basecount++; }
            else if (ch == 'C') { sum[1]++; basecount++; }
            else if (ch == 'G') { sum[2]++; basecount++; }
            else if (ch == 'T') { sum[3]++; basecount++; }
            else if (ch == 'N') { sum[4]++; basecount++; }
        }
        memset(s,0,sizeof(s));
    }
    for (j=0;j<5;j++)
    {
        if (j == 0) printf("A ");
        else if (j == 1) printf("C ");
        else if (j == 2) printf("G ");
        else if (j == 3) printf("T ");
        else if (j == 4) printf("N ");
        printf("%ld ",sum[j]);
        printf("\n");
    }
    printf("bases = %ld \n",basecount);
    printf("reads = %ld \n",readcount);
    return 0;
}

[maubp]

If you don't want error checking Heng Li has a very fast FASTA/FASTQ parser in C which could easily be used for the basic information you requested (read count and total bases):

http://lh3lh3.users.sourceforge.net/parsefastq.shtml