Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • fastest way to 'parse' fasta or fastq?

    I am looking for a real (High performace computing / HPC) fast fasta or fastq parsing program. I just want the most simple statistics imaginable:
    - number of reads
    - total nr of bases.
    Other stuff like average length/ATCG composition is nice, but not required.

    I searched the software page, tried some packages, wrote my own parsers but they are all slow.
    I am looking for something in C code, which can be super fast I hope.
    I also tried this simple bash code:
    " time grep -v '^>' ./test.fa | wc -m -l"

    which is 'fast' ( 30 seconds to scan 1 GB fasta (file in memory)
    My simple python script takes over a minute to scan this file. But I hope this can be done faster, or all in one script.


    If you want to scan gigabytes of files, it would be nice to have a very fast parser.

    Anyone who is aware of such program? Or, what do you think is the fastest program you know?

  • #2
    Hi,

    You could try the FASTQC package if you haven't already. It can take fastq/bam/sam files and gives most of the important statistics for a NGS run.

    Comment


    • #3
      I suggest using native linux tools such as grep, sed, awk in multithreaded environment also 64 bit may be useful in some applications where it is supported. There is option of using CUDA with GPU to do super fast calculations.

      Comment


      • #4
        PRINSEQ and FASTQC

        Comment


        • #5
          If you're up for moding a couple of lines of code for your needs
          this should do the trick ...
          Code:
          #include <stdio.h>
          #include <string.h>
          #include <ctype.h>
          unsigned long int sum[5];
          unsigned long int basecount;
          unsigned long int readcount = 0;
          char s[512];
          int main()
          {
              register int i,j;
              char ch;
              basecount = 0;
              memset(sum,0,sizeof(sum));
              while (gets(s))
              {
                  if (s[0] == '>') continue; // skip fasta entry header
                  readcount++;
                  for (i=0;i<s[i];i++)
                  {
                      ch = toupper(s[i]);
                      if (ch == 'A') { sum[0]++; basecount++; }
                      else if (ch == 'C') { sum[1]++; basecount++; }
                      else if (ch == 'G') { sum[2]++; basecount++; }
                      else if (ch == 'T') { sum[3]++; basecount++; }
                      else if (ch == 'N') { sum[4]++; basecount++; }
                  }
                  memset(s,0,sizeof(s));
              }
              for (j=0;j<5;j++)
              {
                  if (j == 0) printf("A ");
                  else if (j == 1) printf("C ");
                  else if (j == 2) printf("G ");
                  else if (j == 3) printf("T ");
                  else if (j == 4) printf("N ");
                  printf("%ld ",sum[j]);
                  printf("\n");
              }
              printf("bases = %ld \n",basecount);
              printf("reads = %ld \n",readcount);
              return 0;
          }

          Comment


          • #6
            If you don't want error checking Heng Li has a very fast FASTA/FASTQ parser in C which could easily be used for the basic information you requested (read count and total bases):

            Comment

            Latest Articles

            Collapse

            • seqadmin
              Recent Advances in Sequencing Analysis Tools
              by seqadmin


              The sequencing world is rapidly changing due to declining costs, enhanced accuracies, and the advent of newer, cutting-edge instruments. Equally important to these developments are improvements in sequencing analysis, a process that converts vast amounts of raw data into a comprehensible and meaningful form. This complex task requires expertise and the right analysis tools. In this article, we highlight the progress and innovation in sequencing analysis by reviewing several of the...
              05-06-2024, 07:48 AM
            • seqadmin
              Essential Discoveries and Tools in Epitranscriptomics
              by seqadmin




              The field of epigenetics has traditionally concentrated more on DNA and how changes like methylation and phosphorylation of histones impact gene expression and regulation. However, our increased understanding of RNA modifications and their importance in cellular processes has led to a rise in epitranscriptomics research. “Epitranscriptomics brings together the concepts of epigenetics and gene expression,” explained Adrien Leger, PhD, Principal Research Scientist...
              04-22-2024, 07:01 AM

            ad_right_rmr

            Collapse

            News

            Collapse

            Topics Statistics Last Post
            Started by seqadmin, 05-14-2024, 07:03 AM
            0 responses
            27 views
            0 likes
            Last Post seqadmin  
            Started by seqadmin, 05-10-2024, 06:35 AM
            0 responses
            47 views
            0 likes
            Last Post seqadmin  
            Started by seqadmin, 05-09-2024, 02:46 PM
            0 responses
            59 views
            0 likes
            Last Post seqadmin  
            Started by seqadmin, 05-07-2024, 06:57 AM
            0 responses
            47 views
            0 likes
            Last Post seqadmin  
            Working...
            X