Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • fastest way to 'parse' fasta or fastq?

    I am looking for a real (High performace computing / HPC) fast fasta or fastq parsing program. I just want the most simple statistics imaginable:
    - number of reads
    - total nr of bases.
    Other stuff like average length/ATCG composition is nice, but not required.

    I searched the software page, tried some packages, wrote my own parsers but they are all slow.
    I am looking for something in C code, which can be super fast I hope.
    I also tried this simple bash code:
    " time grep -v '^>' ./test.fa | wc -m -l"

    which is 'fast' ( 30 seconds to scan 1 GB fasta (file in memory)
    My simple python script takes over a minute to scan this file. But I hope this can be done faster, or all in one script.


    If you want to scan gigabytes of files, it would be nice to have a very fast parser.

    Anyone who is aware of such program? Or, what do you think is the fastest program you know?

  • #2
    Hi,

    You could try the FASTQC package if you haven't already. It can take fastq/bam/sam files and gives most of the important statistics for a NGS run.

    Comment


    • #3
      I suggest using native linux tools such as grep, sed, awk in multithreaded environment also 64 bit may be useful in some applications where it is supported. There is option of using CUDA with GPU to do super fast calculations.

      Comment


      • #4
        PRINSEQ and FASTQC

        Comment


        • #5
          If you're up for moding a couple of lines of code for your needs
          this should do the trick ...
          Code:
          #include <stdio.h>
          #include <string.h>
          #include <ctype.h>
          unsigned long int sum[5];
          unsigned long int basecount;
          unsigned long int readcount = 0;
          char s[512];
          int main()
          {
              register int i,j;
              char ch;
              basecount = 0;
              memset(sum,0,sizeof(sum));
              while (gets(s))
              {
                  if (s[0] == '>') continue; // skip fasta entry header
                  readcount++;
                  for (i=0;i<s[i];i++)
                  {
                      ch = toupper(s[i]);
                      if (ch == 'A') { sum[0]++; basecount++; }
                      else if (ch == 'C') { sum[1]++; basecount++; }
                      else if (ch == 'G') { sum[2]++; basecount++; }
                      else if (ch == 'T') { sum[3]++; basecount++; }
                      else if (ch == 'N') { sum[4]++; basecount++; }
                  }
                  memset(s,0,sizeof(s));
              }
              for (j=0;j<5;j++)
              {
                  if (j == 0) printf("A ");
                  else if (j == 1) printf("C ");
                  else if (j == 2) printf("G ");
                  else if (j == 3) printf("T ");
                  else if (j == 4) printf("N ");
                  printf("%ld ",sum[j]);
                  printf("\n");
              }
              printf("bases = %ld \n",basecount);
              printf("reads = %ld \n",readcount);
              return 0;
          }

          Comment


          • #6
            If you don't want error checking Heng Li has a very fast FASTA/FASTQ parser in C which could easily be used for the basic information you requested (read count and total bases):

            Comment

            Latest Articles

            Collapse

            • seqadmin
              Current Approaches to Protein Sequencing
              by seqadmin


              Proteins are often described as the workhorses of the cell, and identifying their sequences is key to understanding their role in biological processes and disease. Currently, the most common technique used to determine protein sequences is mass spectrometry. While still a valuable tool, mass spectrometry faces several limitations and requires a highly experienced scientist familiar with the equipment to operate it. Additionally, other proteomic methods, like affinity assays, are constrained...
              04-04-2024, 04:25 PM
            • seqadmin
              Strategies for Sequencing Challenging Samples
              by seqadmin


              Despite advancements in sequencing platforms and related sample preparation technologies, certain sample types continue to present significant challenges that can compromise sequencing results. Pedro Echave, Senior Manager of the Global Business Segment at Revvity, explained that the success of a sequencing experiment ultimately depends on the amount and integrity of the nucleic acid template (RNA or DNA) obtained from a sample. “The better the quality of the nucleic acid isolated...
              03-22-2024, 06:39 AM

            ad_right_rmr

            Collapse

            News

            Collapse

            Topics Statistics Last Post
            Started by seqadmin, 04-11-2024, 12:08 PM
            0 responses
            12 views
            0 likes
            Last Post seqadmin  
            Started by seqadmin, 04-10-2024, 10:19 PM
            0 responses
            17 views
            0 likes
            Last Post seqadmin  
            Started by seqadmin, 04-10-2024, 09:21 AM
            0 responses
            14 views
            0 likes
            Last Post seqadmin  
            Started by seqadmin, 04-04-2024, 09:00 AM
            0 responses
            43 views
            0 likes
            Last Post seqadmin  
            Working...
            X