Header Leaderboard Ad


Demultiplexing FASTQ with custom indices



No announcement yet.
  • Filter
  • Time
  • Show
Clear All
new posts

  • Demultiplexing FASTQ with custom indices

    Hi all,

    I'm fairly new to the realm of bioinformatics with large data sets, so apologies if I've missed something crucial here...

    I've recently received some Illumina HiSeq2500 data in FASTQ format which haven't been demultiplexed. We've used custom i5 and i7 sequences in unique combinations for 96 samples. I was given the data in 8 FASTQ files, 2 per lane (4 lanes) with paired-ends. I've concatenated all of the forward and all of the reverse reads into 2 files for simplicity. I've been using the demuxbyname.sh method through BBMap - but I keep running into a couple of problems:
    1. When I run demuxbyname.sh with a single string I only receive ~2500 reads in the output files. I've noticed that a lot of the index sequences in the FASTQ files contain N's - especially as the first base call (for i5 and i7).
    2. This generally takes ~3hrs, but when I then attempt to run the script with an index.txt file containing multiple index combinations, the compute time increases exponentially.

    Any help on either of these points is greatly appreciated!

  • #2
    Before we get into specifics can you ask your sequence provider to do this demultiplexing with Illumina's program called bcl2fastq (you can't do this since it requires access to the full data folder for the flowcell). That should be trivial for them to do (and they should have done it in first place unless you chose not to give them the sample_ID_index combinations).

    Can you tell us how you are running "demuxbyname.sh" (full command line)? You should run it like this: https://www.biostars.org/p/139395/#139409 You could start multiple runs (even 96 with just one index combo) to speed things up.

    There is also another package called deML that can be used for this.


    • #3


      • #4
        Thanks for your feedback on this, it's much appreciated!

        I've contacted BGI and they've said that they'll help me with the demultiplexing. I thought it was strange that they simply provided FASTQ files for each lane, especially as they contacted me early on and asked me to provide the index sequences...

        I've run the command a few ways, this is ideally what I'm going for:

        ../sw/bbmap/demuxbyname.sh in=all_lanes_1.fq in2=all_lanes_2.fq out=demux_out/%_1.fq out2=demux_out/%_2.fq prefixmode=f substringmode=f names=index_names_s1.txt

        However, I have run it using single sequence strings, and also just running 1 lane of data at a time. Thanks again for your help.


        • #5
          Your indexes most likely look like Index1+Index2 (e.g. GGACTCCT+GCGATCTA) then that is how you need to include them in the file one per line. Is that how you are doing this?


          • #6
            Yep my indexes are index1_index2 in the read header, and my .txt file reflects these. I get output files with the index complex names, but these are typically not populated with reads...


            Latest Articles


            • seqadmin
              A Brief Overview and Common Challenges in Single-cell Sequencing Analysis
              by seqadmin

              ​​​​​​The introduction of single-cell sequencing has advanced the ability to study cell-to-cell heterogeneity. Its use has improved our understanding of somatic mutations1, cell lineages2, cellular diversity and regulation3, and development in multicellular organisms4. Single-cell sequencing encompasses hundreds of techniques with different approaches to studying the genomes, transcriptomes, epigenomes, and other omics of individual cells. The analysis of single-cell sequencing data i...

              01-24-2023, 01:19 PM
            • seqadmin
              Introduction to Single-Cell Sequencing
              by seqadmin
              Single-cell sequencing is a technique used to investigate the genome, transcriptome, epigenome, and other omics of individual cells using high-throughput sequencing. This technology has provided many scientific breakthroughs and continues to be applied across many fields, including microbiology, oncology, immunology, neurobiology, precision medicine, and stem cell research.

              The advancement of single-cell sequencing began in 2009 when Tang et al. investigated the single-cell transcriptomes
              01-09-2023, 03:10 PM