Demultiplexing FASTQ with custom indices

[bpbbentley]

Hi all,

I'm fairly new to the realm of bioinformatics with large data sets, so apologies if I've missed something crucial here...

I've recently received some Illumina HiSeq2500 data in FASTQ format which haven't been demultiplexed. We've used custom i5 and i7 sequences in unique combinations for 96 samples. I was given the data in 8 FASTQ files, 2 per lane (4 lanes) with paired-ends. I've concatenated all of the forward and all of the reverse reads into 2 files for simplicity. I've been using the demuxbyname.sh method through BBMap - but I keep running into a couple of problems:

1. When I run demuxbyname.sh with a single string I only receive ~2500 reads in the output files. I've noticed that a lot of the index sequences in the FASTQ files contain N's - especially as the first base call (for i5 and i7).

2. This generally takes ~3hrs, but when I then attempt to run the script with an index.txt file containing multiple index combinations, the compute time increases exponentially.

Any help on either of these points is greatly appreciated!

[GenoMax]

Before we get into specifics can you ask your sequence provider to do this demultiplexing with Illumina's program called bcl2fastq (you can't do this since it requires access to the full data folder for the flowcell). That should be trivial for them to do (and they should have done it in first place unless you chose not to give them the sample_ID_index combinations).

Can you tell us how you are running "demuxbyname.sh" (full command line)? You should run it like this: https://www.biostars.org/p/139395/#139409 You could start multiple runs (even 96 with just one index combo) to speed things up.

There is also another package called deML that can be used for this.

[liorgalanti]

https://biosails.github.io/pheniqs/

[bpbbentley]

Thanks for your feedback on this, it's much appreciated!

I've contacted BGI and they've said that they'll help me with the demultiplexing. I thought it was strange that they simply provided FASTQ files for each lane, especially as they contacted me early on and asked me to provide the index sequences...

I've run the command a few ways, this is ideally what I'm going for:

../sw/bbmap/demuxbyname.sh in=all_lanes_1.fq in2=all_lanes_2.fq out=demux_out/%_1.fq out2=demux_out/%_2.fq prefixmode=f substringmode=f names=index_names_s1.txt

However, I have run it using single sequence strings, and also just running 1 lane of data at a time. Thanks again for your help.

[GenoMax]

Your indexes most likely look like Index1+Index2 (e.g. GGACTCCT+GCGATCTA) then that is how you need to include them in the file one per line. Is that how you are doing this?

[bpbbentley]

Yep my indexes are index1_index2 in the read header, and my .txt file reflects these. I get output files with the index complex names, but these are typically not populated with reads...