CASAVA 1.8 index barcodes in read header

[ians]

We are migrating to the new CASAVA to do basecalling and generate fastqs for an Illumina sequencing run (both GA-II and Hiseq). When demultiplexing through CASAVA, the read headers include the indexed barcode. Is there an option to get this to happen for non-demultiplexed lanes? We are interested in bar code binning outside of CASAVA and seeing the barcode.

[westerman]

I am a bit confused on what you mean by:

We are interested in bar code binning outside of CASAVA and seeing the barcode.

I presume that what you want to do is to take a barcoded run but not demultiplex. In this way you can see what barcodes are present and in what amounts.

As far I as I know CASAVA will not output the barcodes until they are specified. Personally I just keep a spreadsheet around with the first 24 barcodes in it for all lanes. Then I run this through CASAVA when I want to find out what barcodes are present for a given run. I get not only the barcodes in the fastq files but also that handy demultiplexing report.

[ians]

I presume that what you want to do is to take a barcoded run but not demultiplex.

Yeah exactly. In the past, we've seen barcodes that have N base calls. We use a script to see counts of each species of barcode in the run in order to evaluate library prep and the actual sequencing.

Now that i look deeper, however, i see that the fastq file under Undetermined_indices directory has all the reads the did not pass the demultiplex filter.


What demultiplexing report are you referring to?

[sklages]

What demultiplexing report are you referring to?

Basecall_Stats_FC-ID/Demultiplex_Stats.htm

[westerman]

Now that i look deeper, however, i see that the fastq file under Undetermined_indices directory has all the reads the did not pass the demultiplex filter.

Indeed. But what happens if there are more barcodes you think should be in the run? In other words if over 5% of my reads go into the Undetermined_indices category then I want to know why. Especially if I can blame it on a lab tech screw-up ... gives me something to tease them about during lunch. :-) Thus if I get an large number of Undetermined reads I'll run my 24-barcodes on the run to see what falls out.

What demultiplexing report are you referring to?

In the 'Unaligned' directory I have the directory structure:

Basecall_Stats_C0K74ACXX/
DemultiplexConfig.xml
DemultiplexedBustardConfig.xml
DemultiplexedBustardSummary.xml
Makefile
(various Project_ directories)
SampleSheet.mk
support.txt
Temp/
Undetermined_indices/

And under the 'Basecall_Stats' directory is the 'Demultiplex_Stats.htm' file.

[sklages]

I usually look for the barcodes mentioned in the samplesheet; if there are a lot of 'undetermined' barcodes I inspect the corresponding fastq files with a perl onliner to get a rough idea what's going wrong (what barcode is missing/wrong).

btw, demultiplexing all lanes with all barcodes is only possible when you do not allow for a mismatch in barcode recognition ..

my 2p,
Sven

[cosmarium]

Incorrect fastq files?

Hi all,

I am new to NGS and I have prepared some samples already. I submitted my samples to a core facility that used Hiseq 2000. They gave me the sequences back in a folder that contains all my data. In that folder there are 8 folders, one for each lane. In each folder I see many compressed files named as follows:
PopulationE_NoIndex_L001_R1_001.fastq.gz
When I decompress this file by double click in mac I get a decompressed file named: PopulationE_NoIndex_L001_R1_001.fastq
This file contains all the sequences corresponding for lane 1, for example. I used 12 indexes for lane 1 (and all other lanes as well).
When I open the file it contains the sequences as follows:
@DF9F08P1:1270YT0ACXX:8:1101:1140:2098 1:Y:0:
GTTTTNNNNNNNNNNNNNNGATAAATATTTCAAAAACTAATCTGCCGAAACTACATGCGGTACTGGTGAAAAAAAG
+
############################################################################
@DF9F08P1:1270YT0ACXX:8:1101:1247:2110 1:N:0:
ATTTTGTCCGTAGCCATTTGTCTGGCCGTATCCGGTGCGCCTGCATGGGCGTCTGAACATCAGTCCACGCTGAGCG
+

They told me that I can demultiplex by looking at the indexes and append (copy/paste?) the sequences corresponding to the same index together in a txt file, which I can then run in my pipeline software to do the alignment.

They gave the CASAVA_1_8_2_UG_15011196C.pdf where I can see on pg 40 that the sequence line 1 with the name, should contain the index used, see:

"An example of a valid entry is as follows; note the space preceding the read number element":
@EAS139:136:FC706VJ:2:5:1000:12850 1:Y:18:ATCACG
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
+
BBBBCCCC?<A?BC?7@@???????DBBA@@@@A@@

My question is, did they give me wrong files?

I looked it up and found the index sequences here (http://intron.ccam.uchc.edu/groups/t...Sequences.html).

But I don't think I am suppose to search for these six-base-tags in my sequences, the probability of finding them seems high and doing it this way sounds pretty messy to me. Since, I have about 165 Gb of information!

If anyone can help me please, I will really appreciate it.

C

[sklages]

Hi all,
PopulationE_NoIndex_L001_R1_001.fastq.gz
When I decompress this file by double click in mac I get a decompressed file named: PopulationE_NoIndex_L001_R1_001.fastq
This file contains all the sequences corresponding for lane 1, for example. I used 12 indexes for lane 1 (and all other lanes as well).
When I open the file it contains the sequences as follows:
@DF9F08P1:1270YT0ACXX:8:1101:1140:2098 1:Y:0:
GTTTTNNNNNNNNNNNNNNGATAAATATTTCAAAAACTAATCTGCCGAAACTACATGCGGTACTGGTGAAAAAAAG
+
############################################################################
@DF9F08P1:1270YT0ACXX:8:1101:1247:2110 1:N:0:
ATTTTGTCCGTAGCCATTTGTCTGGCCGTATCCGGTGCGCCTGCATGGGCGTCTGAACATCAGTCCACGCTGAGCG
+

They told me that I can demultiplex by looking at the indexes and append (copy/paste?) the sequences corresponding to the same index together in a txt file, which I can then run in my pipeline software to do the alignment.

Really hard to believe that a core facility is doing such nasty stuff .. :-(
Provide them with the indices you have used in your samples and ask
them to please demultiplex all the lanes for you. This is nothing more
work for the core facility (except for writing a samplesheet).

The conversion step without demultiplexing just "eliminates" the index
read; you only get one or two (PE) reads. For you it is not possible to
demultiplex with just your sequencing read (without the index read).

hth,
Sven

[Alex Renwick]

It's the right files, but it looks like they misconfigured the processing of the basecall files. The index should appear after the colon on the read name line, and it doesn't. I'll agree with Sven and say it's hard to believe that they would try to hand off the data in that condition.

[sklages]

The index should appear after the colon on the read name line, and it doesn't.

Not if it is a NoIndex read ... there is simply no index. The provider has to reprocess the data. We even don't know about the use-bases-mask ..

cheers, Sven

[cosmarium]

I asked the core facility why the files are not demultiplexed and why the index does not appear in the first line of the sequence name as mentioned in pg 40 of CASAVA 1.8 user guide. The replied:

"What the CASAVA manual is saying is that if you used Illumina's barcode method that data would be there. You said that you did not use Illumina's barcode adapters to prepare these libraries. You said that the barcode was in the sequencing region of the library. Since there was no barcode in the adapter which could be separately sequenced, no barcode will show up in the fastq file".

I don't recall saying that I did not use Illumina's index. What would I said that, if I used TrueSeq Kit?

I got the sequence of each index now, I will try to ask the core facility again if they can demultiplex. But also I don't want to spent my time discussing who said what and when, etc.

I think I have two options: (1) I can run a script to search for the index sequence inside my files, I have seen one around the forums here, but I am not sure how I am going to implement it yet. Or, (2) I can download CASAVA and do demultiplexing myself. I have configured an account in a computer cluster in Maxwell, here in my school (with help) to run my sequences in the pipeline that I use (breseq). Now, I asked them in Maxwell cluster, if I can install CASAVA there, so I can do demultiplexing. I am waiting for reply.

I like to learn new things so I don't mind trying them. The only issue is time and of course that I get to understand and work out things. My knowledge is limited in bioinformatics, NGS and command line programs. But, I try.

Any further thoughts you may have for me will be greatly welcome. Thanks for your support. This forum is great!

C

[Heisman]

cosmarium, I am worried that you used indexes and the index was never sequenced. If so, I don't see how you can demultiplex. Does this make sense? If that's the case, you definitely need to discuss this with your core facility.

[cosmarium]

How they could have not sequence the indexes? Aren't the indexes located right after the primer in the adapter? This means, the index sequence would be amplified? (.. and, it will be at the beginning...).

I prepared the samples exactly the same way as before, this is the third set of sequences I sent. The other two, multiplexed too, I sent to other facility and received nice demultiplexed sequences. We did those two as trials and as they came out right, I prepare a whole 8 lane illumina flow cell because we are in hurry and no much people do 76 cycles around here. I have to sent to this facility for same reason: Time. I really hope I have not wasted my time and energy, I really have enthusiasm about this.

C

[Heisman]

How they could have not sequence the indexes? Aren't the indexes located right after the primer in the adapter? This means, the index sequence would be amplified? (.. and, it will be at the beginning...).

I prepared the samples exactly the same way as before, this is the third set of sequences I sent. The other two, multiplexed too, I sent to other facility and received nice demultiplexed sequences. We did those two as trials and as they came out right, I prepare a whole 8 lane illumina flow cell because we are in hurry and no much people do 76 cycles around here. I have to sent to this facility for same reason: Time. I really hope I have not wasted my time and energy, I really have enthusiasm about this.

C

Well I hate to be the potential bearer of bad news, but there is a large distinction between "indexes" and "barcodes". Indexes are placed between the flow cell adapter and the sequencing primer and require a completely separate read. Barcodes are placed at the end of the adapter sequence and will thus be the first bases read in a normal read.

Diagram:

Index
[flowcell adapter][index][sequencing primer][insert][sequencing primer][flowcell adapter]

Barcode
[flowcell adapter][sequencing primer][barcode][insert][sequencing primer][flowcell adapter]

If you used indexes and did not specify that, they may have not included an index read. If you used barcodes, then the barcode sequence should always appear at the end of one of the reads. Could you check a couple of your reads and see if the beginning/ends match any of your barcode sequences (or reverse complements)?

[sklages]

cosmarium, I am worried that you used indexes and the index was never sequenced. If so, I don't see how you can demultiplex. Does this make sense? If that's the case, you definitely need to discuss this with your core facility.

Well, good point .. :-( Would be the worst case, as there is no chance to separate the data ..