[sahiilseth]

[QUOTE=GenoMax;114232]From Bowtie website:
They also say:
'If your computer has more than 3-4 GB of memory and you would like to exploit that fact to make index building faster, use a 64-bit version of the bowtie2-build binary. The 32-bit version of the binary is restricted to using less than 4 GB of memory. If a 64-bit pre-built binary does not yet exist for your platform on the sourceforge download site, you will need to build one from source.'

I thought 64 bit binary, should be able to handle more characters as well; not true?

[GenoMax]

Hi I am using the latest builds of bowtie 1 and 2 with 64 bit support..

But they are still dying with the error:
Reading reference sizes
Error: Reference sequence has more than 2^32-1 characters! Please divide the
reference into batches or chunks of about 3.6 billion characters or less each

64-bit
Built on do-dmxp-mac.win.ad.jhu.edu
Tue Feb 26 13:33:50 EST 2013
Compiler: gcc version 4.1.2 20080704 (Red Hat 4.1.2-54)
Options: -O3 -m64 -msse2 -funroll-loops -g3
Sizeof {int, long, long long, void*, size_t, off_t}: {4, 8, 8, 8, 8, 8}

From Bowtie website:

Because bowtie2-build uses 32-bit pointers internally, it can handle up to a theoretical maximum of 2^32-1 (somewhat more than 4 billion) characters in an index, though, with other constraints, the actual ceiling is somewhat less than that. If your reference exceeds 2^32-1 characters, bowtie2-build will print an error message and abort. To resolve this, divide your reference sequences into smaller batches and/or chunks and build a separate index for each.

BWA is able to handle genomes > 4 GB in size (individual chromosomes < 2 GB).

[sahiilseth]

bowtie2

Hi I am using the latest builds of bowtie 1 and 2 with 64 bit support..

But they are still dying with the error:
Reading reference sizes
Error: Reference sequence has more than 2^32-1 characters! Please divide the
reference into batches or chunks of about 3.6 billion characters or less each

64-bit
Built on do-dmxp-mac.win.ad.jhu.edu
Tue Feb 26 13:33:50 EST 2013
Compiler: gcc version 4.1.2 20080704 (Red Hat 4.1.2-54)
Options: -O3 -m64 -msse2 -funroll-loops -g3
Sizeof {int, long, long long, void*, size_t, off_t}: {4, 8, 8, 8, 8, 8}

[dpryan]

thanks for your quick reply ! Apparently, can't use bowtie1 then for paired-end alignment.

If this sort of thing is common in your library, then it'd be best not to (unless you use the trim_galore trick that I mentioned). I expect for most libraries this is a rare enough occurrence that bowtie1 works fine (I don't actually use bowtie1 these days, so I can't say I've checked).

[sdm]

You didn't miss anything, bowtie1 doesn't deal well with those (or whenever the start/end coordinates of one reads are found completely within another). I believe that functions normally in bowtie2. Alternatively, trim_galore has an option to get around this in your read-trimming step.

thanks for your quick reply ! Apparently, can't use bowtie1 then for paired-end alignment.

[dpryan]

Hi all,

I have used bowtie in paired-end mode. When I checked the results I don't understand the following result:

these are 2 mates, as far as I understand two identical sequences (forward and reverse), which map to chromosome 6, if I map them individually (bowtie -m1 -v2)

1.sam:MISEQ:2:000000000-A26AB:1:1101:17175:1762 0 chr6 72678938 255 77M * 0 0 GGACAATTAAAAAGCAACAACCACAATTAATACGGTTTACACAGGCAAAACTCATTAAGTGTGGGTTGGGGCGCTCT DDDDB9BFFFFF??C;ECFFFEHFFEEGHFGHGFHHHHHFFFGFCCACFHFHHHHHG-ECFBEEECE>>*5+CCHHH XA:i:1MD:Z:76C0 NM:i:1

2.sam:MISEQ:2:000000000-A26AB:1:1101:17175:1762 16 chr6 72678938 255 77M * 0 0 GGACAATTAAAAAGCAACAACCACAATTAATACGGTTTACACAGGCAAAACTCATTAAGTGTGGGTTGGGGCGCTCT CAC-C5-,FFFCAA,C>+A5+-5-AA--CA+C7A9...A-A.EEA,FEAA../A..CC@+@+@@=<+@@==+<,,5, XA:i:1MD:Z:76C0 NM:i:1

however if I run
bowtie-0.12.7/bowtie --phred33-quals -X 2000 --fr --chunkmbs 300 -p 4 -a -v 2 --sam -q -1 1.fq -2 2.fq > paired.sam

the sequence pair is said to be either unmapped or with an insert size of -1109, it should be 0 in this case?

paired.sam:MISEQ:2:000000000-A26AB:1:1101:17175:1762 1:N:0: 77 * 0 0 * * 0 0 GGACAATTAAAAAGCAACAACCACAATTAATACGGTTTACACAGGCAAAACTCATTAAGTGTGGGTTGGGGCGCTCT DDDDB9BFFFFF??C;ECFFFEHFFEEGHFGHGFHHHHHFFFGFCCACFHFHHHHHG-ECFBEEECE>>*5+CCHHH XM:i:0
raw.paired.sam:MISEQ:2:000000000-A26AB:1:1101:17175:1762 147 chr6 72678938 255 77M = 72677906 -1109 GGACAATTAAAAAGCAACAACCACAATTAATACGGTTTACACAGGCAAAACTCATTAAGTGTGGGTTGGGGCGCTCT CAC-C5-,FFFCAA,C>+A5+-5-AA--CA+C7A9...A-A.EEA,FEAA../A..CC@+@+@@=<+@@==+<,,5, XA:i:1 MD:Z:76C0 NM:i:1

If anybody has an idea what I have missed, I would be very grateful.

You didn't miss anything, bowtie1 doesn't deal well with those (or whenever the start/end coordinates of one reads are found completely within another). I believe that functions normally in bowtie2. Alternatively, trim_galore has an option to get around this in your read-trimming step.

[sdm]

Hi all,

I have used bowtie in paired-end mode. When I checked the results I don't understand the following result:

these are 2 mates, as far as I understand two identical sequences (forward and reverse), which map to chromosome 6, if I map them individually (bowtie -m1 -v2)

1.sam:MISEQ:2:000000000-A26AB:1:1101:17175:1762 0 chr6 72678938 255 77M * 0 0 GGACAATTAAAAAGCAACAACCACAATTAATACGGTTTACACAGGCAAAACTCATTAAGTGTGGGTTGGGGCGCTCT DDDDB9BFFFFF??C;ECFFFEHFFEEGHFGHGFHHHHHFFFGFCCACFHFHHHHHG-ECFBEEECE>>*5+CCHHH XA:i:1MD:Z:76C0 NM:i:1

2.sam:MISEQ:2:000000000-A26AB:1:1101:17175:1762 16 chr6 72678938 255 77M * 0 0 GGACAATTAAAAAGCAACAACCACAATTAATACGGTTTACACAGGCAAAACTCATTAAGTGTGGGTTGGGGCGCTCT CAC-C5-,FFFCAA,C>+A5+-5-AA--CA+C7A9...A-A.EEA,FEAA../A..CC@+@+@@=<+@@==+<,,5, XA:i:1MD:Z:76C0 NM:i:1

however if I run
bowtie-0.12.7/bowtie --phred33-quals -X 2000 --fr --chunkmbs 300 -p 4 -a -v 2 --sam -q -1 1.fq -2 2.fq > paired.sam

the sequence pair is said to be either unmapped or with an insert size of -1109, it should be 0 in this case?

paired.sam:MISEQ:2:000000000-A26AB:1:1101:17175:1762 1:N:0: 77 * 0 0 * * 0 0 GGACAATTAAAAAGCAACAACCACAATTAATACGGTTTACACAGGCAAAACTCATTAAGTGTGGGTTGGGGCGCTCT DDDDB9BFFFFF??C;ECFFFEHFFEEGHFGHGFHHHHHFFFGFCCACFHFHHHHHG-ECFBEEECE>>*5+CCHHH XM:i:0
raw.paired.sam:MISEQ:2:000000000-A26AB:1:1101:17175:1762 147 chr6 72678938 255 77M = 72677906 -1109 GGACAATTAAAAAGCAACAACCACAATTAATACGGTTTACACAGGCAAAACTCATTAAGTGTGGGTTGGGGCGCTCT CAC-C5-,FFFCAA,C>+A5+-5-AA--CA+C7A9...A-A.EEA,FEAA../A..CC@+@+@@=<+@@==+<,,5, XA:i:1 MD:Z:76C0 NM:i:1

If anybody has an idea what I have missed, I would be very grateful.

[dpryan]

Hi thanks for the advice, the problem that I'm working on isn't acutally a bisuphate treated sample but now that you mention it I think the matrix I'm trying to use is fairly similar. However I need the matrix to allow TC/GA pairing on one 'direction' (i.e. Database to query) but not in the other: a t in the DB sequence can align to either a T or a c in the query...Is this tha same as a bisulphate alignemnt?

Thanks

Ah, treating the data as if it were bisulfite might not be the best approach, then. There, one typically in silico converts (for example all C's become T's) both the genome (database) and the read (query) prior to alignment to avoid biased alignments. Depending on what you're really trying to do (giving a bit more detail could prove helpful), this might be the more correct way to go, depending upon the exact underlying nature of your problem.

Should that not prove to be the best option, presumably bowtie (or any other aligner) could be modified. I'm not particularly familiar with its internals, so I couldn't point you toward the right place in the code to start making changes.

[sinclaircooper]

Hi thanks for the advice, the problem that I'm working on isn't acutally a bisuphate treated sample but now that you mention it I think the matrix I'm trying to use is fairly similar. However I need the matrix to allow TC/GA pairing on one 'direction' (i.e. Database to query) but not in the other: a t in the DB sequence can align to either a T or a c in the query...Is this tha same as a bisulphate alignemnt?

Thanks

[dpryan]

Hi all, do any of you know if it is possible to change the matrix which bowtie2 uses for local alignment? If I actually have to alter the source code which part should I be looking at?

I'm trying to use a nucleotide identity matrix that counts T-C and G-A as being the same as T-T and G-G matches.

It sounds like you need to align bisulfite converted reads. If so, you can use bismark, which is a front-end for bowtie.

If you have access to a computer cluster and are comfortable compiling source code, I can also send you a program that I wrote that is similar to bismark, but 5-10x faster (just send me a message with your email address). I hope to post the bismark replacement that I wrote this week.

[sinclaircooper]

Hi all, do any of you know if it is possible to change the matrix which bowtie2 uses for local alignment? If I actually have to alter the source code which part should I be looking at?

I'm trying to use a nucleotide identity matrix that counts T-C and G-A as being the same as T-T and G-G matches.

[kumarS_27]

Hi,

I checked the memory consumption by bowtie-align in the wrapping up stage and it was consuming CPU% 67 VIRT 173MB and RES 52MB, quite alot I would say...but this is with fastq format which was finished successfully. But when I used the fasta, it didnt even appeared in the terminal..and gave me the error.

Anyways, I try with chopping the long length contigs in to smaller one and then see..if it works, it will be clear that Bowtie does have an upper limit on the read length.

I shortened the fasta contigs to 500-1000 and it worked well. I did not try to find a threshold as what upper limit contig length is accepted by Bowtie.

Thanks for suggestions.

[kumarS_27]

The amount of memory would be plenty if you were mapping short reads to a 3 Mb genome, but with the very long contigs, I don't know.

Can you monitor how much memory your PC is using before it produces the error?

I know Bowtie2 is supposed to not have an upper limit for length of reads, but you might be better off using blast to map the contigs back to the reference genome.

Hi,

I checked the memory consumption by bowtie-align in the wrapping up stage and it was consuming CPU% 67 VIRT 173MB and RES 52MB, quite alot I would say...but this is with fastq format which was finished successfully. But when I used the fasta, it didnt even appeared in the terminal..and gave me the error.

Anyways, I try with chopping the long length contigs in to smaller one and then see..if it works, it will be clear that Bowtie does have an upper limit on the read length.

[mastal]

Bowtie, an ultrafast, memory-efficient, open source short read aligner

Hey is there a small tutorial or a book which can teach me bowtie in general. I have read the tutorial which comes with the bowtie software. but that did not teach me the beginner things.
I would really appreciate if there is a beginners guide or something. Could you please help me out. Thanks a lot.

Have you tried working through the examples in the Getting Started section of the Bowtie website?

http://bowtie-bio.sourceforge.net/tutorial.shtml

[M4love]

In the terminal, bowtie has nothing to do with R.

Hey is there a small tutorial or a book which can teach me bowtie in general. I have read the tutorial which comes with the bowtie software. but that did not teach me the beginner things.
I would really appreciate if there is a beginners guide or something. Could you please help me out. Thanks a lot.