[chutfilz]

No dice, same error.

dm3.fa is a file under the subdirectory "WholeGenomeFasta" in the file set downloaded from iGenomes.

Also in this subdirectory are files ending in .dict, .fa.fai, and an xml for genome size. I only imported the .fa to my institution's server for mapping purposes.

cat dm3.fa reveals unannotated sequence, as expected, as well as a few stretches of ~1,000 Ns.

[GenoMax]

I see that you are using samtools module as well. With that you can directly write BAM files no need to use SAM.

So let us try a modified command line and see what happens (I am going to assume that you have ~30G of RAM and 4 cores available for this job in command below and the two fastq files are in the current directory). dm3.fa is just a multi-fasta file of Drosophila chromosomes?

bbmap.sh -Xmx30g threads=4 ref=/users/chutfilz/data/chutfilz/Dm3_Index/dm3.fa in1=PoolCH-1_R1_001_val_1.fa.gz in2=PoolCH-1_R2_001_val_2.fa.gz out=PoolCH-1.bam ambig=random maxindel=10000 trd=t

[chutfilz]

Wow! Thanks for the fast reply. I've pared down my job to just one set of directly-called, paired files to eliminate the possibility of a malfunctioning $i.

bbmap.sh [B]ref=/users/chutfilz/data/chutfilz/Dm3_Index/dm3.fa[/B] in=PoolCH-1_R1_001_val_1.fa.gz in2=PoolCH-1_R2_001_val_2.fa.gz out=PoolCH-1.sam

The ref assignment was my first argument of the previous command as well, and I've retained it in this run (deleted the previous 'ref' folder first, from the previous failed run).

No use in posting the error I received - it's exactly the same as before!

[GenoMax]

Have you verified that the $i variable is expanding correctly?

If you have multiple files to align you should first create an index by doing

bbmap.sh ref=/users/chutfilz/data/chutfilz/Dm3_Index/dm3.fa

this will create a "ref" directory with all index files. Don't worry about what is in the directory (bbmap uses it own organization).

Then when you run use

path=dir_containing_ref_dir

in command line, instead of "ref=". This will avoid re-indexing the genome each time.

If you are using a job scheduler then you should submit each alignment job separately (not the way you have the loop setup, which I assume is submitted as a single job?)

[chutfilz]

Negative Array Size in BBMap

Hello,

I'm trying to map 16 paired sequences. I've already trimmed the adapters using trimgalore!

Input (scheduled job run on a server):

for i in "${samples[@]}"; do
bbmap.sh ref=/users/chutfilz/data/chutfilz/Dm3_Index/dm3.fa in="$i"_R1_001_val_1.fa.gz in2="$i"_R2_001_val_2.fa.gz out="$i".sam
done

Requested 16 cpus, 80g RAM, 24h runtime.

Output:

module: loading 'java/8u111'
module: loading 'bbmap/38.23'
module: loading 'samtools/1.9'
java -ea -Xmx47593m -cp /gpfs/runtime/opt/bbmap/38.23/bin/current/ align2.BBMap build=1 overwrite=true fastareadlen=500 ref=/users/chutfilz/data/chutfilz/Dm3_Index/dm3.fa in=PoolCH-1_R1_001_val_1.fa.gz in2=PoolCH-1_R2_001_val_2.fa.gz out=PoolCH-1.sam
Executing align2.BBMap [build=1, overwrite=true, fastareadlen=500, ref=/users/chutfilz/data/chutfilz/Dm3_Index/dm3.fa, in=PoolCH-1_R1_001_val_1.fa.gz, in2=PoolCH-1_R2_001_val_2.fa.gz, out=PoolCH-1.sam]
Version 38.24

Retaining first best site only for ambiguous mappings.
Writing reference.
Executing dna.FastaToChromArrays2 [/users/chutfilz/data/chutfilz/Dm3_Index/dm3.fa, 1, writeinthread=false, genscaffoldinfo=true, retain, waitforwriting=false, gz=true, maxlen=536670912, writechroms=true, minscaf=1, midpad=300, startpad=8000, stoppad=8000, nodisk=false]

Set genScaffoldInfo=true
Writing chunk 1
Set genome to 1

Loaded Reference:	0.057 seconds.
Loading index for chunk 1-1, build 1
No index available; generating from reference genome: /gpfs/data/mtatar/chutfilz/trimgalore/ref/index/1/chr1_index_k13_c4_b1.block
Indexing threads started for block 0-1
Indexing threads finished for block 0-1
Generated Index:	15.892 seconds.
Analyzed Index:   	2.851 seconds.
Started output stream:	1.332 seconds.
Cleared Memory:    	0.264 seconds.
Processing reads in paired-ended mode.
Started read stream.
Started 16 mapping threads.
[B]Exception in thread "Thread-28" java.lang.NegativeArraySizeException[/B]
	at java.util.Arrays.copyOf(Arrays.java:3236)
	at shared.KillSwitch.copyOf(KillSwitch.java:294)
	at stream.FastaReadInputStream.fillBuffer(FastaReadInputStream.java:447)
	at stream.FastaReadInputStream.nextHeader(FastaReadInputStream.java:290)
	at stream.FastaReadInputStream.fillList(FastaReadInputStream.java:174)
	at stream.FastaReadInputStream.hasMore(FastaReadInputStream.java:107)
	at stream.ConcurrentGenericReadInputStream$ReadThread.readLists(ConcurrentGenericReadInputStream.java:664)
	at stream.ConcurrentGenericReadInputStream$ReadThread.run(ConcurrentGenericReadInputStream.java:653)
[B]Exception in thread "Thread-27" java.lang.NegativeArraySizeException[/B]
	at java.util.Arrays.copyOf(Arrays.java:3236)
	at shared.KillSwitch.copyOf(KillSwitch.java:294)
	at stream.FastaReadInputStream.fillBuffer(FastaReadInputStream.java:447)
	at stream.FastaReadInputStream.nextHeader(FastaReadInputStream.java:290)
	at stream.FastaReadInputStream.fillList(FastaReadInputStream.java:174)
	at stream.FastaReadInputStream.hasMore(FastaReadInputStream.java:107)
	at stream.ConcurrentGenericReadInputStream$ReadThread.readLists(ConcurrentGenericReadInputStream.java:664)
	at stream.ConcurrentGenericReadInputStream$ReadThread.run(ConcurrentGenericReadInputStream.java:653)

PoolCH-1.sam has begun to write, but it's been five hours (these are Drosophila reads), and no progress on the other 15 paired fasta files.

Where is the negative array???

[catagui]

bbmap error in output file

Hi I am getting an error when running bbmap. Any ideas what I am doing wrong?
Also how do I check the mapping stats?

This is what I am running:

#!/bin/bash
cd /space/home/aguilar/Ofav_temp/Trim
/space/home/aguilar/Programs/bbmap/bbwrap.sh t=40 in=\
S1_F_paired_1.fq,S10_F_paired_1.fq,S11_F_paired_1.fq,S12_F_paired_1.fq,S13_F_paired_1.fq,S14_F_paired_1.fq,S15_F_paired_1.fq,S16_F_paired_1.fq,S17_F_paired_1.fq,S18_F_paired_1.fq,S19_F_paired_1.fq,S2_F_paired_1.fq,S20_F_paired_1.fq,S21_F_paired_1.fq,S22_F_paired_1.fq,S23_F_paired_1.fq,S24_F_paired_1.fq,S25_F_paired_1.fq,S26_F_paired_1.fq,S27_F_paired_1.fq,S28_F_paired_1.fq,S29_F_paired_1.fq,S3_F_paired_1.fq,S30_F_paired_1.fq,S31_F_paired_1.fq,S32_F_paired_1.fq,S33_F_paired_1.fq,S34_F_paired_1.fq,S35_F_paired_1.fq,S36_F_paired_1.fq,S37_F_paired_1.fq,S38_F_paired_1.fq,S39_F_paired_1.fq,S4_F_paired_1.fq,S40_F_paired_1.fq,S41_F_paired_1.fq,S42_F_paired_1.fq,S43_F_paired_1.fq,S44_F_paired_1.fq,S45_F_paired_1.fq,S46_F_paired_1.fq,S47_F_paired_1.fq,S48_F_paired_1.fq,S5_F_paired_1.fq,S6_F_paired_1.fq,S7_F_paired_1.fq,S8_F_paired_1.fq,S9_F_paired_1.fq \
in2=S1_R_paired_2.fq,S10_R_paired_2.fq,S11_R_paired_2.fq,S12_R_paired_2.fq,S13_R_paired_2.fq,S14_R_paired_2.fq,S15_R_paired_2.fq,S16_R_paired_2.fq,S17_R_paired_2.fq,S18_R_paired_2.fq,S19_R_paired_2.fq,S2_R_paired_2.fq,S20_R_paired_2.fq,S21_R_paired_2.fq,S22_R_paired_2.fq,S23_R_paired_2.fq,S24_R_paired_2.fq,S25_R_paired_2.fq,S26_R_paired_2.fq,S27_R_paired_2.fq,S28_R_paired_2.fq,S29_R_paired_2.fq,S3_R_paired_2.fq,S30_R_paired_2.fq,S31_R_paired_2.fq,S32_R_paired_2.fq,S33_R_paired_2.fq,S34_R_paired_2.fq,S35_R_paired_2.fq,S36_R_paired_2.fq,S37_R_paired_2.fq,S38_R_paired_2.fq,S39_R_paired_2.fq,S4_R_paired_2.fq,S40_R_paired_2.fq,S41_R_paired_2.fq,S42_R_paired_2.fq,S43_R_paired_2.fq,S44_R_paired_2.fq,S45_R_paired_2.fq,S46_R_paired_2.fq,S47_R_paired_2.fq,S48_R_paired_2.fq,S5_R_paired_2.fq,S6_R_paired_2.fq,S7_R_paired_2.fq,S8_R_paired_2.fq,S9_R_paired_2.fq \
ref=/space/home/aguilar/Ofav_temp/Genomes/Orbicella_faveolata_v2_scaffolds.fa \
outu=/space/home/aguilar/Ofav_temp/bbmap/ReadsUnm.R1.fastq.gz \
outu2=/space/home/aguilar/Ofav_temp/bbmap/ReadsUnmR2.fastq.gz \
outm=/space/home/aguilar/Ofav_temp/bbmap/ReadsMappedR1.fastq.gz \
outm2=/space/home/aguilar/Ofav_temp/bbmap/ReadsMappedR2.fastq.gz \

This is the error:

Retaining first best site only for ambiguous mappings.
No output file.
Exception in thread "main" java.lang.AssertionError: ASCII encoding for quality (currently ASCII-33) appears to be wrong.

Thanks

[catagui]

problem with output

Hi I am having troubles when running bbwrap. Also how can I get a file that tells me that perfectage that was mapped and unmapped.

This is what I am running:

#!/bin/bash
cd /space/home/aguilar/Ofav_temp/Trim
/space/home/aguilar/Programs/bbmap/bbwrap.sh t=40 in=\
S1_F_paired_1.fq,S10_F_paired_1.fq,S11_F_paired_1.fq,S12_F_paired_1.fq,S13_F_paired_1.fq,S14_F_paired_1.fq,S15_F_paired_1.fq,S16_F_paired_1.fq,S17_F_paired_1.fq,S18_F_paired_1.fq,S19_F_paired_1.fq,S2_F_paired_1.fq,S20_F_paired_1.fq,S21_F_paired_1.fq,S22_F_paired_1.fq,S23_F_paired_1.fq,S24_F_paired_1.fq,S25_F_paired_1.fq,S26_F_paired_1.fq,S27_F_paired_1.fq,S28_F_paired_1.fq,S29_F_paired_1.fq,S3_F_paired_1.fq,S30_F_paired_1.fq,S31_F_paired_1.fq,S32_F_paired_1.fq,S33_F_paired_1.fq,S34_F_paired_1.fq,S35_F_paired_1.fq,S36_F_paired_1.fq,S37_F_paired_1.fq,S38_F_paired_1.fq,S39_F_paired_1.fq,S4_F_paired_1.fq,S40_F_paired_1.fq,S41_F_paired_1.fq,S42_F_paired_1.fq,S43_F_paired_1.fq,S44_F_paired_1.fq,S45_F_paired_1.fq,S46_F_paired_1.fq,S47_F_paired_1.fq,S48_F_paired_1.fq,S5_F_paired_1.fq,S6_F_paired_1.fq,S7_F_paired_1.fq,S8_F_paired_1.fq,S9_F_paired_1.fq \
in2=S1_R_paired_2.fq,S10_R_paired_2.fq,S11_R_paired_2.fq,S12_R_paired_2.fq,S13_R_paired_2.fq,S14_R_paired_2.fq,S15_R_paired_2.fq,S16_R_paired_2.fq,S17_R_paired_2.fq,S18_R_paired_2.fq,S19_R_paired_2.fq,S2_R_paired_2.fq,S20_R_paired_2.fq,S21_R_paired_2.fq,S22_R_paired_2.fq,S23_R_paired_2.fq,S24_R_paired_2.fq,S25_R_paired_2.fq,S26_R_paired_2.fq,S27_R_paired_2.fq,S28_R_paired_2.fq,S29_R_paired_2.fq,S3_R_paired_2.fq,S30_R_paired_2.fq,S31_R_paired_2.fq,S32_R_paired_2.fq,S33_R_paired_2.fq,S34_R_paired_2.fq,S35_R_paired_2.fq,S36_R_paired_2.fq,S37_R_paired_2.fq,S38_R_paired_2.fq,S39_R_paired_2.fq,S4_R_paired_2.fq,S40_R_paired_2.fq,S41_R_paired_2.fq,S42_R_paired_2.fq,S43_R_paired_2.fq,S44_R_paired_2.fq,S45_R_paired_2.fq,S46_R_paired_2.fq,S47_R_paired_2.fq,S48_R_paired_2.fq,S5_R_paired_2.fq,S6_R_paired_2.fq,S7_R_paired_2.fq,S8_R_paired_2.fq,S9_R_paired_2.fq \
ref=/space/home/aguilar/Ofav_temp/Genomes/Orbicella_faveolata_v2_scaffolds.fa \
outu=/space/home/aguilar/Ofav_temp/bbmap/ReadsUnm.R1.fastq.gz \
outu2=/space/home/aguilar/Ofav_temp/bbmap/ReadsUnmR2.fastq.gz \
outm=/space/home/aguilar/Ofav_temp/bbmap/ReadsMappedR1.fastq.gz \
outm2=/space/home/aguilar/Ofav_temp/bbmap/ReadsMappedR2.fastq.gz \

And I am getting this message:

Retaining first best site only for ambiguous mappings.
No output file.
Exception in thread "main" java.lang.AssertionError: ASCII encoding for quality (currently ASCII-33) appears to be wrong.
+��[ԽMo3͒,6��+.�7��l�.���®�w�.��}m��2"�����F���#Q�

Thanks

[zeam]

Hi Brain, could you please answer my questions posted here at your convenience? http://seqanswers.com/forums/showthread.php?t=85967

Thanks in advance.

[1989sn1027]

@1989sn1027: Brian has not been participating on SA for last few months. You could try to create a ticket at Source Forge and see if he responds to this report.

Thanks for your directing. I'll give that a shot.

[GenoMax]

@1989sn1027: Brian has not been participating on SA for last few months. You could try to create a ticket at Source Forge and see if he responds to this report.

[1989sn1027]

I see that you are assigning 60G of RAM. Have you tried to assign more and see if it helps?

Thanks for your replying. In my test I've tried adding java RAM upper limit from 20G all the way to 400G. Yet still the "maxlen" arguments of dna.FastaToChromArrays2 hadn't changed, neighter the error message.

[Sarah Muller]

That's great new. I will download it asap . Thanks for sharing!

[GenoMax]

I see that you are assigning 60G of RAM. Have you tried to assign more and see if it helps?

[1989sn1027]

Hi Brian & all,
I'm using BBmap 38.26 with a very big reference genome, and some chromosome in this genome is big enough to break the bbmap ref building session.

Here is the fasta index of this reference:

Chr01 301019445 7 60 61
Chr02 163962470 306036450 60 61
Chr03 261511374 472731635 60 61
Chr04 215701946 738601539 60 61
Chr05 217274494 957898525 60 61
Chr06 219521584 1178794268 60 61
Chr07 222112641 1401974553 60 61
Chr08 153299543 1627789079 60 61
Chr09 238794889 1783643622 60 61
Chr10 205736368 2026418433 60 61
Chr11 220335243 2235583748 60 61
Chr12 229934170 2459591253 60 61
Chr00 714758103 2693357667 60 61

Can see that the longest chromosome is beyond 536670912, which cause a problem like this:

bbmap-38.26/bbmap.sh ref=ref.fasta rebuild=t usemodulo=t -Xmx60g

java -ea -Xmx60g -cp /home/sn/software/bbmap-38.26/current/ align2.BBMap build=1 overwrite=true fastareadlen=500 ref=/home/yangjy/16T4/Genome/GEN181516HEB/_db/Capsicum.annuum.L_Zunla-1_Release_2.0.fasta rebuild=t usemodulo=t -Xmx60g
Executing align2.BBMap [build=1, overwrite=true, fastareadlen=500, ref=/home/yangjy/16T4/Genome/GEN181516HEB/_db/Capsicum.annuum.L_Zunla-1_Release_2.0.fasta, rebuild=t, usemodulo=t, -Xmx60g]
Version 38.26

No output file.
Writing reference.
Executing dna.FastaToChromArrays2 [/home/yangjy/16T4/Genome/GEN181516HEB/_db/Capsicum.annuum.L_Zunla-1_Release_2.0.fasta, 1, writeinthread=false, genscaffoldinfo=true, retain, waitforwriting=false, gz=true, maxlen=536670912, writechroms=true, minscaf=1, midpad=300, startpad=8000, stoppad=8000, nodisk=false]

Set genScaffoldInfo=true
Writing chunk 1
Writing chunk 2
Writing chunk 3
Writing chunk 4
Writing chunk 5
Writing chunk 6
Exception in thread "main" java.lang.AssertionError: 714758103, 8000, 7999, 536670912
at dna.FastaToChromArrays2.makeNextChrom(FastaToChromArrays2.java:440)
at dna.FastaToChromArrays2.makeChroms(FastaToChromArrays2.java:343)
at dna.FastaToChromArrays2.main2(FastaToChromArrays2.java:151)
at align2.RefToIndex.makeIndex(RefToIndex.java:147)
at align2.BBMap.setup(BBMap.java:278)
at align2.AbstractMapper.<init>(AbstractMapper.java:57)
at align2.BBMap.<init>(BBMap.java:43)
at align2.BBMap.main(BBMap.java:31)

I'm pretty sure it's the 'maxlen' argument of dna.FastaToChromArrays2 that is not fit my situation, but I'm not sure how can I fix this.

Did anyone deal with this kinda things before? Any suggestion and discussion is of help! >_<

[csmiller]

usejni and compiled C code in BBTools

I just installed the latest version of the BBTools (38.26), and I notice that the C code provided by the usejni=t flag for some tools has been depreciated / disabled.

I found this in the changelog:

Removed JNI path flag from BBMerge, BBMap, and RQCFilter shell scripts.

and this in docs/compiling.txt:

3) C code. This was developed by Jonathan Rood to accelerate BBMap, BBMerge, and Dedupe, but is currently disabled.

Sure enough, it is commented out in the bbmap.sh code:

        #local CMD="java -Djava.library.path=$NATIVELIBDIR $EA $z -cp $CP align2.BBMap build=1 overwrite=true fastareadlen=500 $@"
        local CMD="java $EA $z -cp $CP align2.BBMap build=1 overwrite=true fastareadlen=500 $@"

If I revert to the previous version of the CMD, with the java.library.path set, then the command runs with the compiled C code just fine.

Why was this disabled? Does this affect previous analyses that used this C code? That is, does the C code contain an error that means usejni=t in previous versions will produce different output than the java-only code? Or was this purely a performance or compatibility issue, or something else?

Sorry if I've missed this already posted somewhere, and thanks in advance for any help.

Chris