[Illusive Man]

My fault. Use threads=4 (no hyphen in front). I will edit the example above.

Thanks for your help! I tried that but it gave the same error unfortunately...
Exception in thread "main" java.lang.RuntimeException: BBMap terminated in an error state; the output may be corrupt.
at align2.BBMap.testSpeed(BBMap.java:486)
at align2.BBMap.main(BBMap.java:33)

[GenoMax]

BBMap version 35.85
Exception in thread "main" java.lang.RuntimeException: Unknown parameter: -threads=4
at align2.AbstractMapper.parse(AbstractMapper.java:627)
at align2.AbstractMapper.<init>(AbstractMapper.java:51)
at align2.BBMap.<init>(BBMap.java:41)
at align2.BBMap.main(BBMap.java:29)

My fault. Use threads=4 (no hyphen in front). I will edit the example above.

[Illusive Man]

Can you add an explicit threads= option like below and try again:

 $ bash bbmap.sh -Xmx10g -threads=4 ref=amoa_archaea.fasta in1=Read1.fastq in2=Read2.fastq outm1=mapped_R1.fq outu1=unmapped_R1.fq outm2=mapped_R2.fq outu2=mapped_R2.fq nodisk po int rbm don overwrite=t

BBMap version 35.85
Exception in thread "main" java.lang.RuntimeException: Unknown parameter: -threads=4
at align2.AbstractMapper.parse(AbstractMapper.java:627)
at align2.AbstractMapper.<init>(AbstractMapper.java:51)
at align2.BBMap.<init>(BBMap.java:41)
at align2.BBMap.main(BBMap.java:29)

[GenoMax]

Can you add an explicit threads= option like below and try again:

 $ bash bbmap.sh -Xmx10g threads=4 ref=amoa_archaea.fasta in1=Read1.fastq in2=Read2.fastq outm1=mapped_R1.fq outu1=unmapped_R1.fq outm2=mapped_R2.fq outu2=mapped_R2.fq nodisk po int rbm don overwrite=t

[Illusive Man]

BBMAP error?

Toms-Mac-Pro:bbmap tomhanks$ bash bbmap.sh -Xmx10g ref=amoa_archaea.fasta in1=Read1.fastq in2=Read2.fastq outm1=mapped_R1.fq outu1=unmapped_R1.fq outm2=mapped_R2.fq outu2=mapped_R2.fq nodisk po int rbm don
java -Djava.library.path=/Users/bubba/Downloads/bbmap/jni/ -ea -Xmx10g -cp /Users/bubba/Downloads/bbmap/current/ align2.BBMap build=1 overwrite=true fastareadlen=500 -Xmx10g ref=amoa_archaea.fasta in1=Read1.fastq in2=Read2.fastq outm1=mapped_R1.fq outu1=unmapped_R1.fq outm2=mapped_R2.fq outu2=mapped_R2.fq nodisk po int rbm don
Executing align2.BBMap [build=1, overwrite=true, fastareadlen=500, -Xmx10g, ref=amoa_archaea.fasta, in1=Read1.fastq, in2=Read2.fastq, outm1=mapped_R1.fq, outu1=unmapped_R1.fq, outm2=mapped_R2.fq, outu2=mapped_R2.fq, nodisk, po, int, rbm, don]

BBMap version 35.85
Set INTERLEAVED to true
Retaining first best site only for ambiguous mappings.
Executing dna.FastaToChromArrays2 [amoa_archaea.fasta, 1, writeinthread=false, genscaffoldinfo=true, retain, waitforwriting=false, gz=true, maxlen=536670912, writechroms=false, minscaf=1, midpad=300, startpad=8000, stoppad=8000, nodisk=true]

Set genScaffoldInfo=true
Reset INTERLEAVED to false because paired input files were specified.
Set genome to 1

Loaded Reference: 0.005 seconds.
Loading index for chunk 1-1, build 1
Indexing threads started for block 0-1
Indexing threads finished for block 0-1
Generated Index: 0.481 seconds.
Analyzed Index: 4.322 seconds.
Started output stream: 0.016 seconds.
Started output stream: 0.001 seconds.
Cleared Memory: 0.147 seconds.
Processing reads in paired-ended mode.
Started read stream.
Started 8 mapping threads.
Exception in thread "Thread-8" Exception in thread "Thread-7" java.lang.AssertionError: Seems odd so I added this assertion. I don't see anywhere it was being used. Use -da flag to override.
at stream.FASTQ.customID(FASTQ.java:294)
at stream.FASTQ.toFASTQ(FASTQ.java:359)
at stream.Read.toFastq(Read.java:549)
at stream.ReadStreamByteWriter.writeFastq(ReadStreamByteWriter.java:429)
at stream.ReadStreamByteWriter.processJobs(ReadStreamByteWriter.java:90)
at stream.ReadStreamByteWriter.run2(ReadStreamByteWriter.java:41)
at stream.ReadStreamByteWriter.run(ReadStreamByteWriter.java:27)
java.lang.AssertionError: Seems odd so I added this assertion. I don't see anywhere it was being used. Use -da flag to override.
at stream.FASTQ.customID(FASTQ.java:294)
at stream.FASTQ.toFASTQ(FASTQ.java:359)
at stream.Read.toFastq(Read.java:549)
at stream.ReadStreamByteWriter.writeFastq(ReadStreamByteWriter.java:404)
at stream.ReadStreamByteWriter.processJobs(ReadStreamByteWriter.java:90)
at stream.ReadStreamByteWriter.run2(ReadStreamByteWriter.java:41)
at stream.ReadStreamByteWriter.run(ReadStreamByteWriter.java:27)
Exception in thread "Thread-11" java.lang.RuntimeException: Writing to a terminated thread.
at stream.ConcurrentGenericReadOutputStream.write(ConcurrentGenericReadOutputStream.java:198)
at stream.ConcurrentGenericReadOutputStream.addDisordered(ConcurrentGenericReadOutputStream.java:193)
at stream.ConcurrentGenericReadOutputStream.add(ConcurrentGenericReadOutputStream.java:97)
at align2.AbstractMapThread.writeList(AbstractMapThread.java:585)
at align2.AbstractMapThread.run(AbstractMapThread.java:551)
Detecting finished threads: 0Exception in thread "Thread-16" java.lang.RuntimeException: Writing to a terminated thread.
at stream.ConcurrentGenericReadOutputStream.write(ConcurrentGenericReadOutputStream.java:198)
at stream.ConcurrentGenericReadOutputStream.addDisordered(ConcurrentGenericReadOutputStream.java:193)
at stream.ConcurrentGenericReadOutputStream.add(ConcurrentGenericReadOutputStream.java:97)
at align2.AbstractMapThread.writeList(AbstractMapThread.java:585)
at align2.AbstractMapThread.run(AbstractMapThread.java:551)
Exception in thread "Thread-14" java.lang.RuntimeException: Writing to a terminated thread.
at stream.ConcurrentGenericReadOutputStream.write(ConcurrentGenericReadOutputStream.java:198)
at stream.ConcurrentGenericReadOutputStream.addDisordered(ConcurrentGenericReadOutputStream.java:193)
at stream.ConcurrentGenericReadOutputStream.add(ConcurrentGenericReadOutputStream.java:97)
at align2.AbstractMapThread.writeList(AbstractMapThread.java:585)
at align2.AbstractMapThread.run(AbstractMapThread.java:551)
Exception in thread "Thread-17" java.lang.RuntimeException: Writing to a terminated thread.
at stream.ConcurrentGenericReadOutputStream.write(ConcurrentGenericReadOutputStream.java:198)
at stream.ConcurrentGenericReadOutputStream.addDisordered(ConcurrentGenericReadOutputStream.java:193)
at stream.ConcurrentGenericReadOutputStream.add(ConcurrentGenericReadOutputStream.java:97)
at align2.AbstractMapThread.writeList(AbstractMapThread.java:585)
at align2.AbstractMapThread.run(AbstractMapThread.java:551)
Exception in thread "Thread-13" java.lang.RuntimeException: Writing to a terminated thread.
at stream.ConcurrentGenericReadOutputStream.write(ConcurrentGenericReadOutputStream.java:198)
at stream.ConcurrentGenericReadOutputStream.addDisordered(ConcurrentGenericReadOutputStream.java:193)
at stream.ConcurrentGenericReadOutputStream.add(ConcurrentGenericReadOutputStream.java:97)
at align2.AbstractMapThread.writeList(AbstractMapThread.java:585)
at align2.AbstractMapThread.run(AbstractMapThread.java:551)
Exception in thread "Thread-12" java.lang.RuntimeException: Writing to a terminated thread.
at stream.ConcurrentGenericReadOutputStream.write(ConcurrentGenericReadOutputStream.java:198)
at stream.ConcurrentGenericReadOutputStream.addDisordered(ConcurrentGenericReadOutputStream.java:193)
at stream.ConcurrentGenericReadOutputStream.add(ConcurrentGenericReadOutputStream.java:97)
at align2.AbstractMapThread.writeList(AbstractMapThread.java:585)
at align2.AbstractMapThread.run(AbstractMapThread.java:551)
, 1, 2, 3Exception in thread "Thread-15" java.lang.RuntimeException: Writing to a terminated thread.
at stream.ConcurrentGenericReadOutputStream.write(ConcurrentGenericReadOutputStream.java:198)
at stream.ConcurrentGenericReadOutputStream.addDisordered(ConcurrentGenericReadOutputStream.java:193)
at stream.ConcurrentGenericReadOutputStream.add(ConcurrentGenericReadOutputStream.java:97)
at align2.AbstractMapThread.writeList(AbstractMapThread.java:585)
at align2.AbstractMapThread.run(AbstractMapThread.java:551)
, 4, 5, 6Exception in thread "Thread-18" java.lang.RuntimeException: Writing to a terminated thread.
at stream.ConcurrentGenericReadOutputStream.write(ConcurrentGenericReadOutputStream.java:198)
at stream.ConcurrentGenericReadOutputStream.addDisordered(ConcurrentGenericReadOutputStream.java:193)
at stream.ConcurrentGenericReadOutputStream.add(ConcurrentGenericReadOutputStream.java:97)
at align2.AbstractMapThread.writeList(AbstractMapThread.java:585)
at align2.AbstractMapThread.run(AbstractMapThread.java:551)
, 7

**************************************************************************
Warning! 8 mapping threads did not terminate normally.
Check the error log; the output may be corrupt or incomplete.
Please submit the full stderr output as a bug report, not just this message.
**************************************************************************




------------------ Results ------------------

Genome: 1
Key Length: 13
Max Indel: 16000
Minimum Score Ratio: 0.56
Mapping Mode: normal
Reads Used: 3600 (1064189 bases)

Mapping: 14.865 seconds.
Reads/sec: 242.19
kBases/sec: 71.59


Pairing data: pct reads num reads pct bases num bases

mated pairs: 5.0556% 91 5.0974% 54246
bad pairs: 0.0000% 0 0.0000% 0
insert size avg: 553.14


Read 1 data: pct reads num reads pct bases num bases

mapped: 5.0556% 91 5.0919% 27123
unambiguous: 5.0556% 91 5.0919% 27123
ambiguous: 0.0000% 0 0.0000% 0
low-Q discards: 0.0000% 0 0.0000% 0

perfect best site: 0.0000% 0 0.0000% 0
semiperfect site: 0.0000% 0 0.0000% 0
rescued: 0.0000% 0

Match Rate: NA NA 36.8918% 23738
Error Rate: 18.6858% 91 61.8649% 39807
Sub Rate: 18.6858% 91 4.0174% 2585
Del Rate: 8.4189% 41 57.8475% 37222
Ins Rate: 0.0000% 0 0.0000% 0
N Rate: 18.4805% 90 1.2433% 800


Read 2 data: pct reads num reads pct bases num bases

mapped: 5.0556% 91 5.0973% 27093
unambiguous: 5.0556% 91 5.0973% 27093
ambiguous: 0.0000% 0 0.0000% 0
low-Q discards: 0.1667% 3 0.0198% 105

perfect best site: 0.0000% 0 0.0000% 0
semiperfect site: 0.0000% 0 0.0000% 0
rescued: 0.0556% 1

Match Rate: NA NA 84.0296% 22788
Error Rate: 53.2164% 91 15.9335% 4321
Sub Rate: 53.2164% 91 15.6200% 4236
Del Rate: 7.6023% 13 0.0959% 26
Ins Rate: 9.9415% 17 0.2176% 59
N Rate: 0.5848% 1 0.0369% 10
Exception in thread "main" java.lang.RuntimeException: BBMap terminated in an error state; the output may be corrupt.
at align2.BBMap.testSpeed(BBMap.java:486)
at align2.BBMap.main(BBMap.java:33)

[westerman]

Is this still the same workflow or is there a simpler process? I have amplicons that are about 634 bp in length, and have read1/read2 sequences that are about 300bp in length, hence there is about a ~34bp gap between read1/read2. I would like to merge them with the gaps but so far finding a piece of software that can do this has been very difficult.

That workflow should still be valid (although I have not run it). Basically (1) map the reads, (2) merge the reads that map, (3) merge the reads that do not map. I do not think that the method could be made more simple.

[Illusive Man]

I've uploaded a new version of BBTools; you can merge them like this:

bbmap.sh -Xmx7g ref=reference.fa in=reads.fq outm=mapped.fq outu=unmapped.fq nodisk po int rbm don

bbmerge.sh in=mapped.fq out=merged_by_mapping.fq int usemapping parsecustom

bbmerge.sh in=unmapped.fq out=merged_by_overlap.fq outu=unmerged.fq int

Then you can concatenate the merged_by_mapping.fq and merged_by_overlap.fq files to get a single file.

Note that these commands all assume you are using a single interleaved file. For two files, you can use the "in1", "in2", "out1", and "out2" flags.

Is this still the same workflow or is there a simpler process? I have amplicons that are about 634 bp in length, and have read1/read2 sequences that are about 300bp in length, hence there is about a ~34bp gap between read1/read2. I would like to merge them with the gaps but so far finding a piece of software that can do this has been very difficult.

[jstrohm]

OK sounds good, I was worried that filtering prior to merging would change the read orders in the two files... complicating the merging process

[Brian Bushnell]

Jeff,

I suggest doing quality filtering prior to merging; otherwise the problem with the Ns will be inevitable. For example:
java -ea -Xmx200m -cp /path/ jgi.ReformatReads in1=mapped1.fq in2=mapped2.fq out1=filtered1.fq out2=filtered2.fq maq=15

...where "maq" means "min average quality" (phred scaled) and reads with an average quality below that will be discarded. Pairs will be either discarded or retained together. Then just merge the filtered reads.

[jstrohm]

Ugh, sorry I made a sloppy copy/paste error. It worked!

I looked at some quality stats for the final merged file in UPARSE and I noticed that instead of gaps separating the 5' and 3' fragments, there are N's.

i.e. ATCGTCGAATGCNNNNNNNNNNNNNNCACATTACGAGTTAC

This is making quality filtering impossible as UPARSE thinks every sequence is loaded with bad base calls. I thought of just doing a find/replace command to change N's to gaps but I'm afraid it may mess with the headers... What are your thoughts?

Something else that I didn't consider is that since ITS1 is so length variable between taxa, the variable gap size will make end trimming everything to the same length difficult. UPARSE recommends doing this to prevent spurious OTUs.

Anyway... thanks again for your help! Looks like it worked as advertized

[Brian Bushnell]

Hmmm, it's working fine for me.

I think the problem is that in your command line you are only specifying one pair for the merging. That's fine if the output from BBMap was interleaved, but not if output was in two files. Try this, instead:

java -ea -Xmx200m -cp C:\Users\Jeff\Documents\Masters\bbmap\current\ jgi.BBMerge in1=C:\Users\Jeff\Documents\Masters\bbmap\mapped_R1.fq in2=C:\Users\Jeff\Documents\Masters\bbmap\mapped_R2.fq out=merged_by_mapping.fq usemapping parsecustom

If that still does not work, please post the first 8 lines of each of the mapped files. It's possible that there is something odd in the headers that is messing with the parsing.

[jstrohm]

Hi Brian,

I tried to run BBmerge using this command: java -ea -Xmx200m -cp C:\Users\Jeff\Documents\Masters\bbmap\current\ jgi.BBMerge in=C:\Users\Jeff\Documents\Masters\bbmap\mapped_R1.fq out=merged_by_mapping.fq int usemapping parsecustom

And it spit this at me without generating an output:

BBMerge version 4.3
Set INTERLEAVED to true
Setting FASTQ.PARSE_CUSTOM to true
Writing mergable reads merged.
Started output threads.
Note: Filename indicates non-synthetic data, but FASTQ.PARSE_CUSTOM=true
Exception in thread "Thread-4" java.lang.AssertionError:
Mismatch between length of bases and qualities for read 75 (id=412_chr1_0_157870
44_15787280_-32_Glomeraceae_sp|EF558805|SH003500.06FU|reps_singleton|k__Fungi;p_
_Glomeromycota;c__Glomeromycetes;o__Glomerales;f__Glomeraceae;g__unidentified;s_
_Glomeraceae_sp /2).
# qualities=412, # bases=229

FGHHFGGFFGGHGFGHFFHEGGGGGHEHHHHHHGHHHHGEGGGGFHGGGGG@412_chr1_1_15787336_15787565
_260_Glomeraceae_sp|EF558805|SH003500.06FU|reps_singleton|k__Fungi;p__Glomeromyc
ota;c__Glomeromycetes;o__Glomerales;f__Glomeraceae;g__unidentified;s__@413_chr1_
0_10703971_10704200_-32_Ascomycota_sp|FJ708614|SH229315.06FU|reps|k__Fungi;p__As
comycota;c__unidentified;o__unidentified;f__unidentified;g__unidentified;s__Asco
mycota_sp /2
TTTCCGTAGGTGAACCTGCGGAAGGATCATTACTGATTTGGCAAACCGAGCGTTAGCAAGGTTTTGCGATCAACTTATAT
TTAAAACCCACTCTTTATATGTATATTTTTTATTTTGAATGATAATTAAAGATCACTTTCAACAACGGATCTCTTGGCTC
TCGCATCGATGAAGAACGTAGCGACGTGCGATAAGTAATGTGAATTGCAGAATTCCGTGAATCATCGAA

at stream.Read.<init>(Read.java:72)
at stream.Read.<init>(Read.java:45)
at stream.FASTQ.quadToRead(FASTQ.java:756)
at stream.FASTQ.toReadList(FASTQ.java:620)
at stream.FastqReadInputStream.fillBuffer(FastqReadInputStream.java:124)

at stream.FastqReadInputStream.hasMore(FastqReadInputStream.java:80)
at stream.ConcurrentGenericReadInputStream$ReadThread.readLists(Concurre
ntGenericReadInputStream.java:739)
at stream.ConcurrentGenericReadInputStream$ReadThread.run(ConcurrentGene
ricReadInputStream.java:731)

[Brian Bushnell]

That's really strange. It actually should have worked fine whether you set it to t=4, t=8, or t=20, but at least it was successful now. Maybe an OS reinstall is not a bad idea.

[jstrohm]

Hang on! I made a rookie mistake... I have a quad core processor with 8 "logical processors". So I can't actually run 8 parallel jobs or threads. I dialed it back to t=4 and the job ran in 10 minutes!

When I selected t=8 I had to abort it after three hours.

[jstrohm]

OK! It's running through windows now. It's using 99% of CPU resources and 1GB of RAM. It's still running after an hour though...

Previously, I tried merging these reads with UPARSE and essentially none of them overlap leading me to try this method. Is there any reason why that would slow things down? I didn't actually try the "fast" command yet. You said it slightly decreases sensitivity so I wanted to see if I could get away without it.

I may be forced to give up and just use the 250bp forward reads. Anyway... thanks a bunch for your efforts and assistance!

With regards to my VirtualBox, I used to run scripts that would auto-detect PC resources. The OS has been acting a little strange since I upgraded to the latest version of Ubuntu. It's probably time for a fresh install, but that would require some serious moving about of files...