That's really strange. It actually should have worked fine whether you set it to t=4, t=8, or t=20, but at least it was successful now. Maybe an OS reinstall is not a bad idea.
Seqanswers Leaderboard Ad
Collapse
Announcement
Collapse
No announcement yet.
X
-
Hi Brian,
I tried to run BBmerge using this command: java -ea -Xmx200m -cp C:\Users\Jeff\Documents\Masters\bbmap\current\ jgi.BBMerge in=C:\Users\Jeff\Documents\Masters\bbmap\mapped_R1.fq out=merged_by_mapping.fq int usemapping parsecustom
And it spit this at me without generating an output:
BBMerge version 4.3
Set INTERLEAVED to true
Setting FASTQ.PARSE_CUSTOM to true
Writing mergable reads merged.
Started output threads.
Note: Filename indicates non-synthetic data, but FASTQ.PARSE_CUSTOM=true
Exception in thread "Thread-4" java.lang.AssertionError:
Mismatch between length of bases and qualities for read 75 (id=412_chr1_0_157870
44_15787280_-32_Glomeraceae_sp|EF558805|SH003500.06FU|reps_singleton|k__Fungi;p_
_Glomeromycota;c__Glomeromycetes;o__Glomerales;f__Glomeraceae;g__unidentified;s_
_Glomeraceae_sp /2).
# qualities=412, # bases=229
FGHHFGGFFGGHGFGHFFHEGGGGGHEHHHHHHGHHHHGEGGGGFHGGGGG@412_chr1_1_15787336_15787565
_260_Glomeraceae_sp|EF558805|SH003500.06FU|reps_singleton|k__Fungi;p__Glomeromyc
ota;c__Glomeromycetes;o__Glomerales;f__Glomeraceae;g__unidentified;s__@413_chr1_
0_10703971_10704200_-32_Ascomycota_sp|FJ708614|SH229315.06FU|reps|k__Fungi;p__As
comycota;c__unidentified;o__unidentified;f__unidentified;g__unidentified;s__Asco
mycota_sp /2
TTTCCGTAGGTGAACCTGCGGAAGGATCATTACTGATTTGGCAAACCGAGCGTTAGCAAGGTTTTGCGATCAACTTATAT
TTAAAACCCACTCTTTATATGTATATTTTTTATTTTGAATGATAATTAAAGATCACTTTCAACAACGGATCTCTTGGCTC
TCGCATCGATGAAGAACGTAGCGACGTGCGATAAGTAATGTGAATTGCAGAATTCCGTGAATCATCGAA
at stream.Read.<init>(Read.java:72)
at stream.Read.<init>(Read.java:45)
at stream.FASTQ.quadToRead(FASTQ.java:756)
at stream.FASTQ.toReadList(FASTQ.java:620)
at stream.FastqReadInputStream.fillBuffer(FastqReadInputStream.java:124)
at stream.FastqReadInputStream.hasMore(FastqReadInputStream.java:80)
at stream.ConcurrentGenericReadInputStream$ReadThread.readLists(Concurre
ntGenericReadInputStream.java:739)
at stream.ConcurrentGenericReadInputStream$ReadThread.run(ConcurrentGene
ricReadInputStream.java:731)
Comment
-
Hmmm, it's working fine for me.
I think the problem is that in your command line you are only specifying one pair for the merging. That's fine if the output from BBMap was interleaved, but not if output was in two files. Try this, instead:
java -ea -Xmx200m -cp C:\Users\Jeff\Documents\Masters\bbmap\current\ jgi.BBMerge in1=C:\Users\Jeff\Documents\Masters\bbmap\mapped_R1.fq in2=C:\Users\Jeff\Documents\Masters\bbmap\mapped_R2.fq out=merged_by_mapping.fq usemapping parsecustom
If that still does not work, please post the first 8 lines of each of the mapped files. It's possible that there is something odd in the headers that is messing with the parsing.
Comment
-
Ugh, sorry I made a sloppy copy/paste error. It worked!
I looked at some quality stats for the final merged file in UPARSE and I noticed that instead of gaps separating the 5' and 3' fragments, there are N's.
i.e. ATCGTCGAATGCNNNNNNNNNNNNNNCACATTACGAGTTAC
This is making quality filtering impossible as UPARSE thinks every sequence is loaded with bad base calls. I thought of just doing a find/replace command to change N's to gaps but I'm afraid it may mess with the headers... What are your thoughts?
Something else that I didn't consider is that since ITS1 is so length variable between taxa, the variable gap size will make end trimming everything to the same length difficult. UPARSE recommends doing this to prevent spurious OTUs.
Anyway... thanks again for your help! Looks like it worked as advertized
Comment
-
Jeff,
I suggest doing quality filtering prior to merging; otherwise the problem with the Ns will be inevitable. For example:
java -ea -Xmx200m -cp /path/ jgi.ReformatReads in1=mapped1.fq in2=mapped2.fq out1=filtered1.fq out2=filtered2.fq maq=15
...where "maq" means "min average quality" (phred scaled) and reads with an average quality below that will be discarded. Pairs will be either discarded or retained together. Then just merge the filtered reads.
Comment
-
Originally posted by Brian Bushnell View PostI've uploaded a new version of BBTools; you can merge them like this:
bbmap.sh -Xmx7g ref=reference.fa in=reads.fq outm=mapped.fq outu=unmapped.fq nodisk po int rbm don
bbmerge.sh in=mapped.fq out=merged_by_mapping.fq int usemapping parsecustom
bbmerge.sh in=unmapped.fq out=merged_by_overlap.fq outu=unmerged.fq int
Then you can concatenate the merged_by_mapping.fq and merged_by_overlap.fq files to get a single file.
Note that these commands all assume you are using a single interleaved file. For two files, you can use the "in1", "in2", "out1", and "out2" flags.
Comment
-
Originally posted by Illusive Man View PostIs this still the same workflow or is there a simpler process? I have amplicons that are about 634 bp in length, and have read1/read2 sequences that are about 300bp in length, hence there is about a ~34bp gap between read1/read2. I would like to merge them with the gaps but so far finding a piece of software that can do this has been very difficult.
Comment
-
BBMAP error?
Toms-Mac-Pro:bbmap tomhanks$ bash bbmap.sh -Xmx10g ref=amoa_archaea.fasta in1=Read1.fastq in2=Read2.fastq outm1=mapped_R1.fq outu1=unmapped_R1.fq outm2=mapped_R2.fq outu2=mapped_R2.fq nodisk po int rbm don
java -Djava.library.path=/Users/bubba/Downloads/bbmap/jni/ -ea -Xmx10g -cp /Users/bubba/Downloads/bbmap/current/ align2.BBMap build=1 overwrite=true fastareadlen=500 -Xmx10g ref=amoa_archaea.fasta in1=Read1.fastq in2=Read2.fastq outm1=mapped_R1.fq outu1=unmapped_R1.fq outm2=mapped_R2.fq outu2=mapped_R2.fq nodisk po int rbm don
Executing align2.BBMap [build=1, overwrite=true, fastareadlen=500, -Xmx10g, ref=amoa_archaea.fasta, in1=Read1.fastq, in2=Read2.fastq, outm1=mapped_R1.fq, outu1=unmapped_R1.fq, outm2=mapped_R2.fq, outu2=mapped_R2.fq, nodisk, po, int, rbm, don]
BBMap version 35.85
Set INTERLEAVED to true
Retaining first best site only for ambiguous mappings.
Executing dna.FastaToChromArrays2 [amoa_archaea.fasta, 1, writeinthread=false, genscaffoldinfo=true, retain, waitforwriting=false, gz=true, maxlen=536670912, writechroms=false, minscaf=1, midpad=300, startpad=8000, stoppad=8000, nodisk=true]
Set genScaffoldInfo=true
Reset INTERLEAVED to false because paired input files were specified.
Set genome to 1
Loaded Reference: 0.005 seconds.
Loading index for chunk 1-1, build 1
Indexing threads started for block 0-1
Indexing threads finished for block 0-1
Generated Index: 0.481 seconds.
Analyzed Index: 4.322 seconds.
Started output stream: 0.016 seconds.
Started output stream: 0.001 seconds.
Cleared Memory: 0.147 seconds.
Processing reads in paired-ended mode.
Started read stream.
Started 8 mapping threads.
Exception in thread "Thread-8" Exception in thread "Thread-7" java.lang.AssertionError: Seems odd so I added this assertion. I don't see anywhere it was being used. Use -da flag to override.
at stream.FASTQ.customID(FASTQ.java:294)
at stream.FASTQ.toFASTQ(FASTQ.java:359)
at stream.Read.toFastq(Read.java:549)
at stream.ReadStreamByteWriter.writeFastq(ReadStreamByteWriter.java:429)
at stream.ReadStreamByteWriter.processJobs(ReadStreamByteWriter.java:90)
at stream.ReadStreamByteWriter.run2(ReadStreamByteWriter.java:41)
at stream.ReadStreamByteWriter.run(ReadStreamByteWriter.java:27)
java.lang.AssertionError: Seems odd so I added this assertion. I don't see anywhere it was being used. Use -da flag to override.
at stream.FASTQ.customID(FASTQ.java:294)
at stream.FASTQ.toFASTQ(FASTQ.java:359)
at stream.Read.toFastq(Read.java:549)
at stream.ReadStreamByteWriter.writeFastq(ReadStreamByteWriter.java:404)
at stream.ReadStreamByteWriter.processJobs(ReadStreamByteWriter.java:90)
at stream.ReadStreamByteWriter.run2(ReadStreamByteWriter.java:41)
at stream.ReadStreamByteWriter.run(ReadStreamByteWriter.java:27)
Exception in thread "Thread-11" java.lang.RuntimeException: Writing to a terminated thread.
at stream.ConcurrentGenericReadOutputStream.write(ConcurrentGenericReadOutputStream.java:198)
at stream.ConcurrentGenericReadOutputStream.addDisordered(ConcurrentGenericReadOutputStream.java:193)
at stream.ConcurrentGenericReadOutputStream.add(ConcurrentGenericReadOutputStream.java:97)
at align2.AbstractMapThread.writeList(AbstractMapThread.java:585)
at align2.AbstractMapThread.run(AbstractMapThread.java:551)
Detecting finished threads: 0Exception in thread "Thread-16" java.lang.RuntimeException: Writing to a terminated thread.
at stream.ConcurrentGenericReadOutputStream.write(ConcurrentGenericReadOutputStream.java:198)
at stream.ConcurrentGenericReadOutputStream.addDisordered(ConcurrentGenericReadOutputStream.java:193)
at stream.ConcurrentGenericReadOutputStream.add(ConcurrentGenericReadOutputStream.java:97)
at align2.AbstractMapThread.writeList(AbstractMapThread.java:585)
at align2.AbstractMapThread.run(AbstractMapThread.java:551)
Exception in thread "Thread-14" java.lang.RuntimeException: Writing to a terminated thread.
at stream.ConcurrentGenericReadOutputStream.write(ConcurrentGenericReadOutputStream.java:198)
at stream.ConcurrentGenericReadOutputStream.addDisordered(ConcurrentGenericReadOutputStream.java:193)
at stream.ConcurrentGenericReadOutputStream.add(ConcurrentGenericReadOutputStream.java:97)
at align2.AbstractMapThread.writeList(AbstractMapThread.java:585)
at align2.AbstractMapThread.run(AbstractMapThread.java:551)
Exception in thread "Thread-17" java.lang.RuntimeException: Writing to a terminated thread.
at stream.ConcurrentGenericReadOutputStream.write(ConcurrentGenericReadOutputStream.java:198)
at stream.ConcurrentGenericReadOutputStream.addDisordered(ConcurrentGenericReadOutputStream.java:193)
at stream.ConcurrentGenericReadOutputStream.add(ConcurrentGenericReadOutputStream.java:97)
at align2.AbstractMapThread.writeList(AbstractMapThread.java:585)
at align2.AbstractMapThread.run(AbstractMapThread.java:551)
Exception in thread "Thread-13" java.lang.RuntimeException: Writing to a terminated thread.
at stream.ConcurrentGenericReadOutputStream.write(ConcurrentGenericReadOutputStream.java:198)
at stream.ConcurrentGenericReadOutputStream.addDisordered(ConcurrentGenericReadOutputStream.java:193)
at stream.ConcurrentGenericReadOutputStream.add(ConcurrentGenericReadOutputStream.java:97)
at align2.AbstractMapThread.writeList(AbstractMapThread.java:585)
at align2.AbstractMapThread.run(AbstractMapThread.java:551)
Exception in thread "Thread-12" java.lang.RuntimeException: Writing to a terminated thread.
at stream.ConcurrentGenericReadOutputStream.write(ConcurrentGenericReadOutputStream.java:198)
at stream.ConcurrentGenericReadOutputStream.addDisordered(ConcurrentGenericReadOutputStream.java:193)
at stream.ConcurrentGenericReadOutputStream.add(ConcurrentGenericReadOutputStream.java:97)
at align2.AbstractMapThread.writeList(AbstractMapThread.java:585)
at align2.AbstractMapThread.run(AbstractMapThread.java:551)
, 1, 2, 3Exception in thread "Thread-15" java.lang.RuntimeException: Writing to a terminated thread.
at stream.ConcurrentGenericReadOutputStream.write(ConcurrentGenericReadOutputStream.java:198)
at stream.ConcurrentGenericReadOutputStream.addDisordered(ConcurrentGenericReadOutputStream.java:193)
at stream.ConcurrentGenericReadOutputStream.add(ConcurrentGenericReadOutputStream.java:97)
at align2.AbstractMapThread.writeList(AbstractMapThread.java:585)
at align2.AbstractMapThread.run(AbstractMapThread.java:551)
, 4, 5, 6Exception in thread "Thread-18" java.lang.RuntimeException: Writing to a terminated thread.
at stream.ConcurrentGenericReadOutputStream.write(ConcurrentGenericReadOutputStream.java:198)
at stream.ConcurrentGenericReadOutputStream.addDisordered(ConcurrentGenericReadOutputStream.java:193)
at stream.ConcurrentGenericReadOutputStream.add(ConcurrentGenericReadOutputStream.java:97)
at align2.AbstractMapThread.writeList(AbstractMapThread.java:585)
at align2.AbstractMapThread.run(AbstractMapThread.java:551)
, 7
**************************************************************************
Warning! 8 mapping threads did not terminate normally.
Check the error log; the output may be corrupt or incomplete.
Please submit the full stderr output as a bug report, not just this message.
**************************************************************************
------------------ Results ------------------
Genome: 1
Key Length: 13
Max Indel: 16000
Minimum Score Ratio: 0.56
Mapping Mode: normal
Reads Used: 3600 (1064189 bases)
Mapping: 14.865 seconds.
Reads/sec: 242.19
kBases/sec: 71.59
Pairing data: pct reads num reads pct bases num bases
mated pairs: 5.0556% 91 5.0974% 54246
bad pairs: 0.0000% 0 0.0000% 0
insert size avg: 553.14
Read 1 data: pct reads num reads pct bases num bases
mapped: 5.0556% 91 5.0919% 27123
unambiguous: 5.0556% 91 5.0919% 27123
ambiguous: 0.0000% 0 0.0000% 0
low-Q discards: 0.0000% 0 0.0000% 0
perfect best site: 0.0000% 0 0.0000% 0
semiperfect site: 0.0000% 0 0.0000% 0
rescued: 0.0000% 0
Match Rate: NA NA 36.8918% 23738
Error Rate: 18.6858% 91 61.8649% 39807
Sub Rate: 18.6858% 91 4.0174% 2585
Del Rate: 8.4189% 41 57.8475% 37222
Ins Rate: 0.0000% 0 0.0000% 0
N Rate: 18.4805% 90 1.2433% 800
Read 2 data: pct reads num reads pct bases num bases
mapped: 5.0556% 91 5.0973% 27093
unambiguous: 5.0556% 91 5.0973% 27093
ambiguous: 0.0000% 0 0.0000% 0
low-Q discards: 0.1667% 3 0.0198% 105
perfect best site: 0.0000% 0 0.0000% 0
semiperfect site: 0.0000% 0 0.0000% 0
rescued: 0.0556% 1
Match Rate: NA NA 84.0296% 22788
Error Rate: 53.2164% 91 15.9335% 4321
Sub Rate: 53.2164% 91 15.6200% 4236
Del Rate: 7.6023% 13 0.0959% 26
Ins Rate: 9.9415% 17 0.2176% 59
N Rate: 0.5848% 1 0.0369% 10
Exception in thread "main" java.lang.RuntimeException: BBMap terminated in an error state; the output may be corrupt.
at align2.BBMap.testSpeed(BBMap.java:486)
at align2.BBMap.main(BBMap.java:33)Last edited by Illusive Man; 03-13-2016, 10:53 AM.
Comment
-
Can you add an explicit threads= option like below and try again:
Code:$ bash bbmap.sh -Xmx10g threads=4 ref=amoa_archaea.fasta in1=Read1.fastq in2=Read2.fastq outm1=mapped_R1.fq outu1=unmapped_R1.fq outm2=mapped_R2.fq outu2=mapped_R2.fq nodisk po int rbm don overwrite=t
Comment
-
Originally posted by GenoMax View PostCan you add an explicit threads= option like below and try again:
Code:$ bash bbmap.sh -Xmx10g -threads=4 ref=amoa_archaea.fasta in1=Read1.fastq in2=Read2.fastq outm1=mapped_R1.fq outu1=unmapped_R1.fq outm2=mapped_R2.fq outu2=mapped_R2.fq nodisk po int rbm don overwrite=t
Exception in thread "main" java.lang.RuntimeException: Unknown parameter: -threads=4
at align2.AbstractMapper.parse(AbstractMapper.java:627)
at align2.AbstractMapper.<init>(AbstractMapper.java:51)
at align2.BBMap.<init>(BBMap.java:41)
at align2.BBMap.main(BBMap.java:29)
Comment
-
Originally posted by Illusive Man View PostBBMap version 35.85
Exception in thread "main" java.lang.RuntimeException: Unknown parameter: -threads=4
at align2.AbstractMapper.parse(AbstractMapper.java:627)
at align2.AbstractMapper.<init>(AbstractMapper.java:51)
at align2.BBMap.<init>(BBMap.java:41)
at align2.BBMap.main(BBMap.java:29)
Comment
-
Originally posted by GenoMax View PostMy fault. Use threads=4 (no hyphen in front). I will edit the example above.
Exception in thread "main" java.lang.RuntimeException: BBMap terminated in an error state; the output may be corrupt.
at align2.BBMap.testSpeed(BBMap.java:486)
at align2.BBMap.main(BBMap.java:33)
Comment
-
Originally posted by GenoMax View Post@Brian should be along later tonight and he will provide a definitive answer.
Comment
Latest Articles
Collapse
-
by seqadmin
The field of epigenetics has traditionally concentrated more on DNA and how changes like methylation and phosphorylation of histones impact gene expression and regulation. However, our increased understanding of RNA modifications and their importance in cellular processes has led to a rise in epitranscriptomics research. “Epitranscriptomics brings together the concepts of epigenetics and gene expression,” explained Adrien Leger, PhD, Principal Research Scientist...-
Channel: Articles
04-22-2024, 07:01 AM -
-
by seqadmin
Proteins are often described as the workhorses of the cell, and identifying their sequences is key to understanding their role in biological processes and disease. Currently, the most common technique used to determine protein sequences is mass spectrometry. While still a valuable tool, mass spectrometry faces several limitations and requires a highly experienced scientist familiar with the equipment to operate it. Additionally, other proteomic methods, like affinity assays, are constrained...-
Channel: Articles
04-04-2024, 04:25 PM -
ad_right_rmr
Collapse
News
Collapse
Topics | Statistics | Last Post | ||
---|---|---|---|---|
Started by seqadmin, Today, 08:06 AM
|
0 responses
11 views
0 likes
|
Last Post
by seqadmin
Today, 08:06 AM
|
||
Started by seqadmin, 04-30-2024, 12:17 PM
|
0 responses
14 views
0 likes
|
Last Post
by seqadmin
04-30-2024, 12:17 PM
|
||
Started by seqadmin, 04-29-2024, 10:49 AM
|
0 responses
19 views
0 likes
|
Last Post
by seqadmin
04-29-2024, 10:49 AM
|
||
Started by seqadmin, 04-25-2024, 11:49 AM
|
0 responses
26 views
0 likes
|
Last Post
by seqadmin
04-25-2024, 11:49 AM
|
Comment