[mdavrandi]

Hi Brian! I have a question: I am working on a fungal ITS metagenomic amplicon library with a pretty wide variation in sizes (200-500 bp). We are doing 2x300, and my second reads are a little bit lower in quality compared to the firsts. Is there any setting on the BBMerge that I should modify in order to get the most out of the data? I'm pretty new to the field, so please let me know if you need more information! Thank you.

Hi Peerah,

We are having the same problem in our lab with 2x300 miseq runs- very poor Read 2 >Q30 scores- and I was wondering if Brian`s recommendation improved the number of paired-sequences you obtained from that run.

Cheers

[Brian Bushnell]

Hi! With that range you should have a worst a 100bp overlap, which is plenty. But 2x300 MiSeq runs have had major quality problems in the past, so it's possible that trimming would help. I'd suggest adding the flags "qtrim2 trimq=10,15". This will first try to merge the reads, and if unsuccessful (because the quality was too low so there were too many mismatches) quality-trim to Q10 on the right side and retry; then if still unsuccessful do the same at Q15. This isn't necessary unless the data is quite bad, but it will generally increase your merge rate, and is better than simply quality-trimming all reads prior to merging.

[peerah]

Hi Brian! I have a question: I am working on a fungal ITS metagenomic amplicon library with a pretty wide variation in sizes (200-500 bp). We are doing 2x300, and my second reads are a little bit lower in quality compared to the firsts. Is there any setting on the BBMerge that I should modify in order to get the most out of the data? I'm pretty new to the field, so please let me know if you need more information! Thank you.

[Brian Bushnell]

If you know your adapter sequences (or have a list of typical adapter sequences, or actually, you can just say "adapter=default"), you can do this:

bbmerge.sh in=reads.fq adapter=adapter.fa out=merged.fq outu=unmerged.fq

If BBMerge thinks that you still have untrimmed adapters in those cases... I am quite confident it is correct. Adapter-trimming programs are not perfect (nor is BBMerge or BBDuk). I recommend BBDuk for adapter-trimming because it uses both adapter sequences and overlap information (very conservatively), but you will still end up with some untrimmed reads that actually had adapters. The problem is that Illumina sequence quality declines with each cycle, so by the end of the read (the part that typically overlaps, or has adapter sequence) the error rate can be pretty high. If you use an adapter-trimming program that solely uses sequence-matching to a list of provided adapter sequences, then the high mismatch rate will yield poor adapter-trimming for low-quality reads. BBDuk with the "tbo" flag uses both adapter sequences and overlap information, which for short-insert reads, gives added weight to the high-quality initial bases in a read pair.

So - it's not surprising that Trimmomatic did not do complete trimming. I recommend you use BBDuk instead. It still won't give perfect adapter-trimming, but it will be much better than Trimmomatic.

[j.m.c]

Thank you for your reply.

Yes, my reads were 87 bp after trimming with trimmomatic. I had also removed adapter sequences with trimmomatic and now I think I see the issue if understood correctly what you said:

"The 35bp reads you ended up with are because of the short insert. When you have 2x87bp reads with a 35bp insert, you get 35bp of overlap on the 3' end and then 52bp of the 5' end overhanging on each side; that's adapter sequence. BBMerge trims that off so you are left with only the 35bp of genomic sequence. "

That means the overhangs are removed since BBmerge thinks they are adapter sequences. My reads are from RNA-seq data (not genomic data, I am sorry I didn't specify earlier) and since I removed adapter sequences with trimmomatic, I am actually loosing data if the 5' overhangs were trimmed off...

Is there any way to prevent that with BBmerge?

Otherwise I will try BBmerge with my raw reads without removing adapters.

Thanks!

[tshalev]

The latest version, release 36_62. I'll keep you posted.

[Brian Bushnell]

My overall consensus was that merging and then assembling with both merged and unmerged reads produced better assemblies than not merging, over four different assemblers (Trinity, Velvet+Oases, SOAPdenovoTrans and transABySS). This was gauged using the optimized assembly score from Transrate, as well as by assembly completeness as measured by BUSCO and and contiguity as measured by Conditional Reciprocal Best BLAST (from Transrate) against gene sets of other conifer species. In all cases the gains were enough to warrant the use of merging.

Great, thanks for that info!

I'm interested to see now how using some of these new features will affect my assembly. I already see an increase in merging rate from about ~57% to ~83% using the verystrict parameter, although I won't know whether this includes false positives until I assemble.

OK, please let me know the results - it's useful for giving people guidance on when to use rem flag. I've never tried it in conjunction with RNA-seq data, just isolates, metagenomes, and single-cell, though it improved all of those cases.

Regarding adapters expected versus adapters found, I'm seeing ~430000 adapters expected versus ~6000 adapters found in ~91.5 million reads after adapter trimming, so I guess this is good?

That indicates the adapter trimming was fairly complete. What version of BBMap are you using, by the way?

[tshalev]

@Brian Bushnell

I am working with foliage tissue from a species of coniferous tree. I'm using 100bp reads, on Illumina HiSeq 4000. I did actually do the comparison tests about a year ago for using merging vs. not merging, on some different data that I had. For these I trimmed first using Trimmomatic though and did not use kmer information or adapter recognition (not sure if these were implemented in BBMerge back then).

My overall consensus was that merging and then assembling with both merged and unmerged reads produced better assemblies than not merging, over four different assemblers (Trinity, Velvet+Oases, SOAPdenovoTrans and transABySS). This was gauged using the optimized assembly score from Transrate, as well as by assembly completeness as measured by BUSCO and and contiguity as measured by Conditional Reciprocal Best BLAST (from Transrate) against gene sets of other conifer species. In all cases the gains were enough to warrant the use of merging.

I'm interested to see now how using some of these new features will affect my assembly. I already see an increase in merging rate from about ~57% to ~83% using the verystrict parameter, although I won't know whether this includes false positives until I assemble. Regarding adapters expected versus adapters found, I'm seeing ~430000 adapters expected versus ~6000 adapters found in ~91.5 million reads after adapter trimming, so I guess this is good?

[Brian Bushnell]

The primary reason people use read merging is for 16S or other amplicon analyses, I believe. But I don't personally work with 16S very often, so BBMerge is designed and optimized for improving assemblies. Of course, it works on 16S as well, but I use it to optimize assembly pipelines. That said, I have never used Trinity, so I don't know how it would affect a Trinity assembly. As long as you assemble with both the merged and unmerged reads, most assemblers benefit from BBMerge (some quite a lot) so I would expect it to improve a Trinity assembly, but I'd be interested to hear what you experience, if you have the time and interest to run Trinity both ways. Of course RNA-seq assembly quality is especially hard to measure, but metrics like mapping rate, N50, and size are at least somewhat useful.

What kind of assembly are you doing, how long are your reads, and what organism? Is it just RNA-seq?

[tshalev]

@Brian Bushnell

Ah OK, I see. So it "expects" to not find the adapter sequence there, since it has hopefully been removed by BBDuk. Slightly unrelated, I am using RNA-Seq data for a coniferous tree species, and am assembling using conventional assemblers such as Trinity, Velvet-Oases, etc. BBMerge is appropriate for this purpose, right? I keep noticing a lot of threads talking about 16S data, or amplicon data, and I haven't even heard of the assemblers that you mentioned .

Thanks!

[Brian Bushnell]

My questions:

How can I get merged reads that are 35 bases long if my input PE reads were 87 bases long?

Is there something wrong in my commands?

Your commands look fine, although I'm a little confused about what you mean by 87-bp reads. Were they initially 87 bp long, or was that the average after trimming? Also, I suggest trying BBMerge prior to quality-trimming unless you have very low quality data, but you can also try either way and see which gives a higher merge rate.

Note that BBMerge's default "mininsert" is 35; it does not, by default, look for overlaps indicating an insert size smaller than that. If you have a lot of short-insert reads you may want to set it lower, or just do adapter-trimming prior to BBMerge and throw away the short junk. You can do adapter-trimming with BBDuk as in the above post.

The 35bp reads you ended up with are because of the short insert. When you have 2x87bp reads with a 35bp insert, you get 35bp of overlap on the 3' end and then 52bp of the 5' end overhanging on each side; that's adapter sequence. BBMerge trims that off so you are left with only the 35bp of genomic sequence.

[Brian Bushnell]

@tshalev:

Hi! Sorry it's a little convoluted. Well, allow me to explain:

First off, yep, BBDuk was a typo, thanks for catching it. Secondly, I would change your commands to something like this:

bbduk.sh in=reads.fq out=clean.fq ref=adapters.fa ktrim=r k=23 mink=11 hdist=1 tpe tbo

bbmerge-auto.sh in=clean.fq out=merged.fq [B]outu=unmerged.fq[/B] adapter1=something adapter2=something rem k=62 extend2=50 ecct

bbduk.sh in=merged.fq out=merged_trimmed.fq qtrim=r[B]l[/B] trimq=10

[B]bbduk.sh in=unmerged.fq out=unmerged_trimmed qtrim=r trimq=10[/B]

I always recommend keeping the unmerged reads and using those too for the assembler, particularly if you are assembling with Spades. Some reads, like those in low-complexity areas, will not merge without a very long overlap, so tossing out unmerged reads can incur bias.

As for using adapter sequences for BBMerge - that reduces the false-positive rate. If two reads appear to have a good overlap for merging, but that overlap indicates the insert size is shorter than read length, then if and only if you have specified adapter sequences, BBMerge will look to see if adapter sequences are in the expected locations (the overhanging part outside of the overlap). If there are not adapter sequences there, it will assume that was a false positive and the merge will be rejected. Therefore, if you already did adapter-trimming, all of these pairs that merge with insert shorter than read length should be false positives, but BBMerge won't know that unless you actually give it the adapter sequences. So it will print at the end something like "300 adapter sequences expected, 7 found" or similar, indicating that 293 false-positive merges were prevented.

As for over-processing... that's a good thing to worry about. All the processing steps are beneficial as long as the assembly improves, so I often like to assemble after every step to determine its impact. Adapter-trimming is basically always beneficial. Merging depends on the library, the sequencing mode, and the assembler, but is (in my experience) always beneficial with Spades, Tadpole and Ray, and not beneficial with Megahit. Quality-trimming is generally beneficial if you pick the correct cutoff, but 10 is a nice, conservative value that should generally be fine.

[j.m.c]

BBmerge smaller insert sizes than expected

Hi,

I just run BBmerge on a set of paired-end reads (illumina 2x100bp) that had been trimmed with trimmomatic to remove adapter sequences and low quality bases at the end of the reads. Trimmed reads (87 bases long) were merged with BBmerge (following the recommended command for optimal accuracy) and the merged-reads had an insert size that ranged from 35 to 265 bases with an avg of 124 bases.

My questions:

How can I get merged reads that are 35 bases long if my input PE reads were 87 bases long?

Is there something wrong in my commands?

Below the code I used with trimmomatic and BBmerge:


trimmomatic-0.36.jar PE -threads 4 -trimlog YC1.trimlog.txt YC1_R1.fastq YC1_R2.fastq YC1_R1.trimmed.paired.fastq YC1_R1.trimmed.unpaired.fastq YC1_R2.trimmed.paired.fastq YC1_R2.trimmed.unpaired.fastq HEADCROP:13 SLIDINGWINDOW:4:15 MINLEN:60 &

/usr/local/BBMap_36.62/bbmerge.sh in1=YC1_R1.trimmed.paired.fastq in2=YC1_R2.trimmed.paired.fastq out=YC1.trimmed.paired.merged.fastq outu1=YC1_R1.trimmed.paired.unmerged.fastq outu2=YC1_R2.trimmed.paired.unmerged.fastq rem k=62 extend2=50 ecct -Xmx48000m -threads=6 &

Thank you very much!

[tshalev]

Confusion regarding quality/adapter trimming and read merging

Hi everyone,

First off, I'd like to say thanks Brian, BBMerge has been really useful in speeding up my assemblies and improving assembly quality.

I'm having some confusion regarding how to combine trimming and merging. In the quick guide for BBMerge, it says:

"Adapter-trimming reads is fine and is recommended prior to running BBDuk, particularly if kmers will be used. Quality-trimming is usually not recommended, unless the library has major quality issues at the right end of reads resulting in a low merge rate, in which case only weak trimming (such as qtrim=r trimq=8) should be used."

I'm not sure if "BBDuk" is a typo, as this is the guide for BBMerge. Basically I understand that the process needs to be something like this:

1)

 bbduk.sh in=reads.fq out=clean.fq ref=adapters.fa ktrim=r k=23 mink=11 hdist=1 tpe tbo

To trim adapters;

2)

bbmerge-auto.sh in=clean.fq out=merged.fq adapter1=something adapter2=something rem k=62 extend2=50 ecct

To merge reads;

3)

bbduk.sh in=merged.fq out=merged_trimmed.fq qtrim=r trimq=10

To further trim poor quality reads.

My main issues here are:

1) This seems a bit convoluted, and feels like too much messing with the data.

2) I'm still unsure whether I am actually supposed to trim adapters before merging, and if so, does it make sense to use the adapter flags in BBMerge? Can it use the adapter information if I've already trimmed them?

Thanks!

[Brian Bushnell]

@Moistplus

To clarify, "mix" puts both merged and unmerged reads in the same file, the target of "out=". It should only be used in conjunction with "ecco" which does not actually merge the reads. With "ecco" and "mix" you will get the same number of output reads as input reads. Without "ecco" and "mix" your merged reads will go to "out=" and unmerged reads will go to "outu=", and the total number of reads afterward will be less than the number of input reads.