[Chief_Lazy_Bison]

java.lang.AssertionError

Hello,

I'm using bbtools to preprocess some metagenomic hiseq reads prior to assembly and I've run into a little issue with clumpify. I am using the recommended 3 step error correction found in the AssemblyPipeline.sh script but the second error correction step stalls/freezes.

when I check the stderr file generated by the job I see these exceptions:

Exception in thread "Thread-1202" java.lang.AssertionError
at clump.KmerSort3$FetchThread3.fetchNext_inner(KmerSort3.java:706)
at clump.KmerSort3$FetchThread3.fetchNext(KmerSort3.java:655)
at clump.KmerSort3$FetchThread3.run(KmerSort3.java:577)
Fetched 4595507 reads: 12.948 seconds.
--
Exception in thread "Thread-1203" java.lang.AssertionError
at clump.KmerSort3$FetchThread3.fetchNext_inner(KmerSort3.java:706)
at clump.KmerSort3$FetchThread3.fetchNext(KmerSort3.java:655)
at clump.KmerSort3$FetchThread3.run(KmerSort3.java:577)

I have resubmitted the job and got the exact same exceptions the second time as well.

A little background: this job is running on a cluster with SLURM scheduling. The job requests an entire node with 40 processors and 125G of ram.

The reads are HiSeq PE 2x150 and the total size of the compressed reads is 343G.
This is the command that keeps stalling:
clumpify.sh in=ALL_temp.fq.gz out=ALL.eccc.fq.gz ecc passes=4 reorder

there are 1158 temp files generated by clumpify that occupy ~750G

Once this exception is thrown the whole job kindof just hangs.

Using version 38.43 with java 1.8.0_121

Any feedback would be greatly appreciated.

Thanks!

[Dario1984]

I have not run Clumpify but I will by following the examples which you linked to.

[GenoMax]

@Dario1984: Have you run clumpify on your data? If you have excellent libraries with tightly controlled insert sizes you will find the duplicate rate to be well controlled. Brian has some explicit use cases in his Biostars post.

[Dario1984]

The Clumpify documentation webpage doesn't mention anything about removing duplicates, which I read about in a blog. May it be expanded upon? I have NovaSeq whole genome sequencing data (NovaSeq Control Software version 1.4.0 and Real Time Analysis version 3.3.3 acquisition followed by bcl2fastq version 2.20.0.422 conversion) for human samples of about 90 times coverage, so I think it's important that I use Clumpify. I intend to map the reads with bwa and I'm not sure if it supports some reads being pairs and some being merged (its documentation is minimal), so I plan to skip the clumping, if possible.

[DCZ]

Ah, if only everything was this simple! Should've thought of that! Thanks!

[GenoMax]

@DCZ: That is easy to do. If you do not provide any "out=" argument to most BBTools they will do the operation and produce relevant statistics without writing the output.

Tip: If you ever want to pipe things then you can use "out=stdout.fq" from first tool and then "in=stdin.fq" for next tool. You get the idea.

[DCZ]

Hi Brian,

I'm really appreciating clumpify, it's fast & does exactly what it should.

I'm trying to use it for optical duplicate detection, which works great. However, I wish only to report the number of optical duplicates, without creating the deduplicated output fastq file. Is there a possibility to skip producing output? At the moment the writing output step takes the longest time in my pipeline.

Thanks in advance

[kokyriakidis]

I am having trouble using clumpify with the parameters optical + dedupe, to remove optical duplicates. e.x. clumpify.sh in=temp.fq.gz out=clumped.fq.gz dedupe optical. Clumpify without these parameters works

[GenoMax]

I am able to do something like

for i in `ls -1 *_1*.fastq | sed 's/_1.fastq//'`; do clumpify.sh -Xmx10g in1=$i\_1.fastq in2=$i\_2.fastq out1=$i\_clu_1.fastq out2=$i\_clu_2.fastq; done

and have clumpify.sh produce two files. I am not sure why you are having trouble.

[kokyriakidis]

I use the latest version of BBtools. I can't get it work

[GenoMax]

Are you using the latest version of BBMap? Have you tried to run a test with actual file names instead of shell variables?

[kokyriakidis]

It looks like out1= and out2= variables are not being correctly expanded. BBMap seems to think that your outputs are inputs (in1=./Preproccesing/ERR522065/ERR522065_1_optical.fastq.gz, in2=./Preproccesing/ERR522065/ERR522065_2_optical.fastq.gz). Are the input files in the correct directory with the right names?

Yes! all files are in the same folder! Actually neither clumpify dedupe optical, nor filterbytile work. So I have to remove them in order to complete my pipeline...

[GenoMax]

It looks like out1= and out2= variables are not being correctly expanded. BBMap seems to think that your outputs are inputs (in1=./Preproccesing/ERR522065/ERR522065_1_optical.fastq.gz, in2=./Preproccesing/ERR522065/ERR522065_2_optical.fastq.gz). Are the input files in the correct directory with the right names?

[kokyriakidis]

After clumpify command to remove duplicates using in1 in2 and out1 out2, it seems like it produces only 1 out, messing with the pipeline stream! Why does it happen?

./bbmap/clumpify.sh in1=./Preproccesing/${ERR}/${ERR}_1_1.fastq.gz in2=./Preproccesing/${ERR}/${ERR}_2_1.fastq.gz out1=./Preproccesing/${ERR}/${ERR}_1_optical.fastq.gz out2=./Preproccesing/${ERR}/${ERR}_2_optical.fastq.gz dedupe=true optical=true overwrite=true

------

Reset INTERLEAVED to false because paired input files were specified.
Set INTERLEAVED to false
Input is being processed as paired
Writing interleaved.
Made a comparator with k=31, seed=1, border=1, hashes=4
Time: 22.512 seconds.
Reads Processed: 13371k 593.99k reads/sec
Bases Processed: 1145m 50.88m bases/sec
Executing clump.KmerSort3 [in1=./Preproccesing/ERR522065/ERR522065_1_optical_clumpify_p1_temp%_10a607a7b7090ec6.fastq.gz, in2=, out=./Preproccesing/ERR522065/ERR522065_1_optical.fas
tq.gz, out2=, groups=11, ecco=f, addname=false, shortname=f, unpair=f, repair=false, namesort=false, ow=true]

------

java -Djava.library.path=/mnt/scratchdir/home/kyriakidk/KIWI/bbmap/jni/ -ea -Xmx33412m -Xms33412m -cp /mnt/scratchdir/home/kyriakidk/KIWI/bbmap/current/ jgi.BBDukF in1=./Preproccesi
ng/ERR522065/ERR522065_1_optical.fastq.gz in2=./Preproccesing/ERR522065/ERR522065_2_optical.fastq.gz
Executing jgi.BBDukF [in1=./Preproccesing/ERR522065/ERR522065_1_optical.fastq.gz, in2=./Preproccesing/ERR522065/ERR522065_2_optical.fastq.gz]
Version 38.11

No output stream specified. To write to stdout, please specify 'out=stdout.fq' or similar.
Exception in thread "main" java.lang.RuntimeException: Can't read file './Preproccesing/ERR522065/ERR522065_2_optical.fastq.gz'

[GenoMax]

Interesting point. I have always worked with data that was of uniform length. Based on what you have discovered clumpify does seem to have an underlying need/assumption that the reads are all equal length.

Two options come to mind:

1. You could trim that extra base off from end of the 251 bp reads to make them 250 bp by using bbduk.sh
2. You could try using dedupe.sh which can match subsequences.

dedupe.sh

Written by Brian Bushnell and Jonathan Rood
Last modified March 9, 2017

Description:  Accepts one or more files containing sets of sequences (reads or scaffolds).
Removes duplicate sequences, which may be specified to be exact matches, subsequences, or sequences within some percent identity.
Can also find overlapping sequences and group them into clusters.
Please read bbmap/docs/guides/DedupeGuide.txt for more information.

Usage:     dedupe.sh in=<file or stdin> out=<file or stdout>