Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • GenoMax
    replied
    Have you verified that the $i variable is expanding correctly?

    If you have multiple files to align you should first create an index by doing

    Code:
    bbmap.sh ref=/users/chutfilz/data/chutfilz/Dm3_Index/dm3.fa
    this will create a "ref" directory with all index files. Don't worry about what is in the directory (bbmap uses it own organization).

    Then when you run use
    Code:
    path=dir_containing_ref_dir
    in command line, instead of "ref=". This will avoid re-indexing the genome each time.

    If you are using a job scheduler then you should submit each alignment job separately (not the way you have the loop setup, which I assume is submitted as a single job?)

    Leave a comment:


  • chutfilz
    replied
    Negative Array Size in BBMap

    Hello,

    I'm trying to map 16 paired sequences. I've already trimmed the adapters using trimgalore!

    Input (scheduled job run on a server):

    Code:
    for i in "${samples[@]}"; do
    bbmap.sh ref=/users/chutfilz/data/chutfilz/Dm3_Index/dm3.fa in="$i"_R1_001_val_1.fa.gz in2="$i"_R2_001_val_2.fa.gz out="$i".sam
    done
    Requested 16 cpus, 80g RAM, 24h runtime.

    Output:

    Code:
    module: loading 'java/8u111'
    module: loading 'bbmap/38.23'
    module: loading 'samtools/1.9'
    java -ea -Xmx47593m -cp /gpfs/runtime/opt/bbmap/38.23/bin/current/ align2.BBMap build=1 overwrite=true fastareadlen=500 ref=/users/chutfilz/data/chutfilz/Dm3_Index/dm3.fa in=PoolCH-1_R1_001_val_1.fa.gz in2=PoolCH-1_R2_001_val_2.fa.gz out=PoolCH-1.sam
    Executing align2.BBMap [build=1, overwrite=true, fastareadlen=500, ref=/users/chutfilz/data/chutfilz/Dm3_Index/dm3.fa, in=PoolCH-1_R1_001_val_1.fa.gz, in2=PoolCH-1_R2_001_val_2.fa.gz, out=PoolCH-1.sam]
    Version 38.24
    
    Retaining first best site only for ambiguous mappings.
    Writing reference.
    Executing dna.FastaToChromArrays2 [/users/chutfilz/data/chutfilz/Dm3_Index/dm3.fa, 1, writeinthread=false, genscaffoldinfo=true, retain, waitforwriting=false, gz=true, maxlen=536670912, writechroms=true, minscaf=1, midpad=300, startpad=8000, stoppad=8000, nodisk=false]
    
    Set genScaffoldInfo=true
    Writing chunk 1
    Set genome to 1
    
    Loaded Reference:	0.057 seconds.
    Loading index for chunk 1-1, build 1
    No index available; generating from reference genome: /gpfs/data/mtatar/chutfilz/trimgalore/ref/index/1/chr1_index_k13_c4_b1.block
    Indexing threads started for block 0-1
    Indexing threads finished for block 0-1
    Generated Index:	15.892 seconds.
    Analyzed Index:   	2.851 seconds.
    Started output stream:	1.332 seconds.
    Cleared Memory:    	0.264 seconds.
    Processing reads in paired-ended mode.
    Started read stream.
    Started 16 mapping threads.
    [B]Exception in thread "Thread-28" java.lang.NegativeArraySizeException[/B]
    	at java.util.Arrays.copyOf(Arrays.java:3236)
    	at shared.KillSwitch.copyOf(KillSwitch.java:294)
    	at stream.FastaReadInputStream.fillBuffer(FastaReadInputStream.java:447)
    	at stream.FastaReadInputStream.nextHeader(FastaReadInputStream.java:290)
    	at stream.FastaReadInputStream.fillList(FastaReadInputStream.java:174)
    	at stream.FastaReadInputStream.hasMore(FastaReadInputStream.java:107)
    	at stream.ConcurrentGenericReadInputStream$ReadThread.readLists(ConcurrentGenericReadInputStream.java:664)
    	at stream.ConcurrentGenericReadInputStream$ReadThread.run(ConcurrentGenericReadInputStream.java:653)
    [B]Exception in thread "Thread-27" java.lang.NegativeArraySizeException[/B]
    	at java.util.Arrays.copyOf(Arrays.java:3236)
    	at shared.KillSwitch.copyOf(KillSwitch.java:294)
    	at stream.FastaReadInputStream.fillBuffer(FastaReadInputStream.java:447)
    	at stream.FastaReadInputStream.nextHeader(FastaReadInputStream.java:290)
    	at stream.FastaReadInputStream.fillList(FastaReadInputStream.java:174)
    	at stream.FastaReadInputStream.hasMore(FastaReadInputStream.java:107)
    	at stream.ConcurrentGenericReadInputStream$ReadThread.readLists(ConcurrentGenericReadInputStream.java:664)
    	at stream.ConcurrentGenericReadInputStream$ReadThread.run(ConcurrentGenericReadInputStream.java:653)
    PoolCH-1.sam has begun to write, but it's been five hours (these are Drosophila reads), and no progress on the other 15 paired fasta files.

    Where is the negative array???

    Leave a comment:


  • catagui
    replied
    bbmap error in output file

    Hi I am getting an error when running bbmap. Any ideas what I am doing wrong?
    Also how do I check the mapping stats?

    This is what I am running:

    Code:
    #!/bin/bash
    cd /space/home/aguilar/Ofav_temp/Trim
    /space/home/aguilar/Programs/bbmap/bbwrap.sh t=40 in=\
    S1_F_paired_1.fq,S10_F_paired_1.fq,S11_F_paired_1.fq,S12_F_paired_1.fq,S13_F_paired_1.fq,S14_F_paired_1.fq,S15_F_paired_1.fq,S16_F_paired_1.fq,S17_F_paired_1.fq,S18_F_paired_1.fq,S19_F_paired_1.fq,S2_F_paired_1.fq,S20_F_paired_1.fq,S21_F_paired_1.fq,S22_F_paired_1.fq,S23_F_paired_1.fq,S24_F_paired_1.fq,S25_F_paired_1.fq,S26_F_paired_1.fq,S27_F_paired_1.fq,S28_F_paired_1.fq,S29_F_paired_1.fq,S3_F_paired_1.fq,S30_F_paired_1.fq,S31_F_paired_1.fq,S32_F_paired_1.fq,S33_F_paired_1.fq,S34_F_paired_1.fq,S35_F_paired_1.fq,S36_F_paired_1.fq,S37_F_paired_1.fq,S38_F_paired_1.fq,S39_F_paired_1.fq,S4_F_paired_1.fq,S40_F_paired_1.fq,S41_F_paired_1.fq,S42_F_paired_1.fq,S43_F_paired_1.fq,S44_F_paired_1.fq,S45_F_paired_1.fq,S46_F_paired_1.fq,S47_F_paired_1.fq,S48_F_paired_1.fq,S5_F_paired_1.fq,S6_F_paired_1.fq,S7_F_paired_1.fq,S8_F_paired_1.fq,S9_F_paired_1.fq \
    in2=S1_R_paired_2.fq,S10_R_paired_2.fq,S11_R_paired_2.fq,S12_R_paired_2.fq,S13_R_paired_2.fq,S14_R_paired_2.fq,S15_R_paired_2.fq,S16_R_paired_2.fq,S17_R_paired_2.fq,S18_R_paired_2.fq,S19_R_paired_2.fq,S2_R_paired_2.fq,S20_R_paired_2.fq,S21_R_paired_2.fq,S22_R_paired_2.fq,S23_R_paired_2.fq,S24_R_paired_2.fq,S25_R_paired_2.fq,S26_R_paired_2.fq,S27_R_paired_2.fq,S28_R_paired_2.fq,S29_R_paired_2.fq,S3_R_paired_2.fq,S30_R_paired_2.fq,S31_R_paired_2.fq,S32_R_paired_2.fq,S33_R_paired_2.fq,S34_R_paired_2.fq,S35_R_paired_2.fq,S36_R_paired_2.fq,S37_R_paired_2.fq,S38_R_paired_2.fq,S39_R_paired_2.fq,S4_R_paired_2.fq,S40_R_paired_2.fq,S41_R_paired_2.fq,S42_R_paired_2.fq,S43_R_paired_2.fq,S44_R_paired_2.fq,S45_R_paired_2.fq,S46_R_paired_2.fq,S47_R_paired_2.fq,S48_R_paired_2.fq,S5_R_paired_2.fq,S6_R_paired_2.fq,S7_R_paired_2.fq,S8_R_paired_2.fq,S9_R_paired_2.fq \
    ref=/space/home/aguilar/Ofav_temp/Genomes/Orbicella_faveolata_v2_scaffolds.fa \
    outu=/space/home/aguilar/Ofav_temp/bbmap/ReadsUnm.R1.fastq.gz \
    outu2=/space/home/aguilar/Ofav_temp/bbmap/ReadsUnmR2.fastq.gz \
    outm=/space/home/aguilar/Ofav_temp/bbmap/ReadsMappedR1.fastq.gz \
    outm2=/space/home/aguilar/Ofav_temp/bbmap/ReadsMappedR2.fastq.gz \
    This is the error:

    Code:
    Retaining first best site only for ambiguous mappings.
    No output file.
    Exception in thread "main" java.lang.AssertionError: ASCII encoding for quality (currently ASCII-33) appears to be wrong.
    Thanks

    Leave a comment:


  • catagui
    replied
    problem with output

    Hi I am having troubles when running bbwrap. Also how can I get a file that tells me that perfectage that was mapped and unmapped.

    This is what I am running:

    #!/bin/bash
    cd /space/home/aguilar/Ofav_temp/Trim
    /space/home/aguilar/Programs/bbmap/bbwrap.sh t=40 in=\
    S1_F_paired_1.fq,S10_F_paired_1.fq,S11_F_paired_1.fq,S12_F_paired_1.fq,S13_F_paired_1.fq,S14_F_paired_1.fq,S15_F_paired_1.fq,S16_F_paired_1.fq,S17_F_paired_1.fq,S18_F_paired_1.fq,S19_F_paired_1.fq,S2_F_paired_1.fq,S20_F_paired_1.fq,S21_F_paired_1.fq,S22_F_paired_1.fq,S23_F_paired_1.fq,S24_F_paired_1.fq,S25_F_paired_1.fq,S26_F_paired_1.fq,S27_F_paired_1.fq,S28_F_paired_1.fq,S29_F_paired_1.fq,S3_F_paired_1.fq,S30_F_paired_1.fq,S31_F_paired_1.fq,S32_F_paired_1.fq,S33_F_paired_1.fq,S34_F_paired_1.fq,S35_F_paired_1.fq,S36_F_paired_1.fq,S37_F_paired_1.fq,S38_F_paired_1.fq,S39_F_paired_1.fq,S4_F_paired_1.fq,S40_F_paired_1.fq,S41_F_paired_1.fq,S42_F_paired_1.fq,S43_F_paired_1.fq,S44_F_paired_1.fq,S45_F_paired_1.fq,S46_F_paired_1.fq,S47_F_paired_1.fq,S48_F_paired_1.fq,S5_F_paired_1.fq,S6_F_paired_1.fq,S7_F_paired_1.fq,S8_F_paired_1.fq,S9_F_paired_1.fq \
    in2=S1_R_paired_2.fq,S10_R_paired_2.fq,S11_R_paired_2.fq,S12_R_paired_2.fq,S13_R_paired_2.fq,S14_R_paired_2.fq,S15_R_paired_2.fq,S16_R_paired_2.fq,S17_R_paired_2.fq,S18_R_paired_2.fq,S19_R_paired_2.fq,S2_R_paired_2.fq,S20_R_paired_2.fq,S21_R_paired_2.fq,S22_R_paired_2.fq,S23_R_paired_2.fq,S24_R_paired_2.fq,S25_R_paired_2.fq,S26_R_paired_2.fq,S27_R_paired_2.fq,S28_R_paired_2.fq,S29_R_paired_2.fq,S3_R_paired_2.fq,S30_R_paired_2.fq,S31_R_paired_2.fq,S32_R_paired_2.fq,S33_R_paired_2.fq,S34_R_paired_2.fq,S35_R_paired_2.fq,S36_R_paired_2.fq,S37_R_paired_2.fq,S38_R_paired_2.fq,S39_R_paired_2.fq,S4_R_paired_2.fq,S40_R_paired_2.fq,S41_R_paired_2.fq,S42_R_paired_2.fq,S43_R_paired_2.fq,S44_R_paired_2.fq,S45_R_paired_2.fq,S46_R_paired_2.fq,S47_R_paired_2.fq,S48_R_paired_2.fq,S5_R_paired_2.fq,S6_R_paired_2.fq,S7_R_paired_2.fq,S8_R_paired_2.fq,S9_R_paired_2.fq \
    ref=/space/home/aguilar/Ofav_temp/Genomes/Orbicella_faveolata_v2_scaffolds.fa \
    outu=/space/home/aguilar/Ofav_temp/bbmap/ReadsUnm.R1.fastq.gz \
    outu2=/space/home/aguilar/Ofav_temp/bbmap/ReadsUnmR2.fastq.gz \
    outm=/space/home/aguilar/Ofav_temp/bbmap/ReadsMappedR1.fastq.gz \
    outm2=/space/home/aguilar/Ofav_temp/bbmap/ReadsMappedR2.fastq.gz \

    And I am getting this message:

    Retaining first best site only for ambiguous mappings.
    No output file.
    Exception in thread "main" java.lang.AssertionError: ASCII encoding for quality (currently ASCII-33) appears to be wrong.
    +��[ԽMo3͒,6��+.�7��l�.���®�w�.��}m��2"�����F���#Q�

    Thanks

    Leave a comment:


  • zeam
    replied
    Hi Brain, could you please answer my questions posted here at your convenience? http://seqanswers.com/forums/showthread.php?t=85967

    Thanks in advance.

    Leave a comment:


  • 1989sn1027
    replied
    Originally posted by GenoMax View Post
    @1989sn1027: Brian has not been participating on SA for last few months. You could try to create a ticket at Source Forge and see if he responds to this report.
    Thanks for your directing. I'll give that a shot.

    Leave a comment:


  • GenoMax
    replied
    @1989sn1027: Brian has not been participating on SA for last few months. You could try to create a ticket at Source Forge and see if he responds to this report.

    Leave a comment:


  • 1989sn1027
    replied
    Originally posted by GenoMax View Post
    I see that you are assigning 60G of RAM. Have you tried to assign more and see if it helps?
    Thanks for your replying. In my test I've tried adding java RAM upper limit from 20G all the way to 400G. Yet still the "maxlen" arguments of dna.FastaToChromArrays2 hadn't changed, neighter the error message.

    Leave a comment:


  • Sarah Muller
    replied
    That's great new. I will download it asap . Thanks for sharing!

    Leave a comment:


  • GenoMax
    replied
    I see that you are assigning 60G of RAM. Have you tried to assign more and see if it helps?

    Leave a comment:


  • 1989sn1027
    replied
    Hi Brian & all,
    I'm using BBmap 38.26 with a very big reference genome, and some chromosome in this genome is big enough to break the bbmap ref building session.

    Here is the fasta index of this reference:
    Chr01 301019445 7 60 61
    Chr02 163962470 306036450 60 61
    Chr03 261511374 472731635 60 61
    Chr04 215701946 738601539 60 61
    Chr05 217274494 957898525 60 61
    Chr06 219521584 1178794268 60 61
    Chr07 222112641 1401974553 60 61
    Chr08 153299543 1627789079 60 61
    Chr09 238794889 1783643622 60 61
    Chr10 205736368 2026418433 60 61
    Chr11 220335243 2235583748 60 61
    Chr12 229934170 2459591253 60 61
    Chr00 714758103 2693357667 60 61
    Can see that the longest chromosome is beyond 536670912, which cause a problem like this:
    bbmap-38.26/bbmap.sh ref=ref.fasta rebuild=t usemodulo=t -Xmx60g
    java -ea -Xmx60g -cp /home/sn/software/bbmap-38.26/current/ align2.BBMap build=1 overwrite=true fastareadlen=500 ref=/home/yangjy/16T4/Genome/GEN181516HEB/_db/Capsicum.annuum.L_Zunla-1_Release_2.0.fasta rebuild=t usemodulo=t -Xmx60g
    Executing align2.BBMap [build=1, overwrite=true, fastareadlen=500, ref=/home/yangjy/16T4/Genome/GEN181516HEB/_db/Capsicum.annuum.L_Zunla-1_Release_2.0.fasta, rebuild=t, usemodulo=t, -Xmx60g]
    Version 38.26

    No output file.
    Writing reference.
    Executing dna.FastaToChromArrays2 [/home/yangjy/16T4/Genome/GEN181516HEB/_db/Capsicum.annuum.L_Zunla-1_Release_2.0.fasta, 1, writeinthread=false, genscaffoldinfo=true, retain, waitforwriting=false, gz=true, maxlen=536670912, writechroms=true, minscaf=1, midpad=300, startpad=8000, stoppad=8000, nodisk=false]

    Set genScaffoldInfo=true
    Writing chunk 1
    Writing chunk 2
    Writing chunk 3
    Writing chunk 4
    Writing chunk 5
    Writing chunk 6
    Exception in thread "main" java.lang.AssertionError: 714758103, 8000, 7999, 536670912
    at dna.FastaToChromArrays2.makeNextChrom(FastaToChromArrays2.java:440)
    at dna.FastaToChromArrays2.makeChroms(FastaToChromArrays2.java:343)
    at dna.FastaToChromArrays2.main2(FastaToChromArrays2.java:151)
    at align2.RefToIndex.makeIndex(RefToIndex.java:147)
    at align2.BBMap.setup(BBMap.java:278)
    at align2.AbstractMapper.<init>(AbstractMapper.java:57)
    at align2.BBMap.<init>(BBMap.java:43)
    at align2.BBMap.main(BBMap.java:31)
    I'm pretty sure it's the 'maxlen' argument of dna.FastaToChromArrays2 that is not fit my situation, but I'm not sure how can I fix this.

    Did anyone deal with this kinda things before? Any suggestion and discussion is of help! >_<

    Leave a comment:


  • csmiller
    replied
    usejni and compiled C code in BBTools

    I just installed the latest version of the BBTools (38.26), and I notice that the C code provided by the usejni=t flag for some tools has been depreciated / disabled.

    I found this in the changelog:
    Removed JNI path flag from BBMerge, BBMap, and RQCFilter shell scripts.
    and this in docs/compiling.txt:
    3) C code. This was developed by Jonathan Rood to accelerate BBMap, BBMerge, and Dedupe, but is currently disabled.
    Sure enough, it is commented out in the bbmap.sh code:
    Code:
            #local CMD="java -Djava.library.path=$NATIVELIBDIR $EA $z -cp $CP align2.BBMap build=1 overwrite=true fastareadlen=500 $@"
            local CMD="java $EA $z -cp $CP align2.BBMap build=1 overwrite=true fastareadlen=500 $@"
    If I revert to the previous version of the CMD, with the java.library.path set, then the command runs with the compiled C code just fine.

    Why was this disabled? Does this affect previous analyses that used this C code? That is, does the C code contain an error that means usejni=t in previous versions will produce different output than the java-only code? Or was this purely a performance or compatibility issue, or something else?

    Sorry if I've missed this already posted somewhere, and thanks in advance for any help.

    Chris

    Leave a comment:


  • csmiller
    replied
    usejni and compiled C code in BBTools

    I just installed the latest version of the BBTools (38.26), and I notice that the C code provided by the usejni=t flag for some tools has been depreciated / disabled.

    I found this in the changelog:
    Removed JNI path flag from BBMerge, BBMap, and RQCFilter shell scripts.
    and this in docs/compiling.txt:
    3) C code. This was developed by Jonathan Rood to accelerate BBMap, BBMerge, and Dedupe, but is currently disabled.
    Sure enough, it is commented out in the bbmap.sh code:
    Code:
            #local CMD="java -Djava.library.path=$NATIVELIBDIR $EA $z -cp $CP align2.BBMap build=1 overwrite=true fastareadlen=500 $@"
            local CMD="java $EA $z -cp $CP align2.BBMap build=1 overwrite=true fastareadlen=500 $@"
    If I revert to the previous version of the CMD, with the java.library.path set, then the command runs with the compiled C code just fine.

    Why was this disabled? Does this affect previous analyses that used this C code? That is, does the C code contain an error that means usejni=t in previous versions will produce different output than the java-only code? Or was this purely a performance or compatibility issue, or something else?

    Sorry if I've missed this already posted somewhere, and thanks in advance for any help.

    Chris

    Leave a comment:


  • csmiller
    replied
    How to use usejni with latest version (

    I just installed the latest version of the BBTools (38.26), and I can't seem to get the usejni flag to work. The java .so file compiles fine, but then I get error messages like this when I run, e.g., bbmap.sh with usejni=t:
    Code:
    Native library can not be found in java.library.path.
    I found this in the changelog:
    Removed JNI path flag from BBMerge, BBMap, and RQCFilter shell scripts.
    and this in docs/compiling.txt:
    3) C code. This was developed by Jonathan Rood to accelerate BBMap, BBMerge, and Dedupe, but is currently disabled.
    And sure enough, it is commented out in the bbmap.sh code:
    Code:
            #local CMD="java -Djava.library.path=$NATIVELIBDIR $EA $z -cp $CP align2.BBMap build=1 overwrite=true fastareadlen=500 $@"
            local CMD="java $EA $z -cp $CP align2.BBMap build=1 overwrite=true fastareadlen=500 $@"
    If I revert to the previous version of the CMD, with the java.library.path set, then the command runs with the C code fine.

    Why was this disabled? Does this affect previous analyses that used this C code? Or was this purely a performance issue?

    Sorry if I've missed this posted somewhere else, and thanks in advance for any help.

    Chris

    Leave a comment:


  • SNPsaurus
    replied
    Originally posted by juanita View Post
    I am sorry if this question is very basic but I am getting a low percentage of mapping reads to the reference genome, about the 36% of the pct reads mapped. Any clue what this is the case?

    I am using as the reference genome the genome in scaffolds and paired-end reads...
    Have you trimmed adapters away from the reads (short fragments will create reads that are part genomic and part adapter and may not map). You could use the related BBmap tool sendsketch to get a sense of what is in your reads (after trimming). When we do genotyping of samples, many samples have contaminating species...so using sendsketch can help figure out what is in there. You can input the entire fastq file with sendsketch, or go to read mose and get a result on a per read basis.

    You can also grab 100 reads, turn them into fasta format and do blastn with them (if online use the blastn rather than megablast option) and see read by read what is in there.

    Other options...your sample is not highly related to the reference, the reference may be incomplete and missing regions, the reference is lacking high copy repeat content like mtDNA or chloroplast and many reads go to those.

    Leave a comment:

Latest Articles

Collapse

  • seqadmin
    Strategies for Sequencing Challenging Samples
    by seqadmin


    Despite advancements in sequencing platforms and related sample preparation technologies, certain sample types continue to present significant challenges that can compromise sequencing results. Pedro Echave, Senior Manager of the Global Business Segment at Revvity, explained that the success of a sequencing experiment ultimately depends on the amount and integrity of the nucleic acid template (RNA or DNA) obtained from a sample. “The better the quality of the nucleic acid isolated...
    03-22-2024, 06:39 AM
  • seqadmin
    Techniques and Challenges in Conservation Genomics
    by seqadmin



    The field of conservation genomics centers on applying genomics technologies in support of conservation efforts and the preservation of biodiversity. This article features interviews with two researchers who showcase their innovative work and highlight the current state and future of conservation genomics.

    Avian Conservation
    Matthew DeSaix, a recent doctoral graduate from Kristen Ruegg’s lab at The University of Colorado, shared that most of his research...
    03-08-2024, 10:41 AM

ad_right_rmr

Collapse

News

Collapse

Topics Statistics Last Post
Started by seqadmin, Yesterday, 06:37 PM
0 responses
10 views
0 likes
Last Post seqadmin  
Started by seqadmin, Yesterday, 06:07 PM
0 responses
9 views
0 likes
Last Post seqadmin  
Started by seqadmin, 03-22-2024, 10:03 AM
0 responses
49 views
0 likes
Last Post seqadmin  
Started by seqadmin, 03-21-2024, 07:32 AM
0 responses
67 views
0 likes
Last Post seqadmin  
Working...
X