Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • #61
    testformat says: illumina fastq raw single-ended 108bp

    As far as I remember this was a HiSeq run.

    I tried the reformat line suggested by Brian but the process stops after a while with errors. Apparently short of memory. Will try to improve that and try again.
    Hi there

    Comment


    • #62
      Hmm... can you post the errors? Reformat by default uses very little memory, which is all that should be needed for a correctly-formatted file containing reads. It will run out of memory if you use it without increasing the default memory allocation on extremely long sequences (over tens of megabases) such as the human genome. It will never run out of memory on a correctly-formatted Illumina fastq file.

      So, it would also be helpful if you could post the results of "head" (the first 10 lines of the file).

      Comment


      • #63
        See below. I run it for only one fo the files because doing both would go over my storage quota.

        java -da -Xmx200m -cp /home/cslamovi/CLARKSCV1.2.2-b/bbmap/current/ jgi.ReformatReads -da ibq qin=33 in=scratch/s_3_1_sequence.fastq out=scratch/fixed_1.fq
        Executing jgi.ReformatReads [-da, ibq, qin=33, in=scratch/s_3_1_sequence.fastq, out=scratch/fixed_1.fq]

        Input is being processed as unpaired
        java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:2786)
        at fileIO.ByteFile1.fillBuffer(ByteFile1.java:180)
        at fileIO.ByteFile1.nextLine(ByteFile1.java:136)
        at stream.FASTQ.toReadList(FASTQ.java:648)
        at stream.FastqReadInputStream.fillBuffer(FastqReadInputStream.java:111)
        at stream.FastqReadInputStream.nextList(FastqReadInputStream.java:96)
        at stream.ConcurrentGenericReadInputStream$ReadThread.readLists(ConcurrentGenericReadInputStream.java:656)
        at stream.ConcurrentGenericReadInputStream$ReadThread.run(ConcurrentGenericReadInputStream.java:635)
        Input: 32775200 reads 3539721600 bases
        Output: 32775200 reads (100.00%) 3539721600 bases (100.00%)

        Time: 807.191 seconds.
        Reads Processed: 32775k 40.60k reads/sec
        Bases Processed: 3539m 4.39m bases/sec
        Exception in thread "main" java.lang.RuntimeException: ReformatReads terminated in an error state; the output may be corrupt.
        at jgi.ReformatReads.process(ReformatReads.java:1032)
        at jgi.ReformatReads.main(ReformatReads.java:45)
        Hi there

        Comment


        • #64
          Looks like you only ran this with 200MB of RAM. Can you try with -Xmx2g?

          How old is this data BTW (in years)?

          Comment


          • #65
            Reformat should never run out of memory with the default settings and short (<200kbp) reads. I think the input file is corrupt, and should be re-downloaded. The corruption probably occurs somewhere around the 32.77 millionth read, but it's hard to be sure...

            Comment


            • #66
              Hi

              I came across this when searching for a way to demultiplex non-overlapping paired end reads that were sequenced using combinatorial barcodes. I don't suppose there is a way of doing that somehow using seal (or other tools?).

              Originally posted by Brian Bushnell View Post
              It is almost possible to do this with Seal, which outputs reads into bins based on kmer matching.

              seal.sh in=reads.fq pattern=%.fq k=6 restrictleft=6 mm=f ref=barcodes.fa rcomp=f

              That would require a file "barcodes.fa" like this:
              >AACTGA
              AACTGA
              >GGCCTT
              GGCCTT

              etc., with one fasta entry per barcode, so the output reads would be in file AACTGA.fq and so forth. This is sort of a common request, so maybe I will make it unnecessary to provide a fasta file of the barcodes. Does that matter to you either way?

              However, BBDuk has the flags "skipr1" and "skipr2", which allow it to only do kmer operations on one read or the other. Seal currently lacks this, but it's essential for processing inline barcodes. I'll add it for the next release.

              Comment


              • #67
                Newbie here! I have unzipped and untared bbmap but it wont run any commands. I have a Linux virtual box in windows 10. Am I missing some software to use BBMap?

                Comment


                • #68
                  Originally posted by mcauchy View Post
                  Newbie here! I have unzipped and untared bbmap but it wont run any commands. I have a Linux virtual box in windows 10. Am I missing some software to use BBMap?
                  What do you mean "it won't run any commands"? Can you see the shell scripts in the "bbmap" folder. Try the following command and see if it produces help output on screen after you change to bbmap directory.

                  Code:
                  $ ./bbmap.sh

                  Comment


                  • #69
                    What I mean is I run:
                    $ ./repair.sh in1=/media/sf_D_DRIVE/champagnefastqs/srr1290816_1.fastq in2=/media/sf_D_DRIVE/champagnefastqs/srr1290816_2.fastq out1=fixed1.fq out2=fixed2.fq outsingle=single.fq

                    ...and get:
                    java -ea -Xmx-211m -cp /media/sf_D_DRIVE/bbmap/current/ jgi.SplitPairsAndSingles rp in1=/media/sf_D_DRIVE/champagnefastqs/srr1290816_1.fastq in2=/media/sf_D_DRIVE/champagnefastqs/srr1290816_2.fastq out1=fixed1.fq out2=fixed2.fq outsingle=single.fq
                    Invalid maximum heap size: -Xmx-211m
                    Error: Could not create the Java Virtual Machine.
                    Error: A fatal exception has occurred. Program will exit.

                    Comment


                    • #70
                      How much memory have you allocated to the VM? You should at least have 2+ GB to have enough available for programs to run.

                      Comment


                      • #71
                        I have allocated 2.9Gb, which is all I have to give. It seems that is not enough. Thanks for your help.

                        Comment


                        • #72
                          Originally posted by mcauchy View Post
                          I have allocated 2.9Gb, which is all I have to give. It seems that is not enough. Thanks for your help.
                          That may be true but in case BBMap was not able to allocate RAM correctly can you try running the command as follows:

                          Code:
                          $ ./repair.sh -Xmx2g in1=/media/sf_D_DRIVE/champagnefastqs/srr1290816_1.fastq in2=/media/sf_D_DRIVE/champagnefastqs/srr1290816_2.fastq out1=fixed1.fq out2=fixed2.fq outsingle=single.fq

                          Comment


                          • #73
                            Also it seems to me that '-Xmx-211m' is odd. Why the negative 211? I am not sure that makes a difference but it might.

                            Comment


                            • #74
                              Didn't run for very long....

                              $ ./repair.sh -Xmx2g in1=/media/sf_D_DRIVE/champagnefastqs/srr1290816_1.fastq in2=/media/sf_D_DRIVE/champagnefastqs/srr1290816_2.fastq out1=fixed1.fq out2=fixed2.fq outsingle=single.fq

                              java -ea -Xmx2g -cp /media/sf_D_DRIVE/bbmap/current/ jgi.SplitPairsAndSingles rp -Xmx2g in1=/media/sf_D_DRIVE/champagnefastqs/srr1290816_1.fastq in2=/media/sf_D_DRIVE/champagnefastqs/srr1290816_2.fastq out1=fixed1.fq out2=fixed2.fq outsingle=single.fq
                              Executing jgi.SplitPairsAndSingles [rp, -Xmx2g, in1=/media/sf_D_DRIVE/champagnefastqs/srr1290816_1.fastq, in2=/media/sf_D_DRIVE/champagnefastqs/srr1290816_2.fastq, out1=fixed1.fq, out2=fixed2.fq, outsingle=single.fq]

                              Set INTERLEAVED to false
                              Started output stream.
                              Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
                              at java.util.HashMap.resize(HashMap.java:580)
                              at java.util.HashMap.addEntry(HashMap.java:879)
                              at java.util.LinkedHashMap.addEntry(LinkedHashMap.java:427)
                              at java.util.HashMap.put(HashMap.java:505)
                              at jgi.SplitPairsAndSingles.repair(SplitPairsAndSingles.java:751)
                              at jgi.SplitPairsAndSingles.process3_repair(SplitPairsAndSingles.java:538)
                              at jgi.SplitPairsAndSingles.process2(SplitPairsAndSingles.java:304)
                              at jgi.SplitPairsAndSingles.process(SplitPairsAndSingles.java:230)
                              at jgi.SplitPairsAndSingles.main(SplitPairsAndSingles.java:45)

                              Comment


                              • #75
                                How about setting the VM aside and running BBMap directly on windows 10. How much RAM is there on the machine? BBMap is written in java and will run there but you would need to take into account windows versions of the command line usage for BBMap.

                                Comment

                                Latest Articles

                                Collapse

                                • seqadmin
                                  Choosing Between NGS and qPCR
                                  by seqadmin



                                  Next-generation sequencing (NGS) and quantitative polymerase chain reaction (qPCR) are essential techniques for investigating the genome, transcriptome, and epigenome. In many cases, choosing the appropriate technique is straightforward, but in others, it can be more challenging to determine the most effective option. A simple distinction is that smaller, more focused projects are typically better suited for qPCR, while larger, more complex datasets benefit from NGS. However,...
                                  10-18-2024, 07:11 AM
                                • seqadmin
                                  Non-Coding RNA Research and Technologies
                                  by seqadmin




                                  Non-coding RNAs (ncRNAs) do not code for proteins but play important roles in numerous cellular processes including gene silencing, developmental pathways, and more. There are numerous types including microRNA (miRNA), long ncRNA (lncRNA), circular RNA (circRNA), and more. In this article, we discuss innovative ncRNA research and explore recent technological advancements that improve the study of ncRNAs.

                                  Nobel Prize for MicroRNA Discovery
                                  This week,...
                                  10-07-2024, 08:07 AM

                                ad_right_rmr

                                Collapse

                                News

                                Collapse

                                Topics Statistics Last Post
                                Started by seqadmin, Yesterday, 05:31 AM
                                0 responses
                                10 views
                                0 likes
                                Last Post seqadmin  
                                Started by seqadmin, 10-24-2024, 06:58 AM
                                0 responses
                                20 views
                                0 likes
                                Last Post seqadmin  
                                Started by seqadmin, 10-23-2024, 08:43 AM
                                0 responses
                                48 views
                                0 likes
                                Last Post seqadmin  
                                Started by seqadmin, 10-17-2024, 07:29 AM
                                0 responses
                                58 views
                                0 likes
                                Last Post seqadmin  
                                Working...
                                X