Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • BBmap dedupe help

    Greetings,

    I am currently running the dedupe command and while I have not had too much trouble with it, I am having what seems to be an input reading error. Rather than putting in my files one at a time separated by commas (which works but takes a lot of time if Im running a lot of files). How can I run a directory that contains all my files? I have used ${line} but it doesn't seem to give me a correct output. It doesn't seem to read through all the files in the folder.

    Thanks!

  • #2
    What is the reason for deduplicating the data, if I may ask? What kind of data is this? Generally you should not need to dedupe the data upfront.

    I know @Brian had allowed ref= to be a directory for BBSplit but I don't know if a similar option exists for dedupe.sh.

    Comment


    • #3
      Hi... sorry, there's no such option right now. But if you want to deduplicate a bunch of files together, you can do this:

      cat *.fasta | dedupe.sh in=stdin.fasta out=deduped.fasta

      I'm also curious as to the nature of the data; are you deduplicating multiple assemblies?

      Comment


      • #4
        Thank you both for the reply. Brian I will give that a try and see if it works! I am currently running raw peatland microbial data but am currently working with just a few of the files to get some practice. I am also looking to see the differences in output between BBmap and pandaseq.

        Comment


        • #5
          Hello again, I still seem to be running into some problems. When I attempt to run my files at once, using ${line} (which is a loop correct?) my output file is only 4.1MB. When I put my fasta files in one at a time separated by commas, my dedupe file is 12MB, which is the roughly the correct size it should be. This way of doing it works well for the moment, but I am only using 4 files for practice and will be using much more in the future, so putting them in one at a time may not be a great way of doing it. @Brian I have tried the method you had requested earlier but I may be formatting my command line incorrectly, Id be happy to show you if that helps in anyway.
          Any advice is appreciated!

          Thank you!

          Comment


          • #6
            Hi James,

            Please post the exact command you used and the complete error message.

            Also, I'm not really very good at command-line one-liners, but I'm sure it's possible to combine "ls *.fasta" with sed or awk to get a comma-delimited list of files.

            Comment


            • #7
              for line in $(cat /Users/jamesseward/Desktop/Canada/MappingFiles/plate10_map2.txt);do sh /Users/jamesseward/Desktop/Canada/bbmap/dedupe.sh -Xmx1g in=/Users/jamesseward/Desktop/Canada/bbDuk/Final_Fastq/Merge/${line}_Merge.fasta out=/Users/jamesseward/Desktop/Canada/bbDuk/dedupe/Dereplicated.fasta; done

              while I am not getting an error message, this way produces an output file that is much smaller than when I use commas for my input.

              Thank you very much for the help!

              James

              Comment


              • #8
                Since you are outputting all of the files to the same destination, the output keeps getting overwritten, so the final result is just the deduplicated version of the last file.

                Note that even if you appended subsequent output to the file instead of overwriting it (with the flags "ow=f append=t"), you'd still get a different output than using all of the files at once with commas. Running dedupe on multiple files at once will deduplicate them together; you are deduplicating them independently.

                Comment

                Latest Articles

                Collapse

                • seqadmin
                  Latest Developments in Precision Medicine
                  by seqadmin



                  Technological advances have led to drastic improvements in the field of precision medicine, enabling more personalized approaches to treatment. This article explores four leading groups that are overcoming many of the challenges of genomic profiling and precision medicine through their innovative platforms and technologies.

                  Somatic Genomics
                  “We have such a tremendous amount of genetic diversity that exists within each of us, and not just between us as individuals,”...
                  05-24-2024, 01:16 PM
                • seqadmin
                  Recent Advances in Sequencing Analysis Tools
                  by seqadmin


                  The sequencing world is rapidly changing due to declining costs, enhanced accuracies, and the advent of newer, cutting-edge instruments. Equally important to these developments are improvements in sequencing analysis, a process that converts vast amounts of raw data into a comprehensible and meaningful form. This complex task requires expertise and the right analysis tools. In this article, we highlight the progress and innovation in sequencing analysis by reviewing several of the...
                  05-06-2024, 07:48 AM

                ad_right_rmr

                Collapse

                News

                Collapse

                Topics Statistics Last Post
                Started by seqadmin, Yesterday, 01:32 PM
                0 responses
                10 views
                0 likes
                Last Post seqadmin  
                Started by seqadmin, 05-24-2024, 07:15 AM
                0 responses
                199 views
                0 likes
                Last Post seqadmin  
                Started by seqadmin, 05-23-2024, 10:28 AM
                0 responses
                221 views
                0 likes
                Last Post seqadmin  
                Started by seqadmin, 05-23-2024, 07:35 AM
                0 responses
                232 views
                0 likes
                Last Post seqadmin  
                Working...
                X