Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Append two fastq files repeatedly

    Hello, I am on an Ubuntu 12 system.
    I am trying to write a bash loop to run
    Code:
    cat /data/rad1/ang_TP30124.fastq  /data/rad2/ang_TP30124.fastq >  /data/rad3/ang_TP30124.fastq
    for 184 files.

    I have 184 files in two directories with identical file names (/data/rad1 and /data/rad2). I want to append each of the like named fastq files into a single like named file into a third directory /data/rad3.

    I am just learning and have come up with:
    Code:
    topdir=/data/rad3
    dir1=/data/rad1
    dir2=/data/rad1
    
    for f in $topdir/$dir1/*.fastq
    do
        outf=$topdir/`basename $f .fastq`
        cp $f $outf
        cat $topdir/$dir2/`basename $f` >> $outf
    done
    which is not working. Any advice for a beginner?

  • #2
    Here are a couple of things I noticed.

    1. dir2 should be set to /data/rad2
    2. instead of $topdir/$dir1, I think it should just be $dir1 (likewise for $dir2)
    3. basename includes the fastq extension, so no need to append .fastq

    This would be my attempt at it:

    Code:
    topdir=/data/rad3
    dir1=/data/rad1
    dir2=/data/rad2
    
    for f in $dir1/*.fastq
    do
        outf=$topdir/`basename $f`
        echo $outf
        cp $f $outf
        cat $dir2/`basename $f` >> $outf
    done
    I threw in an echo command, just to illustrate the usefulness of printing out your variables when debugging to see if the variable gets set to what you think it should be.

    Also, it might be more straightforward to do "cat a b > c" rather than "cp a c; cat b >> c;"

    So maybe the body could be replaced with

    Code:
    outf=$topdir/`basename $f`
    cat $f $dir2/`basename $f` > $outf
    Hope that helps.
    Justin

    Comment


    • #3
      Thank you Justin, that worked perfectly As you suggested, I didn't need the 'cp' command.

      Comment


      • #4
        time to learn xargs?

        here is a one-liner that uses xargs

        Code:
        basename -a data/rad1/*.dat | xargs -t -I {} bash -c 'cat data/rad1/"{}" data/rad2/"{}"  > data/rad3/"{}" ' \;
        notes:
        • this works on my mac/OSX; if you're on linux the options for xargs might be different
        • also, you might need to upgrade your GNU coreutils to get the version of basename that supports -a
        • xargs also supports -P for doing multiple cats in parallel if you have multiple processors this could speed things up....
        • the double quotes are protection again odd characters in your filenames, if any

        Comment


        • #5
          Thanks malcook. I will play with this, especially the xargs -P option as I have 32 cores to play with.

          Comment


          • #6
            Hi malcook, I tried to run this xargs one-liner, but I keep getting a basename error - "basename: invalid option -- 'a'"
            I have the latest Ubuntu coreutils (8.13-3ubuntu3.1).
            I looked at the basename man page and there are no listed options, only help and version.

            Comment

            Latest Articles

            Collapse

            • seqadmin
              Strategies for Sequencing Challenging Samples
              by seqadmin


              Despite advancements in sequencing platforms and related sample preparation technologies, certain sample types continue to present significant challenges that can compromise sequencing results. Pedro Echave, Senior Manager of the Global Business Segment at Revvity, explained that the success of a sequencing experiment ultimately depends on the amount and integrity of the nucleic acid template (RNA or DNA) obtained from a sample. “The better the quality of the nucleic acid isolated...
              03-22-2024, 06:39 AM
            • seqadmin
              Techniques and Challenges in Conservation Genomics
              by seqadmin



              The field of conservation genomics centers on applying genomics technologies in support of conservation efforts and the preservation of biodiversity. This article features interviews with two researchers who showcase their innovative work and highlight the current state and future of conservation genomics.

              Avian Conservation
              Matthew DeSaix, a recent doctoral graduate from Kristen Ruegg’s lab at The University of Colorado, shared that most of his research...
              03-08-2024, 10:41 AM

            ad_right_rmr

            Collapse

            News

            Collapse

            Topics Statistics Last Post
            Started by seqadmin, Yesterday, 06:37 PM
            0 responses
            10 views
            0 likes
            Last Post seqadmin  
            Started by seqadmin, Yesterday, 06:07 PM
            0 responses
            9 views
            0 likes
            Last Post seqadmin  
            Started by seqadmin, 03-22-2024, 10:03 AM
            0 responses
            49 views
            0 likes
            Last Post seqadmin  
            Started by seqadmin, 03-21-2024, 07:32 AM
            0 responses
            67 views
            0 likes
            Last Post seqadmin  
            Working...
            X