Announcement

Collapse
No announcement yet.

Append two fastq files repeatedly

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Append two fastq files repeatedly

    Hello, I am on an Ubuntu 12 system.
    I am trying to write a bash loop to run
    Code:
    cat /data/rad1/ang_TP30124.fastq  /data/rad2/ang_TP30124.fastq >  /data/rad3/ang_TP30124.fastq
    for 184 files.

    I have 184 files in two directories with identical file names (/data/rad1 and /data/rad2). I want to append each of the like named fastq files into a single like named file into a third directory /data/rad3.

    I am just learning and have come up with:
    Code:
    topdir=/data/rad3
    dir1=/data/rad1
    dir2=/data/rad1
    
    for f in $topdir/$dir1/*.fastq
    do
        outf=$topdir/`basename $f .fastq`
        cp $f $outf
        cat $topdir/$dir2/`basename $f` >> $outf
    done
    which is not working. Any advice for a beginner?

  • #2
    Here are a couple of things I noticed.

    1. dir2 should be set to /data/rad2
    2. instead of $topdir/$dir1, I think it should just be $dir1 (likewise for $dir2)
    3. basename includes the fastq extension, so no need to append .fastq

    This would be my attempt at it:

    Code:
    topdir=/data/rad3
    dir1=/data/rad1
    dir2=/data/rad2
    
    for f in $dir1/*.fastq
    do
        outf=$topdir/`basename $f`
        echo $outf
        cp $f $outf
        cat $dir2/`basename $f` >> $outf
    done
    I threw in an echo command, just to illustrate the usefulness of printing out your variables when debugging to see if the variable gets set to what you think it should be.

    Also, it might be more straightforward to do "cat a b > c" rather than "cp a c; cat b >> c;"

    So maybe the body could be replaced with

    Code:
    outf=$topdir/`basename $f`
    cat $f $dir2/`basename $f` > $outf
    Hope that helps.
    Justin

    Comment


    • #3
      Thank you Justin, that worked perfectly As you suggested, I didn't need the 'cp' command.

      Comment


      • #4
        time to learn xargs?

        here is a one-liner that uses xargs

        Code:
        basename -a data/rad1/*.dat | xargs -t -I {} bash -c 'cat data/rad1/"{}" data/rad2/"{}"  > data/rad3/"{}" ' \;
        notes:
        • this works on my mac/OSX; if you're on linux the options for xargs might be different
        • also, you might need to upgrade your GNU coreutils to get the version of basename that supports -a
        • xargs also supports -P for doing multiple cats in parallel if you have multiple processors this could speed things up....
        • the double quotes are protection again odd characters in your filenames, if any

        Comment


        • #5
          Thanks malcook. I will play with this, especially the xargs -P option as I have 32 cores to play with.

          Comment


          • #6
            Hi malcook, I tried to run this xargs one-liner, but I keep getting a basename error - "basename: invalid option -- 'a'"
            I have the latest Ubuntu coreutils (8.13-3ubuntu3.1).
            I looked at the basename man page and there are no listed options, only help and version.

            Comment

            Working...
            X