Append two fastq files repeatedly

[wbsimey]

Hello, I am on an Ubuntu 12 system.
I am trying to write a bash loop to run

cat /data/rad1/ang_TP30124.fastq  /data/rad2/ang_TP30124.fastq >  /data/rad3/ang_TP30124.fastq

for 184 files.

I have 184 files in two directories with identical file names (/data/rad1 and /data/rad2). I want to append each of the like named fastq files into a single like named file into a third directory /data/rad3.

I am just learning and have come up with:

topdir=/data/rad3
dir1=/data/rad1
dir2=/data/rad1

for f in $topdir/$dir1/*.fastq
do
    outf=$topdir/`basename $f .fastq`
    cp $f $outf
    cat $topdir/$dir2/`basename $f` >> $outf
done

which is not working. Any advice for a beginner?

[BAMseek]

Here are a couple of things I noticed.

1. dir2 should be set to /data/rad2
2. instead of $topdir/$dir1, I think it should just be $dir1 (likewise for $dir2)
3. basename includes the fastq extension, so no need to append .fastq

This would be my attempt at it:

topdir=/data/rad3
dir1=/data/rad1
dir2=/data/rad2

for f in $dir1/*.fastq
do
    outf=$topdir/`basename $f`
    echo $outf
    cp $f $outf
    cat $dir2/`basename $f` >> $outf
done

I threw in an echo command, just to illustrate the usefulness of printing out your variables when debugging to see if the variable gets set to what you think it should be.

Also, it might be more straightforward to do "cat a b > c" rather than "cp a c; cat b >> c;"

So maybe the body could be replaced with

outf=$topdir/`basename $f`
cat $f $dir2/`basename $f` > $outf

Hope that helps.
Justin

[wbsimey]

Thank you Justin, that worked perfectly As you suggested, I didn't need the 'cp' command.

[malcook]

time to learn xargs?

here is a one-liner that uses xargs

basename -a data/rad1/*.dat | xargs -t -I {} bash -c 'cat data/rad1/"{}" data/rad2/"{}"  > data/rad3/"{}" ' \;

notes:

• this works on my mac/OSX; if you're on linux the options for xargs might be different
• also, you might need to upgrade your GNU coreutils to get the version of basename that supports -a
• xargs also supports -P for doing multiple cats in parallel if you have multiple processors this could speed things up....
• the double quotes are protection again odd characters in your filenames, if any

[wbsimey]

Thanks malcook. I will play with this, especially the xargs -P option as I have 32 cores to play with.

[wbsimey]

Hi malcook, I tried to run this xargs one-liner, but I keep getting a basename error - "basename: invalid option -- 'a'"
I have the latest Ubuntu coreutils (8.13-3ubuntu3.1).
I looked at the basename man page and there are no listed options, only help and version.