help with basemount copy script

[thermophile]

I have a script that I've cobbled together to copy fastq from each sample within a project into one folder on my computer.

        for f in ../basemountpoint/basespace/Projects/PROJECTNAME/Samples/*/Files/*.gz;
        do cp $f PROJECTNAME"/fastq/"${f##*Files/};
        done

This works, except for the occasions that I have to resequence a particular sample. Only the original sample is copied because basemount put " (2)" on the sample folder name for the second run. How can I tweak this so it will copy the fastq for the second run (the fastq are unique because they all get the run info as part of the name). I think I need to change this so the second part of the cp command ignores the " (2)" but can't figure out how. My first thought was to remove the * but that made it fail for all samples, not just the duplicates.

Rereading the bash documentation isn't helping http://tldp.org/LDP/abs/html/string-manipulation.html

[neavemj]

Hi thermophile,

Can you give us the complete name of the old and new fastq files?

The first line in that script is where the variable 'f' is assigned to each of the file names, so it might be at this point where the files with (2) are being missed.

Cheers,

Matt.

[Bukowski]

Use rsync?

[thermophile]

I don't want to just rsync because I need all the fastq in a single folder for downstream processing

Here I've echo'd the cp line and added a comma for readability

[CODE]for f in ../basemountpoint/basespace/Projects/PROJECTNAME/Samples/*/Files/*.gz;
do echo $f "," "/PROJECTNAME"/fastq/"${f##*Files/};
done
[CODE]

../basemountpoint/basespace/Projects/PROJECTNAME/Samples/ADB2017Dec13SI1 (2),/Files/ADB2017Dec13SI1_S150_L001_R1_001.fastq.gz PROJECTNAME/fastq/ADB2017Dec13SI1_S150_L001_R1_001.fastq.gz
../basemountpoint/basespace/Projects/PROJECTNAME/Samples/ADB2017Dec13SI1 (2),/Files/ADB2017Dec13SI1_S150_L001_R2_001.fastq.gz PROJECTNAME/fastq/ADB2017Dec13SI1_S150_L001_R2_001.fastq.gz
../basemountpoint/basespace/Projects/PROJECTNAME/Samples/ADB2017Dec13SI1,/Files/ADB2017Dec13SI1_S70_L001_R1_001.fastq.gz PROJECTNAME/fastq/ADB2017Dec13SI1_S70_L001_R1_001.fastq.gz
../basemountpoint/basespace/Projects/PROJECTNAME/Samples/ADB2017Dec13SI1,/Files/ADB2017Dec13SI1_S70_L001_R2_001.fastq.gz PROJECTNAME/fastq/ADB2017Dec13SI1_S70_L001_R2_001.fastq.gz

[neavemj]

Hi thermophile,

I'd say the problem with the new file names are the extra space and the parentheses. The extra space makes it difficult for the 'cp' command to know what is the file to copy and what is the destination. Also, parentheses need to be 'escaped' if you want to use them in a filename. Something like the below (note the extra backslashes):

ls ADB2017Dec13SI1\ \(2\)/

This makes the whole thing pretty complicated, but I think if you put some quotes around the file names it will treat them as a whole rather than their parts:

for f in ../basemountpoint/basespace/Projects/PROJECTNAME/Samples/*/Files/*.gz;
        do cp "$f" PROJECTNAME"/fastq/"${f##*Files/};
        done

The only thing I changed was to put the $f in quotes. I'm not entirely sure if this will work without actually trying it out. There are some other quotes in the 'destination' bit and some other things going on that might mess it up.

Let me know if it works!

Cheers,

Matt.

[neavemj]

P.S. I guess the best idea would be to change your workflow so that spaces and parentheses are not introduced into the file names. If you use some linux programs for trimming or other processing, these will probably also fail with these file names.

Good luck!

Matt.

[fmd]

For anyone that comes across this thread looking for help with BaseMount, I've made a Python script that might be useful to you. It doesn't exactly do what thermophile asked for, but should make retrieving the fastq files straightforward. Given a BaseMount project directory, it will extract all of the runs and simulate the folder structure you'd expect from a local MiSeq run for each. In addition to the reads, it will grab the sample sheet, InterOp directory contents, and log files.

Here it is:

https://github.com/bfssi-forest-dussault/BaseMountRetrieve