Announcement

Collapse
No announcement yet.

Fetch fastqs from basespace with command line

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Fetch fastqs from basespace with command line

    Dear seqanswers community,
    I'd like to know if anyone of you ever downloaded fastqs from basespace from the command line? If so, I'd really appreciate some help or point me in the right direction. Just to clarify, I don't want to use any basespace apps at all: some data of a collaborator sits in basespace and I just want to download it on my HPC cluster.

    Many thanks,
    Julio

  • #2
    See https://gist.github.com/lh3/54f535b11a9ee5d3be8e

    Comment


    • #3
      Thanks lh3. This works! Do you know if there is a way to download all .fastq.gz for a given run. I found out how to download bcl files, but would like to get fastqs instead.
      Cheers,

      Comment


      • #4
        I don't know how, but there must be ways with their APIs.

        Comment


        • #5
          Originally posted by julio514 View Post
          Dear seqanswers community,
          I'd like to know if anyone of you ever downloaded fastqs from basespace from the command line? If so, I'd really appreciate some help or point me in the right direction. Just to clarify, I don't want to use any basespace apps at all: some data of a collaborator sits in basespace and I just want to download it on my HPC cluster.

          Many thanks,
          Julio
          I've used BaseSpaceR to get fastq files via R. (This was while ago).

          If I correctly remember and things haven't changed, it's a bit long winded to get started as you need to get a token. Also it's not enough to have a run shared with you, the owner of the project has to share the entire project (I think...). Then something on these lines should work:

          Code:
          library(BaseSpaceR)
          ACCESS_TOKEN<- 'dd9...mytoken...43'
          PROJECT_ID<- '123456'  ## Get proj ID from url of the project
          
          aAuth<- AppAuth(access_token = ACCESS_TOKEN)
          selProj <- Projects(aAuth, id = PROJECT_ID, simplify = TRUE) 
          sampl <- listSamples(selProj, limit= 1000)
          inSample <- Samples(aAuth, id = Id(sampl), simplify = TRUE)
          for(s in inSample){ 
              f <- listFiles(s, Extensions = ".gz")
              print(Name(f))
              getFiles(aAuth, id= Id(f), destDir = 'outdir/', verbose = TRUE)
          }

          Comment


          • #6
            Thanks dariober, works like a charm!

            Comment


            • #7
              A while ago I wrote an interactive BaseSpace command-line client for downloading data to headless servers. You might wish to try it as an alternative to the other options here. It has an FTP-like interface that allows you to browse the data in your account based on the Project/Sample/File hierarchy currently used. You can download individual files or entire projects at a time. As with the other alternatives, you need to obtain a developer's access token to use with it, but it can store this token to disk using symmetric encryption to make future use a bit easier. It's meant for interactive use, so if you want batch or scripting capabilities use one of the other suggestions.

              I just got around today to uploading it to SourceForge:

              http://sourceforge.net/projects/bsfetch/

              It is written in Perl and I use it on Linux. I've also tested it briefly on Windows, where it seems to work with the exception of password masking on the command line.
              Last edited by jdv; 12-06-2014, 12:19 PM.

              Comment


              • #8
                Also, a quick note for anyone using lh3's method to manually construct URLs - not only does this open up your access_token for viewing in the process table by anyone else on the same machine (e.g. with top, ps, etc) as already mentioned, but it is also transmitted in cleartext with the HTTPS request URL and can be trivially captured by anyone happening to be watching the network traffic.

                The secure way of sending the request is with the access token specified in the 'x-access-token' HTTP header, which is encrypted in the SSL connection. See here:

                https://developer.basespace.illumina...e_Access_Token

                A modification of lh3's method using curl instead of wget would be like this:

                Code:
                curl -L -J --config token_header.txt https://api.basespace.illumina.com/v1pre3/files/YOUR-FILE-ID/content -O
                where the arbitrarily named 'token_header.txt' contains something like this:

                Code:
                header = "x-access-token: YOUR-TOKEN-HERE"
                This prevents snooping via the process table or network traffic. Obviously you won't want to leave the 'token_header.txt' file sitting around on shared disk space.

                Comment


                • #9
                  Nice tips. Thanks!

                  Comment


                  • #10
                    Originally posted by julio514 View Post
                    Thanks lh3. This works! Do you know if there is a way to download all .fastq.gz for a given run. I found out how to download bcl files, but would like to get fastqs instead.
                    Cheers,
                    It might be too late, but in the first link lh provided in his gist, there is a python script to allow you to download all files in one run by specifying the run ID.
                    https://support.basespace.illumina.c...run-downloader

                    I also have a small python script to allow you download all fastq files in one project by specifying project name and access Token.
                    https://github.com/yu68/tools/tree/m...aseSpace-tools

                    Comment


                    • #11
                      Also these python scripts that work well (tested)
                      https://github.com/nh13/basespace-invaders
                      Cheers,

                      Comment


                      • #12
                        Too late to answer OP, but in case anyone else needs an alternative, the BaseSpace command line interface was really straightforward to install and use on Ubuntu. To download fastq files by project:

                        Code:
                        bs download project --id <project_id> -o <target_directory>

                        Comment

                        Working...
                        X