Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Fetch fastqs from basespace with command line

    Dear seqanswers community,
    I'd like to know if anyone of you ever downloaded fastqs from basespace from the command line? If so, I'd really appreciate some help or point me in the right direction. Just to clarify, I don't want to use any basespace apps at all: some data of a collaborator sits in basespace and I just want to download it on my HPC cluster.

    Many thanks,
    Julio

  • #2
    See https://gist.github.com/lh3/54f535b11a9ee5d3be8e

    Comment


    • #3
      Thanks lh3. This works! Do you know if there is a way to download all .fastq.gz for a given run. I found out how to download bcl files, but would like to get fastqs instead.
      Cheers,

      Comment


      • #4
        I don't know how, but there must be ways with their APIs.

        Comment


        • #5
          Originally posted by julio514 View Post
          Dear seqanswers community,
          I'd like to know if anyone of you ever downloaded fastqs from basespace from the command line? If so, I'd really appreciate some help or point me in the right direction. Just to clarify, I don't want to use any basespace apps at all: some data of a collaborator sits in basespace and I just want to download it on my HPC cluster.

          Many thanks,
          Julio
          I've used BaseSpaceR to get fastq files via R. (This was while ago).

          If I correctly remember and things haven't changed, it's a bit long winded to get started as you need to get a token. Also it's not enough to have a run shared with you, the owner of the project has to share the entire project (I think...). Then something on these lines should work:

          Code:
          library(BaseSpaceR)
          ACCESS_TOKEN<- 'dd9...mytoken...43'
          PROJECT_ID<- '123456'  ## Get proj ID from url of the project
          
          aAuth<- AppAuth(access_token = ACCESS_TOKEN)
          selProj <- Projects(aAuth, id = PROJECT_ID, simplify = TRUE) 
          sampl <- listSamples(selProj, limit= 1000)
          inSample <- Samples(aAuth, id = Id(sampl), simplify = TRUE)
          for(s in inSample){ 
              f <- listFiles(s, Extensions = ".gz")
              print(Name(f))
              getFiles(aAuth, id= Id(f), destDir = 'outdir/', verbose = TRUE)
          }

          Comment


          • #6
            Thanks dariober, works like a charm!

            Comment


            • #7
              A while ago I wrote an interactive BaseSpace command-line client for downloading data to headless servers. You might wish to try it as an alternative to the other options here. It has an FTP-like interface that allows you to browse the data in your account based on the Project/Sample/File hierarchy currently used. You can download individual files or entire projects at a time. As with the other alternatives, you need to obtain a developer's access token to use with it, but it can store this token to disk using symmetric encryption to make future use a bit easier. It's meant for interactive use, so if you want batch or scripting capabilities use one of the other suggestions.

              I just got around today to uploading it to SourceForge:

              http://sourceforge.net/projects/bsfetch/

              It is written in Perl and I use it on Linux. I've also tested it briefly on Windows, where it seems to work with the exception of password masking on the command line.
              Last edited by jdv; 12-06-2014, 12:19 PM.

              Comment


              • #8
                Also, a quick note for anyone using lh3's method to manually construct URLs - not only does this open up your access_token for viewing in the process table by anyone else on the same machine (e.g. with top, ps, etc) as already mentioned, but it is also transmitted in cleartext with the HTTPS request URL and can be trivially captured by anyone happening to be watching the network traffic.

                The secure way of sending the request is with the access token specified in the 'x-access-token' HTTP header, which is encrypted in the SSL connection. See here:



                A modification of lh3's method using curl instead of wget would be like this:

                Code:
                curl -L -J --config token_header.txt https://api.basespace.illumina.com/v1pre3/files/YOUR-FILE-ID/content -O
                where the arbitrarily named 'token_header.txt' contains something like this:

                Code:
                header = "x-access-token: YOUR-TOKEN-HERE"
                This prevents snooping via the process table or network traffic. Obviously you won't want to leave the 'token_header.txt' file sitting around on shared disk space.

                Comment


                • #9
                  Nice tips. Thanks!

                  Comment


                  • #10
                    Originally posted by julio514 View Post
                    Thanks lh3. This works! Do you know if there is a way to download all .fastq.gz for a given run. I found out how to download bcl files, but would like to get fastqs instead.
                    Cheers,
                    It might be too late, but in the first link lh provided in his gist, there is a python script to allow you to download all files in one run by specifying the run ID.


                    I also have a small python script to allow you download all fastq files in one project by specifying project name and access Token.

                    Comment


                    • #11
                      Also these python scripts that work well (tested)

                      Cheers,

                      Comment


                      • #12
                        Too late to answer OP, but in case anyone else needs an alternative, the BaseSpace command line interface was really straightforward to install and use on Ubuntu. To download fastq files by project:

                        Code:
                        bs download project --id <project_id> -o <target_directory>

                        Comment

                        Latest Articles

                        Collapse

                        • seqadmin
                          Strategies for Sequencing Challenging Samples
                          by seqadmin


                          Despite advancements in sequencing platforms and related sample preparation technologies, certain sample types continue to present significant challenges that can compromise sequencing results. Pedro Echave, Senior Manager of the Global Business Segment at Revvity, explained that the success of a sequencing experiment ultimately depends on the amount and integrity of the nucleic acid template (RNA or DNA) obtained from a sample. “The better the quality of the nucleic acid isolated...
                          03-22-2024, 06:39 AM
                        • seqadmin
                          Techniques and Challenges in Conservation Genomics
                          by seqadmin



                          The field of conservation genomics centers on applying genomics technologies in support of conservation efforts and the preservation of biodiversity. This article features interviews with two researchers who showcase their innovative work and highlight the current state and future of conservation genomics.

                          Avian Conservation
                          Matthew DeSaix, a recent doctoral graduate from Kristen Ruegg’s lab at The University of Colorado, shared that most of his research...
                          03-08-2024, 10:41 AM

                        ad_right_rmr

                        Collapse

                        News

                        Collapse

                        Topics Statistics Last Post
                        Started by seqadmin, Yesterday, 06:37 PM
                        0 responses
                        8 views
                        0 likes
                        Last Post seqadmin  
                        Started by seqadmin, Yesterday, 06:07 PM
                        0 responses
                        8 views
                        0 likes
                        Last Post seqadmin  
                        Started by seqadmin, 03-22-2024, 10:03 AM
                        0 responses
                        49 views
                        0 likes
                        Last Post seqadmin  
                        Started by seqadmin, 03-21-2024, 07:32 AM
                        0 responses
                        67 views
                        0 likes
                        Last Post seqadmin  
                        Working...
                        X