Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Fetch fastqs from basespace with command line

    Dear seqanswers community,
    I'd like to know if anyone of you ever downloaded fastqs from basespace from the command line? If so, I'd really appreciate some help or point me in the right direction. Just to clarify, I don't want to use any basespace apps at all: some data of a collaborator sits in basespace and I just want to download it on my HPC cluster.

    Many thanks,
    Julio

  • #2
    See https://gist.github.com/lh3/54f535b11a9ee5d3be8e

    Comment


    • #3
      Thanks lh3. This works! Do you know if there is a way to download all .fastq.gz for a given run. I found out how to download bcl files, but would like to get fastqs instead.
      Cheers,

      Comment


      • #4
        I don't know how, but there must be ways with their APIs.

        Comment


        • #5
          Originally posted by julio514 View Post
          Dear seqanswers community,
          I'd like to know if anyone of you ever downloaded fastqs from basespace from the command line? If so, I'd really appreciate some help or point me in the right direction. Just to clarify, I don't want to use any basespace apps at all: some data of a collaborator sits in basespace and I just want to download it on my HPC cluster.

          Many thanks,
          Julio
          I've used BaseSpaceR to get fastq files via R. (This was while ago).

          If I correctly remember and things haven't changed, it's a bit long winded to get started as you need to get a token. Also it's not enough to have a run shared with you, the owner of the project has to share the entire project (I think...). Then something on these lines should work:

          Code:
          library(BaseSpaceR)
          ACCESS_TOKEN<- 'dd9...mytoken...43'
          PROJECT_ID<- '123456'  ## Get proj ID from url of the project
          
          aAuth<- AppAuth(access_token = ACCESS_TOKEN)
          selProj <- Projects(aAuth, id = PROJECT_ID, simplify = TRUE) 
          sampl <- listSamples(selProj, limit= 1000)
          inSample <- Samples(aAuth, id = Id(sampl), simplify = TRUE)
          for(s in inSample){ 
              f <- listFiles(s, Extensions = ".gz")
              print(Name(f))
              getFiles(aAuth, id= Id(f), destDir = 'outdir/', verbose = TRUE)
          }

          Comment


          • #6
            Thanks dariober, works like a charm!

            Comment


            • #7
              A while ago I wrote an interactive BaseSpace command-line client for downloading data to headless servers. You might wish to try it as an alternative to the other options here. It has an FTP-like interface that allows you to browse the data in your account based on the Project/Sample/File hierarchy currently used. You can download individual files or entire projects at a time. As with the other alternatives, you need to obtain a developer's access token to use with it, but it can store this token to disk using symmetric encryption to make future use a bit easier. It's meant for interactive use, so if you want batch or scripting capabilities use one of the other suggestions.

              I just got around today to uploading it to SourceForge:

              http://sourceforge.net/projects/bsfetch/

              It is written in Perl and I use it on Linux. I've also tested it briefly on Windows, where it seems to work with the exception of password masking on the command line.
              Last edited by jdv; 12-06-2014, 12:19 PM.

              Comment


              • #8
                Also, a quick note for anyone using lh3's method to manually construct URLs - not only does this open up your access_token for viewing in the process table by anyone else on the same machine (e.g. with top, ps, etc) as already mentioned, but it is also transmitted in cleartext with the HTTPS request URL and can be trivially captured by anyone happening to be watching the network traffic.

                The secure way of sending the request is with the access token specified in the 'x-access-token' HTTP header, which is encrypted in the SSL connection. See here:



                A modification of lh3's method using curl instead of wget would be like this:

                Code:
                curl -L -J --config token_header.txt https://api.basespace.illumina.com/v1pre3/files/YOUR-FILE-ID/content -O
                where the arbitrarily named 'token_header.txt' contains something like this:

                Code:
                header = "x-access-token: YOUR-TOKEN-HERE"
                This prevents snooping via the process table or network traffic. Obviously you won't want to leave the 'token_header.txt' file sitting around on shared disk space.

                Comment


                • #9
                  Nice tips. Thanks!

                  Comment


                  • #10
                    Originally posted by julio514 View Post
                    Thanks lh3. This works! Do you know if there is a way to download all .fastq.gz for a given run. I found out how to download bcl files, but would like to get fastqs instead.
                    Cheers,
                    It might be too late, but in the first link lh provided in his gist, there is a python script to allow you to download all files in one run by specifying the run ID.


                    I also have a small python script to allow you download all fastq files in one project by specifying project name and access Token.

                    Comment


                    • #11
                      Also these python scripts that work well (tested)

                      Cheers,

                      Comment


                      • #12
                        Too late to answer OP, but in case anyone else needs an alternative, the BaseSpace command line interface was really straightforward to install and use on Ubuntu. To download fastq files by project:

                        Code:
                        bs download project --id <project_id> -o <target_directory>

                        Comment

                        Latest Articles

                        Collapse

                        • seqadmin
                          Exploring the Dynamics of the Tumor Microenvironment
                          by seqadmin




                          The complexity of cancer is clearly demonstrated in the diverse ecosystem of the tumor microenvironment (TME). The TME is made up of numerous cell types and its development begins with the changes that happen during oncogenesis. “Genomic mutations, copy number changes, epigenetic alterations, and alternative gene expression occur to varying degrees within the affected tumor cells,” explained Andrea O’Hara, Ph.D., Strategic Technical Specialist at Azenta. “As...
                          07-08-2024, 03:19 PM
                        • seqadmin
                          Exploring Human Diversity Through Large-Scale Omics
                          by seqadmin


                          In 2003, researchers from the Human Genome Project (HGP) announced the most comprehensive genome to date1. Although the genome wasn’t fully completed until nearly 20 years later2, numerous large-scale projects, such as the International HapMap Project and 1000 Genomes Project, continued the HGP's work, capturing extensive variation and genomic diversity within humans. Recently, newer initiatives have significantly increased in scale and expanded beyond genomics, offering a more detailed...
                          06-25-2024, 06:43 AM

                        ad_right_rmr

                        Collapse

                        News

                        Collapse

                        Topics Statistics Last Post
                        Started by seqadmin, 07-16-2024, 05:49 AM
                        0 responses
                        27 views
                        0 likes
                        Last Post seqadmin  
                        Started by seqadmin, 07-15-2024, 06:53 AM
                        0 responses
                        33 views
                        0 likes
                        Last Post seqadmin  
                        Started by seqadmin, 07-10-2024, 07:30 AM
                        0 responses
                        40 views
                        0 likes
                        Last Post seqadmin  
                        Started by seqadmin, 07-03-2024, 09:45 AM
                        0 responses
                        205 views
                        0 likes
                        Last Post seqadmin  
                        Working...
                        X