Hi all,
I'm setting up an RNA-seq pipeline and currently working with GEO series [GSE210020], which points to SRA accessions like SRR20701*. I'm interested in optimizing the download and storage workflow, ideally accessing the paired-end FASTQ files directly from S3 (e.g., s3://sra-pub-run-odp/... or mirrored ENA buckets).
I’ve already scripted downloads via wget from ENA FTP and tested conversion via fasterq-dump, but:
💡 Main questions:
Appreciate any pointers, workflows, or alternative tools I might be missing!
(Eventually this will be part of a scalable RNA-seq-to-DESeq2 pipeline.)
Thanks!
I'm setting up an RNA-seq pipeline and currently working with GEO series [GSE210020], which points to SRA accessions like SRR20701*. I'm interested in optimizing the download and storage workflow, ideally accessing the paired-end FASTQ files directly from S3 (e.g., s3://sra-pub-run-odp/... or mirrored ENA buckets).
I’ve already scripted downloads via wget from ENA FTP and tested conversion via fasterq-dump, but:
- fasterq-dump is quite slow for large files.
- FTP works, but I'd prefer using S3 for speed and robustness.
- I'm uploading resulting FASTQs to our own S3 bucket for reuse, but would like to skip re-downloads if possible.
💡 Main questions:
- Is there a reliable way to access FASTQ files directly from public S3 buckets (e.g., from ENA or NCBI)?
- Are there any tools/scripts you recommend that natively use S3 as a source or destination?
- Are there common pitfalls with region mismatches or index paths (e.g., odd directory layout under sra-pub-run-odp)?
Appreciate any pointers, workflows, or alternative tools I might be missing!
(Eventually this will be part of a scalable RNA-seq-to-DESeq2 pipeline.)
Thanks!