1000genomes Accessibility

[giror]

Hi all,
I am interested in a particular genomic region that I would like to retrieve from all 1103 mapped genomes that are available today. I used this line in a python script to extract my region:

os.system('samtools view -b ftp://ftp-trace.ncbi.nih.gov/1000genomes/ftp/{0} 2:69,130,000-69,160,000 > {1}'.format(sample, fn))

It works about 20% of the time, but other times I get this kind of responses:
[kftp_connect_file] 331 Anonymous login ok, send your complete email address as your password.
[main_samview] fail to open "ftp://ftp-trace.ncbi.nih.gov/1000genomes/ftp/data/HG00611/alignment/HG00611.mapped.ILLUMINA.bwa.CHS.low_coverage.20101123.bam" for reading.

OR
[kftp_connect_file] 227 Entering Passive Mode (130,14,250,10,195,247).
[main_samview] fail to open "ftp://ftp-trace.ncbi.nih.gov/1000genomes/ftp/data/HG00628/alignment/HG00628.mapped.ILLUMINA.bwa.CHS.low_coverage.20101123.bam" for reading.

OR
[kftp_connect_file] 331 Anonymous login ok, send your complete email address as your password.
[bam_header_read] EOF marker is absent. The input is probably truncated.
[bam_header_read] invalid BAM binary header (this is not a BAM file).
[main_samview] fail to read the header from "ftp://ftp-trace.ncbi.nih.gov/1000genomes/ftp/data/HG00614/alignment/HG00614.mapped.ILLUMINA.bwa.CHS.low_coverage.20101123.bam".

Has anyone previously encountered these types of errors? Are there any easy fixes?

Thanks in advance for any help!
-giror

p.s. I have also tried the new 1000genomes slicer tool for some of these bam files only to get a 404 error.

[Jim Robinson]

I had similiar errors with the ncbi ftp servers, they don't seem to like getting multiple "REST" commands in quick succession. In my case I was doing this from java and inserted a delay after closing the stream for one query, and sending the "REST" command for the next. Unless you want to try patching samtools to do the same you might try forcing a ~3 second delay between queries for successive regions.

You might also want to post this question on the samtools-help email list.

Jim

[giror]

Thanks a lot! That worked perfectly.

[laura]

As a note for the data slicer unfortunately due to the vagueries of the EBI firewall it only works for ftp sites within the EBI but it will work for http anywhere

thanks