What's your transatlantic bandwidth like?

[bobmaccallum]

Hi,

The other day I was downloading (to imperial.ac.uk) a VM from a partner .edu FTP server and was getting around 800KB/s

Today I am FTPing from the Broad Institute and it's around 300KB/s

13 hours for a 14GB file seems a bit much!

For comparison I'm getting 10MiB/s between Imperial and EBI via FTP.

Has nobody paid for a bigger pipe across the pond?

cheers,
Bob.

[GenoMax]

Either institutions where the traffic is originating from and terminating could be throttling the flow. Would also likely depend on if your institution is on internet2 (Broad is).

Just this one cable alone is supposed to provide 320 gbps bandwidth and that certainly can't be the only connectivity between UK and US.

Edit: Tested FTP with a chromosome file from Ensembl: (4.22 MB/s) - `Homo_sapiens.GRCh37.72.dna.chromosome.12.fa.gz' saved [39532274]

[rhinoceros]

This can also depend on the time of the day. For example, all things NCBI tend to get slower as the USA wakes up..

[bobmaccallum]

thanks GenoMax - maybe there is some throttling - I can imagine the Broad server is fairly busy, I am logged in as anonymous*, priority bandwidth probably goes to "real" users?

(*although I am collaborating directly with someone at Broad - I'm just using the credentials I was given)

rhinoceros: yes that was the case the other week - it definitely slowed down as the US woke up, but this Broad xfer has been pretty rock solid at around 275 KB/s

[GenoMax]

One thing that is easy to overlook is the "last mile" problem. What kind of link does the server have to the net where you are downloading the files to.

As an anonymous user at broad this is what I see from FTP: (11.0 MB/s) - `contigs.bases.gz' saved [765664848] from a server that has a 10G fiber connection.

Broad may be throttling anonymous FTP traffic (at around that limit) since I am seeing about 8 MB/s to the desktop connected via 1G ethernet for the same file.

[bobmaccallum]

I just had a reply from our ICT people and they suspect that it's a problem with lost packets and an over-worked firewall causing a bottleneck in an out of Imperial College which disproportionately affects high-latency connections (e.g. to the US). It will be fixed later this year they tell me.

They suggested a parallel approach with 'lftp'. I'll give that a go, even though I guess that puts even more pressure on the last mile here at Imperial!

Thanks for your tests and input - I was right to question the slowness!