Large (10-100 G) File Transfers To Us

[retiree]

I have the following problem: I create fairly large filesets here in Thailand (primarily sound, and hi-res scans of books). I need to get them to a server in the US on a regular basis.

Uploading from home (in Bangkok) is too slow, of course. I assume that two possible solutions are:

1. periodically walk my hard drive down to someplace with a high-bandwidth connection (but where?), or

2. use a local ISP for a kind of 'store and forward' service (but who?). This assumes that my ADSL line is fast enough for local uploads, and that their link to the backbone is fast enough for the international leg.

Has anybody out there dealt with a similar problem and come up with a workable solution?

Thanks in advance,

Retiree

[autonomous_unit]

What is "a regular basis"? Uploading at a steady 512 KB/s would take about 54 hours to transfer 10GB. I have found that I can usually get close to this speed with my 1024/512 ADSL from BKK to the USA. Unless you can get a different kind of local connection, your upload is probably limited to 512 as well, so no intermediary will help get the data from your home. You would have to hand-carry the data to someplace with a real symmetric network link. I would assume that most "consumer" places w/ commercial network access would have similar asymmetric networks because they are catering to users who mostly download...

However, if you can leave a system on constantly, uploading large files isn't that difficult... I run regular transfers using the "rsync" utility on Linux, which can easily restart where it leaves off and can even be scripted to automatically keep retrying until it completes. You just need to figure whether the rate is feasible to sustain, e.g. that you generate less than 10 GB per week or so (allowing for some downtime).

If you really have large data, but the response-time isn't super critical, you might want to seriously consider just burning a set of DVDs, or writing tapes even, and sending them by FedEx or similar? That way, the cost is relatively low, even if a package gets damaged periodically.

BTW, if the filesets are not all completely new every week, you can get further benefits from something like "rsync" which can effectively transfer only the differences, if the previous set of data is still available on the destination server... I've done things before like carry a set of data on the plane and then use rsync just to "fix up" the differences that had occurred since my last trip. It works amazingly well if your data doesn't change dramatically every time. (It would not work well for things that are already compressed or encrypted, since a small change will lead to many changes all over the encoded version.)

[h90]

What is "a regular basis"? Uploading at a steady 512 KB/s would take about 54 hours to transfer 10GB. I have found that I can usually get close to this speed with my 1024/512 ADSL from BKK to the USA. Unless you can get a different kind of local connection, your upload is probably limited to 512 as well, so no intermediary will help get the data from your home. You would have to hand-carry the data to someplace with a real symmetric network link. I would assume that most "consumer" places w/ commercial network access would have similar asymmetric networks because they are catering to users who mostly download...

However, if you can leave a system on constantly, uploading large files isn't that difficult... I run regular transfers using the "rsync" utility on Linux, which can easily restart where it leaves off and can even be scripted to automatically keep retrying until it completes. You just need to figure whether the rate is feasible to sustain, e.g. that you generate less than 10 GB per week or so (allowing for some downtime).

If you really have large data, but the response-time isn't super critical, you might want to seriously consider just burning a set of DVDs, or writing tapes even, and sending them by FedEx or similar? That way, the cost is relatively low, even if a package gets damaged periodically.

BTW, if the filesets are not all completely new every week, you can get further benefits from something like "rsync" which can effectively transfer only the differences, if the previous set of data is still available on the destination server... I've done things before like carry a set of data on the plane and then use rsync just to "fix up" the differences that had occurred since my last trip. It works amazingly well if your data doesn't change dramatically every time. (It would not work well for things that are already compressed or encrypted, since a small change will lead to many changes all over the encoded version.)

But at 100 G=540 hours and double or tripple it for thailand if the connection is not good.....

Beside that I am not sure what the normal services do with you if you have that much traffic....

[friend2]

For your sake I hope the new "Blue Ray" technology gets avaliable soon.

1 DVD disk=40-50 gig.

That way all you have to do is send 2 disks via the mail...