Anyone Had Experience Of Long Distance Rsync?

[Cromarty]

Ok, please say if this sounds stupid!

I plan to set my Multimedia PC (with MCE2005) up at my parents house connected to broadband. I have pcAnwhere setup so I can remotely look after scheduling and deleting etc and I planned on using Orb to view it's contents. trouble is they only have 256k upload and it looks ropey.

Soooooo, I thought what about taking a snapshop of the orginal machine and building a MCE in Bkk, then have an Rsnyc service to keep the machines the same. Ok there would be some lag of a few days, but I could live with that.

There is a way of running Rsync on Windows with the cwRsync packaging, I just wondered if anyone had first hand experience of running it in another country.

[Gumballl]

Ok, please say if this sounds stupid!

I plan to set my Multimedia PC (with MCE2005) up at my parents house connected to broadband. I have pcAnwhere setup so I can remotely look after scheduling and deleting etc and I planned on using Orb to view it's contents. trouble is they only have 256k upload and it looks ropey.

Soooooo, I thought what about taking a snapshop of the orginal machine and building a MCE in Bkk, then have an Rsnyc service to keep the machines the same. Ok there would be some lag of a few days, but I could live with that.

There is a way of running Rsync on Windows with the cwRsync packaging, I just wondered if anyone had first hand experience of running it in another country.

MCE = Media Center Edition = Microsoft XP

Why is this post in the Linux forum?

[autonomous_unit]

Rsync is a very good protocol for file transfer over wide area networks! It does a good job of pipelining files, so it gets much better throughput than naive FTP or "scp" protocols which require a round-trip handshake for each file and do not support pipelining. So, it tolerates the international latency much better.

You need to consider security. I run it conventionally over SSH connections to my own remote Linux servers. There are rsync servers I used to access mirrors of public software, e.g. Fedora Linux distribution updates, and these do not use SSH. I do not know whether raw rsync services have any authentication and encryption modes, but I personally would not risk exposing my own server with anything but SSH-based access on the public Internet.

Rsync also has a "partial transfer" mode where it can recover from failures without retransfering the whole file it was working on. There is a limit to this, in that there is a lot of per-file size, timestamp, and/or checksum data that get transferred at the beginning of a session. If your connection is too unreliable, your retstarts will never get out of this startup phase and will never move your real data. The bigger the collection of files, the longer this startup phase lasts.

I successfully synchronize about 6 GB of mirrored data over DTAC GPRS that involves 10-20 thousand files, and maybe 50-100 MB of new data in a typical week. Sometimes it takes me more than a week to catch up, running it overnight when I do not need the GPRS link for anything else. Is that positive enough feedback?

It works well for handling a large static set of files where a few new files appear periodically. It works very well for handling small changes to the contents of existing files. It also supports removal of files that disappear, but you need to be careful or you might have it delete things you didn't mean to delete... use backups at the destination before playing with this feature!

[Cromarty]

autonomous_unit, many thanks for that, that was just the kind of feedback I was hoping for. I'm going to plough ahead and set it up, maybe of use to others in the future so I will post my real life findings. I have a VPS and I do plan to ssh to there from BKK, so my side will be ok, the MCE to host however will be insecure. I will have to see if there is anything I can do.

MCE = Media Center Edition = Microsoft XP

Why is this post in the Linux forum?

Gumballl,

seeing as rsync is primarily a linux piece of software I was asking in the appropriate forum. The more obvious question would be to you, that is, if you haven't got anything constructive to say then why don't you keep your comments to yourself?

[Gumballl]

autonomous_unit, many thanks for that, that was just the kind of feedback I was hoping for. I'm going to plough ahead and set it up, maybe of use to others in the future so I will post my real life findings. I have a VPS and I do plan to ssh to there from BKK, so my side will be ok, the MCE to host however will be insecure. I will have to see if there is anything I can do.

MCE = Media Center Edition = Microsoft XP

Why is this post in the Linux forum?

Gumballl,

seeing as rsync is primarily a linux piece of software I was asking in the appropriate forum. The more obvious question would be to you, that is, if you haven't got anything constructive to say then why don't you keep your comments to yourself?

sensitive little pr*ck aren't you? if you already have the answer to your question, then why plague us with your post? Or do you think that 1s and 0s behave differently overseas than in your home country?

[rcm]

autonomous_unit, many thanks for that, that was just the kind of feedback I was hoping for. I'm going to plough ahead and set it up, maybe of use to others in the future so I will post my real life findings. I have a VPS and I do plan to ssh to there from BKK, so my side will be ok, the MCE to host however will be insecure. I will have to see if there is anything I can do.

MCE = Media Center Edition = Microsoft XP

Why is this post in the Linux forum?

Gumballl,

seeing as rsync is primarily a linux piece of software I was asking in the appropriate forum. The more obvious question would be to you, that is, if you haven't got anything constructive to say then why don't you keep your comments to yourself?

Cromarty,

rsync is open source software...just because it's open source doesn't mean it's linux...and since you are using this open source software in conjunction with MCE i too think it is in the wrong forum....

very friendly post , by the way

rcm

[Cromarty]

sensitive little pr*ck aren't you? if you already have the answer to your question, then why plague us with your post? Or do you think that 1s and 0s behave differently overseas than in your home country?

I love keyboard heroes!! But I won't take it personally, looking at some of your other posts you are obviously the village idiot of TV. Every forum has one, so I'm pleased to meet you.

rcm, I asked here as rsync is a linux programme, open source or not. In fact the open source dosen't matter. It doesn't matter that I'm using it with MCE, I wanted to know if anyone was using it in Thailand to sync with a PC in another Country. I explained the use to show the type of files and size.

autonomous_unit gave me a considered and excellent reply and didn't seem to be precious about the guise of rsync used.

[Lannig]

Rsync normally works great even on slow WAN links, but in my experience the Windows port isn't as reliable as the

native Unix version. I (also) used it to sync two Windows servers as well, one is on a colocation facility. It works, but some syncs mysteriously fail, actually as soon as my Internet connection gets heavily loaded.

Also, being a Unix app, it has no knowledge of the Windows file system protections, and I have all kind of trouble with newly created files or folders. It doesn't carry the ACLs of course.

I eventually gave up and now use shared folders inside a VPN connection + a freeware folder sync program instead of rsync.

Last note: rsync over ssh is slow. Better use the native (unencrypted) client-server mode. The security exposure is very minimal if you configure it properly (filtering on IP of client etc.). Unless you object having the data going across in cleatext of course. I'm not sure rsync over ssh is possible at all on Windows anyway (maybe with Cygwin).

--Lannig

[autonomous_unit]

Even 4-5 years ago, it was possible to get 12 MB/s (96 Mb/s) throughput over SSH between two workstation-class Pentium III or Xeon PCs, e.g. enough to saturate a 100baseT network interface in full-duplex mode. My current Pentium-M laptop has a much faster processor than those did.

Ever since I left behind my multi-gigabit employers, I have never seen SSH be the bottleneck in a WAN transfer.

edit: I have to add that my Linux manpage for rsync has --acls and --xattrs options to preserve ACLs and extended attributes. I don't know if these are available in a modern Windows port...

Edited July 5, 2006 by autonomous_unit

[Lannig]

Even 4-5 years ago, it was possible to get 12 MB/s (96 Mb/s) throughput over SSH between two workstation-class Pentium III or Xeon PCs, e.g. enough to saturate a 100baseT network interface in full-duplex mode. My current Pentium-M laptop has a much faster processor than those did.

Ever since I left behind my multi-gigabit employers, I have never seen SSH be the bottleneck in a WAN transfer.

Well, don't know, but I certainly have seen a huge difference between native and ssh transport modes when doing daily backups of a Linux-based mail server (~150Gb of data) using rsnapshot which in turn uses rsync. My daily backup times were almost cut by half.

One box is a dual 2.0G Xeon, the other one a 2.6G P4. Ah well, they are on a gigabit Ethernet segment I must tell.

It probably won't make any difference on a WAN though, I do agree.

Other reasons why I like the native transport much better:

- you don't have to allow root logins through ssh

- you can sort out the rsync traffic from the ssh remote admin traffic, and filter them independantly on your f/w

edit: I have to add that my Linux manpage for rsync has --acls and --xattrs options to preserve ACLs and extended attributes. I don't know if these are available in a modern Windows port...

Interesting... I'll look into this.

--Lannig