Multi Isp (wan) Router?

[rawnnie]

Very interested project, niller74!

What I want to know is does Load Balancing also mean combining the speeds of multiple WAN connections and acheiving a higher speed? IE. If I have two 2.5Mbps/512Kbps connections from True, I'll end up with a single connection that's 5Mbps/1Mbps?

But isn't this actually called Connection Bonding? I see people are refering to that as Load Balancing, which is actually different. Load-Balancing is for redundancy, while Connection Bonding is for performance.

So my question to you, niller74, is whether what you're working on supports Connection Bonding? Or do you know of a way to do connection bonding?

[niller74]

Very interested project, niller74!

What I want to know is does Load Balancing also mean combining the speeds of multiple WAN connections and acheiving a higher speed? IE. If I have two 2.5Mbps/512Kbps connections from True, I'll end up with a single connection that's 5Mbps/1Mbps?

But isn't this actually called Connection Bonding? I see people are refering to that as Load Balancing, which is actually different. Load-Balancing is for redundancy, while Connection Bonding is for performance.

So my question to you, niller74, is whether what you're working on supports Connection Bonding? Or do you know of a way to do connection bonding?

As I wrote just above your post it is in fact load balancing. Let us get the terms right

Load Balancing (#1): Attempt to share load between multiple interfaces, typically using round-robin and thereby getting more bandwidth.

Load Bonding (#2): Similar to 'Load Ballancing'. Difference is that you need 2 or more connections from the same provider and they have to support Bonding at their end too, which most don't do on standard DSL connections.

Load Redundancy (#3): If one connection fail the other one will take over. You are not getting the benefit of extra bandwidth from the idle connection. This solution however is the only one where a connection can actually fail and still maintain established connections.

The setup I have could support all three types, but since #2 probably wont be supported by the ISPs and would not give the added uptime from 2 different ISPs, I doubt this will ever be used. Unless you are bonding 2 lease lines.

The redundancy solution (#3) is only really important if you have VERY critical real time data you have to send. No private user would use this feature. One setup where this could be interesting however is if you need payment verification, but here would would just throw in a sim card that would establish a GPRS connection if the DSL failed.

Edited July 23, 2008 by niller74

[monty]

Just to add a bit on this, stuff I learned from fiddling with PFsense:

Round robin load balancing, meaning each request will get assigned to another WAN connection: request 1 goes to ADSL 1, request 2 goes to ADSL 2 request 3 goes to ADSL 1 again and so on.

Pro's: we all know a webpage consists of many many parts, often coming from different servers. If all these parts get spread over the two connections, you will see a pretty big gain in page loading speed, especially on graphics heavy sites.

P2P and torrents, where a big amount of connections are open simultaneously will see a big gain in speed, potentially the combined speed of both your connections.

Cons:

*Security sensitive websites (banking etc) will NOT work doing this. They will see two different IP addresses accessing your banking user interface simultaneously, and their security software will block you out! In PFsense you can solve this using "sticky" connections, meaning all requests going to the same host (your bank) will get routed through the same ADSL connection. Problem solved, however you lose a part of your double speed! BAnners hosted somewhere else will still get spread around though...

*Single ftp or http downloads only get the speed of one of your connections. Can solve this using download managers spreading the download in several sections.

*Only works well with two connections of roughly the same speed. If you have a fast connection and add a slower one, you can lose speed. Example, the text in a website goes to your fast connection while the big flash file gets assigned to your slow connection. Result is slow page loading. Would have been faster if both simply were assigned to your fast connection instead of spread around! This scenario is better suited for fail over.

In short, I found to get a reliable connection results in you not getting the full benefit of the two connections. Too many sites won't work when they see connections coming in from 2 different ISP's. Actually I never managed to get torrents to work properly using round robin load balancing. Might be possible though.

Best result I got was with simple download managers downloading from example from rapidshare. Speeds of exactly the sum of my two connections were achieved very reliable. 450 kBps downloads on two 2mbit ADSL lines through PFsense.

[niller74]

*Only works well with two connections of roughly the same speed. If you have a fast connection and add a slower one, you can lose speed. Example, the text in a website goes to your fast connection while the big flash file gets assigned to your slow connection. Result is slow page loading. Would have been faster if both simply were assigned to your fast connection instead of spread around! This scenario is better suited for fail over.

This is not entirely true for my solution. It is for PfSense though, if my VERY limited experience with it still holds.

With the kind of round-robin I have integrated you can define the speed of the connections compared to each other.

Like:

CAT (2/0.5) = 100

TOT (1/0.5) = 50

This mean that double the number of connections will be established on the CAT. Currently I cannot differentiate between up and download so they will be treated the same, but it work just fine with a fast and slower connection.

Side note on sticky connections.

The VERY few sites that require sticky connections would be overcome by configuring the router to use only one connection to those specific sites.

In my time testing this I have only experienced a problem with this once in a currency trading program.

Edited July 24, 2008 by niller74

[yeti]

Side note on sticky connections.

The VERY few sites that require sticky connections would be overcome by configuring the router to use only one connection to those specific sites.

In my time testing this I have only experienced a problem with this once in a currency trading program.

I'd have configured the router to use sticky connections for all secure (https) connections.

[Naam]

this thread is fascinating and i wish i could understand the language

[yeti]

this thread is fascinating and i wish i could understand the language

I believe you can find understandable explanations for the vocabulary used here in wikipedia.

[Naam]

are you located in Thailand Yeti? if yes where?

[yeti]

Yes, in Bangkok, why?

[Naam]

Yes, in Bangkok, why?

please check PM. thanks!

[A_Traveller]

Very interesting topic I agree, however maybe I missed it but are there are suppliers for embedded boards in Bangkok?

Regards

[niller74]

Very interesting topic I agree, however maybe I missed it but are there are suppliers for embedded boards in Bangkok?

Regards

I'm sure you can find embedded systems in Bangkok. They will most likely by way overpriced, since this is a niche market.

If you happen to find anything, I am very interested in hearing about your experiences.

Would like to build an embedded thin client as a media terminal for my TV. There are a few nice products for this in Europe, but I haven't been able to find them here.

[Prasert]

As this balancing router is also doing NAT, balancing outgoing traffic over 2 or more lines works fine when it is straight-forward traffic, like from a webbrowser. A request is sent to port 80 from a tcp-highport and the webserver sends the reply from port 80 to that tcp-highport. The next request maybe sent out over the next line, which won't be a problem as these sessions are not related.

However, when setting up a ftp connection, a (control-)connection is made to port 21 on the webserver. After logging in, both hosts negotiate ports for the data connection over which the actual transfers (like directory listings) are sent.

The load-balancing router must be able to inspect the control-traffic to decide that this connection is sent out over the same line. In case of passive ftp, the client sets up the connection to the ftp server - if that connection comes from a different IP address, the ftp server will not accept the session. In case of active ftp, the data session is initiated from

[JSixpack]

So what remains to do on this project? Are you going to offer these for sale? It sounds so neat I think I must have one whether I need it or not.

[monty]

As this balancing router is also doing NAT, balancing outgoing traffic over 2 or more lines works fine when it is straight-forward traffic, like from a webbrowser. A request is sent to port 80 from a tcp-highport and the webserver sends the reply from port 80 to that tcp-highport. The next request maybe sent out over the next line, which won't be a problem as these sessions are not related.

However, when setting up a ftp connection, a (control-)connection is made to port 21 on the webserver. After logging in, both hosts negotiate ports for the data connection over which the actual transfers (like directory listings) are sent.

The load-balancing router must be able to inspect the control-traffic to decide that this connection is sent out over the same line. In case of passive ftp, the client sets up the connection to the ftp server - if that connection comes from a different IP address, the ftp server will not accept the session. In case of active ftp, the data session is initiated from

I had exactly that experience, that's one of the applications I had to set sticky connections for.

I'm guessing the same issue was the cause of me not getting proper p2p connections!

For security sensitive applications setting https to use sticky connections would be a good solution as Yeti suggests!

As said, web browsing and http downloads with a download manager gave me the biggest gains.

Now Niller somewhere stated that you either choose load balancing or fail over, one thing pf sense can do is combine the two. PF sense checks WAN connectivity by almost non stop (like every 10 seconds, user configurable) pinging a server of your choice. I just pinged the dns server of my isp because of the very low latency. Once pf sence fails to ping an ISP twice (again user configurable) it stops to do round robbin and routes everything to the active ISP.

At the moment the biggest plus of Niller's sytem is to be able to set the speed ratio of your WAN connections. It's still not entirely fool proof, but will definitely smooth things out.

Although with most idiotic Thai (hello AIS) site design you might still get the html through the fast connection and the 7 MB flash intro trough the slow one

[Prasert]

Somehow my entire post didn't get through. Here's the post again, since I can't edit my previous post anymore:

As this balancing router is also doing NAT, balancing outgoing traffic over 2 or more lines works fine when it is straight-forward traffic, like from a webbrowser. A request is sent to port 80 from a tcp-highport and the webserver sends the reply from port 80 to that tcp-highport. The next request maybe sent out over the next line, which won't be a problem as these sessions are not related.

However, when setting up a ftp connection, a (control-)connection is made to port 21 on the server. After logging in, both hosts negotiate ports for the data connection over which the actual transfers (like directory listings) are sent.

The load-balancing router must be able to inspect the control-traffic to decide that this connection is sent out over the same line. In case of passive ftp, the client sets up the connection to the ftp server - if that connection comes from a different IP address, the ftp server will not accept the session. In case of active ftp, the data session is initiated from the ftp server and the router must open the corresponding port and add a NAT translation. This is usually only possible with firewalls that are capable of stateful inspection combined with NAT.

More or less the same problem arises when using msn and some other programs that can start multiple sessions. MSN starts a connection by connecting to a central server on port 1863. Chat sessions to other msn users go over the server and will work fine, but file transfers and video sessions are usually established peer-to-peer, which may cause problems when those additional sessions are sent out over the next line and therefor have another IP address.

Anyway I'm not trying to discourage this project. It's way cheaper than using a router like Cisco and has a lot more capabilities than most SOHO routers. But when you're running into problems with specific applications, it's good to keep the above in mind.

Another remark to the balancing: I think it's not load-balancing, but session-balancing. If the router is capable of inspecting the traffic-per-session, it can determine the interface to use for the next session, based on the load of all running sessions per interface. If it's just a round-robin mechanism, only the sessions are balanced based on quantity.

Edited July 25, 2008 by Prasert

[Prasert]

I'm still curious about the other capabilities besides balancing.

Any routing protocols supported? Like OSPF/BGP?

VLAN support? Like 802.1Q?

...

Would like to build an embedded thin client as a media terminal for my TV. There are a few nice products for this in Europe, but I haven't been able to find them here.

Niller, PM me. I'm currently working on this.

[niller74]

Thanks yeti, monty and Prasert for the good feedback. I never intented to make this commercial available, I was merely getting a feel of the interest for such a project, and I think this thread has proven that the interest is there.

yeti, I think using sticky connections on all https will be default. I like the idea, thanks.

monty, the system I'm building will also do different connection tests. I have been thinking about adding a latency test, where you can tell the router to use the connection with the lowest latency to a certain service. That way you can optimize latency critical connections to always use the best connection at the time. What I have experienced from using 3 different ISPs over the past year is that it does in fact change. It is not always the same ISP that has the lowest latency to services.

Praesert, I am well aware that this router will never be able to compete with a Cisco solution on anything than price. If I really wanted to build such router this wouldn't be the place to discuss it

EDIT: At the moment the router can't do traffic accounting on a session basis, but can still spread out the sessions based on the load of the individual interfaces. I believe, I could make it do accounting per session, but I doubt the benefit of this in a small installation like this.

Edited July 25, 2008 by niller74

[niller74]

I'm still curious about the other capabilities besides balancing.

Any routing protocols supported? Like OSPF/BGP?

VLAN support? Like 802.1Q?

Currently none of this is supported.

[inventorinthailand]

We are planing to install some more ADSL from different ISP's to our office. We need higher up/down speed and also need it as a backup if one ISP fails. I have found this TP-Link TL-R488T at Fortune Tower for 11500baht. Any one who knows if it will work for us??

Edited August 6, 2008 by inventorinthailand

[niller74]

We are planing to install some more ADSL from different ISP's to our office. We need higher up/down speed and also need it as a backup if one ISP fails. I have found this TP-Link TL-R488T at Fortune Tower for 11500baht. Any one who knows if it will work for us??

I have had VERY bad experience with TP-Link products.

It seems like it could do the job though.

[inventorinthailand]

Would PePlink 30 be a better choice - but it is so expensive 29500baht.

[Prasert]

For those considering a business solution, my setup might be an option:

We have 4 Cisco routers which each have a dsl connection (on 4 different DSLAMs - buy the phone guys a beer...).

These four routers are connected by a single network. Traffic from client computers arrives on the first router (seperate interface).

This router inspects the traffic and makes routing decisions based on the type of traffic - it redirects traffic to one of the three other routers (set ip next-hop).

All un-shaped traffic (e.g. bittorrent, p2p) is sent out over it's own line. So one line maybe full with entertainment traffic, while skype calls are still working fine and webpages load reasonably fast.

All routers tell each other if they have a default route to the internet (routing protocol). In case a dsl line fails, all traffic destined to go onto the internet there, is instantly sent to the next available router (based on a pre-set priority).

Client computers might notice this: a webpage doesn't load completely but does so with F5, msn and skype quickly reconnect, and bittorrent starts rebuilding all it's peer connections.

Buying brand new equipment will indeed be expensive. eBay is a bit cheaper - Cisco 837 routers are capable of doing all the above for adsl lines, Cisco 878 routers are excellent in doing the same with sdsl lines. Software image required: IP PLUS - usually included on the router in every eBay auction. Bit cheaper but less capable: 827 adsl and 828 sdsl routers.

With a bit of patience this won't cost much more then buying new SOHO routers in Pantip.

Setting up the above is NOT possible through a GUI - you'll have to use the command line interface, which means telnet (/ssh/console). Anyone with a CCNP qualification (note: I'm not writing certification) will be able to set this up. Hire a CCNP for a day.

But again: a business solution.

Niller has created a great box to use at home!

Edited August 6, 2008 by Prasert

[niller74]

My little box has just had its OS updated, so it is now compliant with the new logging requirements. The log files are compressed and sent to a logging server for 90 days.

[george]

Would PePlink 30 be a better choice - but it is so expensive 29500baht.

I just bought a PePlink 30 via eBay for USD$459.00 (approx 15,600 THB). I expect delivery later this week. It might be some Thai customs fee added, a few thousand Baht maybe.

I will use it to balance a IPstar connection as WAN1 and a CAT-CDMA as WAN2. I will later maybe try to add an EDGE connection as WAN 3.

I will do the "drop in mode" which means no configuration is necessary on CAT router and IPstar box.

Will report when i have it up and running this weekend.

[niller74]

Would PePlink 30 be a better choice - but it is so expensive 29500baht.

I just bought a PePlink 30 via eBay for USD$459.00 (approx 15,600 THB). I expect delivery later this week. It might be some Thai customs fee added, a few thousand Baht maybe.

I will use it to balance a IPstar connection as WAN1 and a CAT-CDMA as WAN2. I will later maybe try to add an EDGE connection as WAN 3.

I will do the "drop in mode" which means no configuration is necessary on CAT router and IPstar box.

Will report when i have it up and running this weekend.

Interesting. Please keep us posted. I am considering going commercial with my little box in a couple of months. It will be cheaper and hopefully better than a PePlink product after the new improvements.

[Crushdepth]

George, how'd it go?

[JSixpack]

My little box has just had its OS updated, so it is now compliant with the new logging requirements. The log files are compressed and sent to a logging server for 90 days.

Any more updates? Is this little box for sale yet?

[Phil Conners]

I still have the kit mentioned back in Post #15 if anyone is interested

[unomi]

Interesting thread,

Was looking to get my hands on one of the nice boxen from Soekris (http://www.soekris.com/) a while ago when there was interest regarding the CCA (computer crimes act).

Now though I figure that very few people care about compliance to any great degree, no market, no product.

Anyway, its nice to know that there are people on the board that share the interests of 'making things work, better.'

In regards to load bonding on 'consumer' equipment, it shouldn't be that hard provided that you take an intermediate step of going thru a VPN/Vlan.

If you want we can talk about making that happen as I work for a guy who has a business selling VPN connections from thailand(multiple servers around the world).

The basic idea is that you take all the lines you have locally, TOT,TTT,CAT, EDGE,IPSTAR, whatever.

get them to connect to a VPN server which then handles the traffic from that point.

So you get the multiple lines connected to a single IP at the VPN side and ofcourse internally the NAT does the same thing for you.

I could write an interface that looks at ack rates and does more or less realtime load balancing between your connections to the vpn, this would enable using multiple lines for a single connection; overcome the problem of sticky connections and whatnot.