FTTC Bonding Problems

I wonder if anyone can help. Upon testing FTTC lines provided by BT I can see that although the contention is not great, overall they provide good speeds.

The problem I have is with bonding FTTC lines, the solutions never seem to be able to achieve the speeds they should and it just doesn't seem to work properly. I wonder if its anything to do with contention or BT's backhaul between the cabinet and the exchange that just doesn't allow 1 site to use up that much bandwidth? Any help is appreciated.

have you tested the bonding devices and routers whether they can manage the throughput while also bonding at those sorts of speeds.

The backhaul from a cabinet to the handover node (main fibre exchange) is at least 1 Gbps.

Test each line individually but at the same time, and if good speeds then its not contention but points toward the system doing the bonding.

Don't know why you think the contention isn't good - it's not the case.

I agree with Andrew. How are you bonding them? You need to provide more detail on your setup.

I'm using a Cisco 2911 which can handle the bonding fine, I have tested all lines individually out of the bonding and they work fine. There may be a certain way of configuring them that solves the problem but I don't know what that is, I am currently using MLPPP, do you know of a better way?

In reply to a post by helix_quazer:

I'm using a Cisco 2911 which can handle the bonding fine,

How do you know that?

I have tested all lines individually out of the bonding and they work fine.

But simultaneously as requested?

Sorry not simultaneously, what would be the benefit of that? According to the spec of the Cisco it can handle about 180mb/s of bandwidth and it has gig ports ect. I am currently attempting to run 3 40mb/s down 10mb/s up fttc lines into it and only achieving speeds of just over 1 line.

Running tests at the same time on both lines independent of the bonding would show if the issue is the backhaul.

Are the two lines with the same provider?
The spec sheet for the Cisco, which direction is the 180 Mbps, and if presenting the connection to the Cisco as PPPoE that can have an impact. We have seen other routers than can handle 700 Mbps of WAN/LAN traffic struggle with 200 Mbps of PPPoE WAN to LAN traffic.

If the issue is the Cisco it should be apparent from CPU/memory utilisation.

I will see if its possible to test the 3 lines at the same time. There are 3 lines and yes they are all with BT, as far as I can tell the quoted 180mb/s is WAN to LAN and I am running PPPoA over the lines. SNMP monitoring on the router shows the CPU is not having any problems at all

PPPoA? You sure?

The protocol between the Openreach Fibre modem and the router is usually PPPoE.

Sorry you are correct it's PPPoE

Edited by deleted (Thu 30-Jan-14 18:16:53)

So what modem are you using to connect to the VDSL2 service?

BT GEA-FTTC is usually a PPPoE presentation. VCMux PPPoA 0/38 is the old ADSL/ADSL2+ type presentation.

Okay so it really is PPPoE - in which case testing each line at the same time to three computers needs to be done, as if that works fine you are looking at an oddity with the Cisco rather than something in the Openreach or BT network

In reply to a post by helix_quazer:

I will see if its possible to test the 3 lines at the same time. There are 3 lines and yes they are all with BT, as far as I can tell the quoted 180mb/s is WAN to LAN and I am running PPPoA over the lines. SNMP monitoring on the router shows the CPU is not having any problems at all

Do you not have the VDSL modules installed in the 2911? It can, I believe support up to four and they will support profile 17a.

I have 3 VDSL modems where the lines terminate, they then plug into the Cisco via an ethernet cable. I have just plugged them into gig ports.

For testing you won't need three routers, you can create a PPPoE session on a PC just ensure you have a firewall up to protect the machine as it will be directly on the Internet

How do you mean? I'll definitely run the test if I have a way to do so without using multiple routers, I assume I will still need to take the lines out the bonding?

Yes disconnect lines from the Cisco and connect Ethernet cable from Openreach modem to the PC Ethernet port and then configure a PPPoE session.

How to do that exactly will vary according to what operating system you have.

you are looking at an oddity with the Cisco

You so know it's their set up and/or the kit.

But I have 3 VDSL modems that connect to the Cisco, I can't plug them all into 1 PC

Use 3 PC's dude.

Edited by deleted (Thu 30-Jan-14 19:14:00)

Would bonding the 3 lines actually give a cumulative speed when doing a speed test?

Would each download not just pick the line with the most available bandwidth at that moment in time therefore the download speed would just be as fast as the one single line?

Depends on the bonding method, whether it's per-packet load balancing with 3 separate WAN IP addresses, or true bonding with 1 WAN IP address.

I have load balancing technology with 3 WAN IP's, I'd be very grateful if you could explain true bonding with 1 WAN IP

In reply to a post by helix_quazer:

I have load balancing technology with 3 WAN IP's, I'd be very grateful if you could explain true bonding with 1 WAN IP

It needs ISP support, but its properly known as an "aggregate connection" where the IP traffic is split down all the connections and reassembled at your end.

Servers in data centres use similar techniques, you can have 4 ethernet cards on a server all running at 1Gigabit/sec and connected to the same switch. You then aggregate them together so that you have a logical channel running at 4 gigabit/sec.

I believe Cisco call it EtherChannel, Juniper call it Aggregate Ethernet (ae0).

I was thinking of asking if it really was bonding, or load balancing.

You might find this AAISP page helpful. Note re load balancing:-

The only downside of this method is that sometimes packets can arrive in a slight different order to which they were sent. This is normal and part of the IP protocol. However, not all systems handle out of order packets as well as they could so the more lines you have the less effective the bonding is.

On the bonding itself has the original poster checked the provider supports multilink ppp

Yes my ISP does support mlppp, thanks for the link I will have a read

Yeah sounds like its just load balanced, so you are not tripling the speed but you are giving each computer on your network an option of 3 different broadband lines when it choses to download data.

It is effectively tripling the bandwidth but will require a multi-threaded download to utilise it. Anyway, we still have to discover if the 3 streams will run at full speed independently but simultaneously.

yep time to get out the 3 pc's

Yeh I will give that a go, so if I use something like NutTCP to test the bandwidth from my end to lets say my ISP's core. If I was to max out the line by sending as much traffic down there as possible, would the line not utilize more than 1 lines worth of bandwidth with the load balancing?

I thought MLPPP was the "virtual" bonding solution which overcame single threads not achieving any more than 1 single connections speed.
The same way that my old isdn line could add 64k using multilink up to 128k and drop back to 64k without actually dropping the connection.

I believed the issue with MLPPP over anything other than isdn would either involve an isp supporting MLPPP on their gateways or a separate bandwidth provider but then you have to pay 2 companies for the bandwidth twice!

Could be way off on this, but just my thoughts!

cheers,
flipdee

Post deleted by uno

In reply to a post by helix_quazer:

I have load balancing technology with 3 WAN IP's, I'd be very grateful if you could explain true bonding with 1 WAN IP

3.5 hours ago you said you were using MLPPP so I am somewhat confused.

If it is MLPP remember there are two ends to this and the thing at the ISP end may be the constraint. They may have a configuration that limits you to one lines worth of capacity, in error, if MLPPP is presented as a single IP.

Also check individual speeds both up and down on all three lines as asymmetry can be an issue with MLPPP. I have seen two lines from the same cab & ISP have a massive difference in performance.

Yeh I wondered if that was the case, does anyone know how that actually works pushing down 1 WAN IP across multiple lines?

Who is your ISP ?

BT are my ISP, upon looking into bonding I am finding some very interesting stuff. It seems you can bond at layer 2 and layer 3 is even possible now. So MLPPP would be layer 2 bonding that requires equipment at the exchange, layer 3 bonding however would bypass the exchange and either require a device in the ISP's data centre, or if you want you could bond from site to site yourself with layer 3. I am currently unaware of how layer 3 bonding is done but i intend to try and find out.

I don't think BT offer or support MLPPP bonding.

I don't believe they do.

The login they give needs to terminate on the same gateway and deliver the same IP details at a minimum but also be specifically enabled for MLPPP.

Matt

In reply to a post by helix_quazer:

BT are my ISP

Retail/SOHO, or BT Business? There may be a difference wrt this issue.

BT business I believe. What I am trying to figure out at the moment is if I download traffic destined for my singular Public LAN IP and the ISP receives the traffic initially on their LNS (still destined for my 1 public LAN IP), the ISP then load balances across the 3 lines to get to my end so why would that not allow the speeds to be added together?

Load blaancing works in a number of different ways. If it is a connection that has been opened up then that connection will probably be set to always use the same path - if this isn't done then it can cause all sorts of problems trying to put it together again at the other end.

Most connections on simple load balancing use round robin. The first connection goes through pipe A. The second pipe B. Third pipe C and 4th back to pipe A. And it would keep doing this with no reference to how busy the pipe is and once a connection is open it would always go down the same pipe (so a connection to do a 10GB download would always use the same pipe).

An extension of this is to select the pipe used by the one that has most available capacity. But, a single connection would still stick with the same pipe.

Full load balancing would be packet level using the least used pipe for each individual packet. This requires a lot more intelligence and processor power. It also requires both ends to be powerful to be able to ensure packets are reassembled in the correct order.

Load balancing is a complex area and many systems (especially relatively low cost) will go straight round robin. There is the potential with round robin that the first connection is a 10GB download, second is 1KB, third 1KB and 4th back to 10GB. That means 20GB would come down 1 pipe and 1KB on each of the others - this can happen unless you invest in more expensive capabilities on load balancers.

The lines are load balancing equally and all 3 lines are the same 40mb/s down and 10mb/s up so it should be working fine. I am probably missing something here but as far as I can tell I should be getting 40 x 3 mb/s download as the ISP's LNS is receiving traffic destined for my 1 public LAN IP and then load balancing across the links.

Have you discussed your setup with BT support, i.e. do they confirm this should work?

There is a reason why the providers who support bonding in its various forms shout about it, i.e. it is fairly rare and I did not think BT Business Broadband of the GEA-FTTC or ADSL2+ variety supported bonding.

Load balancing is a different matter as you do not need ISP side support for that.

They have said that it should work, they also have had to configure a device on their side (the LNS) to load balance traffic they receive from the internet across my lines so as far as I can tell it should work.

I'm getting the impression that you have no idea what you're doing here .

I've never heard of BT retail offering bonded solutions or MLPPP support.

Your best option to test this is to do VPN bonding with a datacentre somewhere. This can be done regardless of ISP/router config.

Get a PC with two/three ethernet ports and a cheap gigabit VPS in the uk and tryy it out: http://simonmott.co.uk/vpn-bonding

How are you testing with only 1 PC, what spec Ethernet are you using to get to the router?

If the PC is only using 100Mb the router won't work much faster than 50Mb. (Make sure the PC is using 1Ge over a Ge port. (Not 100Mb over a Ge port)

Likewise the router is being set to use the Ge port as Ge not as 100Mb.

Are you downloading multiple things at a time the othe end may be restricting it if only one, ( Many sites will not stream at 100Mb to one end point).

(I am assuming the PC has the umph to actually do more than 100Mb itself with at least 3 download threads)

If the PC is only using 100Mb the router won't work much faster than 50Mb

Why? The PC on an ethernet 100Mb should be full duplex and therefore can very nearly max 100Mb. If the PC is only running half duplex then that in itself is a problem that needs to be fixed.

You rarely fill to the full capacity so a 100Mb ethernet port usually will max out at around 80Mb.

Which a tad more than "not much" over 50Mbps .

In reply to a post by kitcat:

You rarely fill to the full capacity so a 100Mb ethernet port usually will max out at around 80Mb.

On a full duplex port, I've had no problems transmitting at 99 Mbps. Main problem is getting data off the disk that fast, you really need SSD

Here's something that might help you out.

Run this command on your Cisco, the output should reveal if you're actually utilising MLPPP:

show ppp multilink

If it's working, you should see a single virtual-access bundle with three member virtual access interfaces.

Only so much help I can give on this, but hopefully points you in the right direction.
I'd be very surprised if BT Business were actually offering this, there's only a few ISPs like us actually utilising this for circuits.

Looks like I dont have mlppp running, i spoke to BT and they said I do have a bonded line but its done by load balancing over the 3 lines. I just cant work out if this is providing me with true bonding, i understand that usually it doesnt mean that but there is config on the ISP end in this case telling their device to route traffic destined for my public LAN IP down the 3 lines equally, does anyone know of a reason why this would not mean i get the speeds of all 3 lines? If anyone works at an ISP id be very grateful if you could explain how your bonding works, providing you offer it.

Are you sure BT said this?
What product is it where they offer load balancing by telling their device to route traffic destined for your public LAN IP down the 3 lines equally?
What device are they using?

If you pick a machine on your network to do a speedtest your router will just pick a line with the best availability and the speed test result will be the speed of that line.

If you ran three speed tests from different pc's then chances are each machine would pick up a different one of the 3 lines.

What you are looking for is one machine to do a Speedtest what the packets be split over your three separate lines so you can see a higher mb/s throughput, but this isn't going to happen as there is no mechanism at the ISP and to do this.

To get what you want all three of the lines need to be combined into a virtual path. To do this you can pay and ISP to host a little fire brick box for you in the data centre. Then you would have a fire brick at your end, and all would be combined as you want.

If you search you can find this on the AAISP website. It specifically tells you they cost for hosting a fire brick in one of their racks. I imagine however that you would have to pay for their data transfer between the fire break and their core network

You need a couple of these, one for the ISP end and one for your end...

http://www.firebrick.co.uk/products_2500.php

Then to host it the AAISP site states,

Special cases
There are some special cases such as hosting your own links to other racks, fibre converters, small FireBricks, etc. Please discuss with sales. For example the ongoing cost for a FireBrick works out to be £15.10+VAT plus bandwidth a month.

Plus bandwidth may be the key point.

In reply to a post by Ripley:

If you pick a machine on your network to do a speedtest your router will just pick a line with the best availability and the speed test result will be the speed of that line.

If you ran three speed tests from different pc's then chances are each machine would pick up a different one of the 3 lines.

Why bother risking it? Just plug each of the 3 modems into 3 PC's and run speedtests simultaneously. No need to risk "chances are".

Yeah that takes the router out of the equation.

What ever he does he's not going to get the combined speed of all three lines on an individual computer to triple the speed test results, unless the set up is changed by including something at the ISP end.

Like this? http://www.speedtest.net/result/3219692669.png

Hello,

Also consider some EntaNet resellers allow MLPPP on their LNS. I have heard of this working with FTTC.

I have three 20CN ADSL1 lines bonded using MLPPP and a Cisco 2821 through an EntaNet reseller.

Much cheaper than using firebricks. Alas with 20CN I do find backhaul contention can be a problem during peak times.

Tell Max, I mean helixquazey

In reply to a post by BatBoy:

Like this? http://www.speedtest.net/result/3219692669.png

That looks familiar!

That was using 2 different ISPs and per-flow load balancing. Speed test.net servers can be configured to run in single and multi-threaded modes. It's not quite there upload wise but works fine downstream.

The router I use can load balance 4 PPPoE lines however it does not have the horsepower to run 4 x 76Mb at line rate.

In reply to a post by Ignitionnet:

That was using 2 different ISPs and per-flow load balancing. Speed test.net servers can be configured to run in single and multi-threaded modes. It's not quite there upload wise but works fine downstream.

The router I use can load balance 4 PPPoE lines however it does not have the horsepower to run 4 x 76Mb at line rate.

There is some nice kit out there. PPPoE sadly mostly implemented in software, so 4 x PPPoE links running at 76Mbps and suddenly you need a full Intel Core series CPU, not often fitted to routers

Given the extra bytes needed to carry the IP payload over Ethernet, I'd be surprised if you managed 99 Mbps of IP data on a 100 Mbps port. 93 Mbps would be more like it.

In reference to en.wikipedia.org/wiki/Channel_utilization:

For example, in Ethernet the maximum frame size 1526 bytes (maximum 1500 byte payload + 8 byte preamble + 14 byte header + 4 Byte trailer). An additional minimum interframe gap corresponding to 12 byte is inserted after each frame. This corresponds to a maximum channel utilization of 1526/(1526+12)�100% = 99.22%, or a maximum channel use of 99.22 Mbit/s inclusive of Ethernet datalink layer protocol overhead in a 100 Mbit/s Ethernet connection. The maximum throughput or channel efficiency is then 1500/(1526+12) = 97.5 Mbit/s exclusive of Ethernet protocol overhead.

The router does everything in software, it is an SME bit of kit so does all kinds of stuff. TL-ER5120.

Not fibre, but just as an example - we have 3 standard BT ADSL lines bonded using routers/service from http://www.sharedband.com

Where's the example?

In reply to a post by MrSaffron:

Given the extra bytes needed to carry the IP payload over Ethernet, I'd be surprised if you managed 99 Mbps of IP data on a 100 Mbps port. 93 Mbps would be more like it.

Probably rounding errors on the FTP software I was using transferring a few tens of gigabytes from a very powerful IBM POWER server

In reply to a post by phil_w73:

Not fibre, but just as an example - we have 3 standard BT ADSL lines bonded using routers/service from http://www.sharedband.com

Hmm.

Between 49 and 299 GBP one-off router fee PER LINE and then a 10/20GBP monthly fee?

I kinda prefer paying a one-off charge of 160GBP for a single piece of hardware with no additional monthly fees or bandwidth limits.

I don't actually do anything that couldn't handle a single disconnect while my lines converge and I'd hope that anyone who needs such availability is using a dedicated leased line with a high availability SLA, not consumer broadband services.

Overpriced gimmick IMHO. For pretty much every application imaginable the kind of load balancing I use would be absolutely fine.

I'm starting to think my service is bonded properly, whoever my actual provider are and whatever they are doing sounds right. My line is essentially 1 line across the internet as far as I can work out. The provider has a device that takes traffic coming in destined for my singular public LAN IP and load balances across my 3 lines to my router, this traffic has a 1 source and destination IP and uses the 3 IP's across the lines purely for routing. If anyone knows of a technical reason why this would not work please let me know.

Hey helix_quazer,
Quick question, do you get a separate bill from another company (not BT) for a bonded service?
Or infact, do BT itemise an extra service on your bill for bonding? Having some more details on what you are actually being sold might help come to a conclusion on this.

Cheers,
flipdee

Nope, as far as I can tell I have 1 bill for 1 product which they have said is a bonded FTTC line. Altogether I should be getting UP TO 120 mb/s down and 30 mb/s up.

Excuse me for being nosy, but is there an "exact" product listed on the bill?
BT are very much a we sell what we list as a published product company, so in effect, what is listed on your bill should very much be published somewhere with exact details on what you get for your money.

cheers,
flipdee

Sorry for the lack of information but as far as I can see it says bonded FTTC solution, I am going to look into this more to see if there isn't a wholesaler sitting on the other end of my line or something, so far I have however just been dealing with BT.

BT used to resell Shareband, who were mentioned somewhere in the tread, software and products for their bonded solutions but I'm not sure if they still do.

Ok just to add a tuppence worth here.

Can you post your Cisco config on here?

If not can you confirm the following:

Are all your WICs are VDSL compatable EHWICs?

Do all the FTTC connections have the same credentials and therefore only needing to use one dialer?

Are you using the correct config?

interface Ethernet0/0/0
description FTTC2
no ip address
duplex auto
speed auto
pppoe enable group global
pppoe-client dial-pool-number 1


interface Ethernet0/1/0
description FTTC1
no ip address
duplex auto
speed auto
pppoe enable group global
pppoe-client dial-pool-number 1


interface Dialer0
description FTTC Dialer
mtu 1492
ip address negotiated
ip nat outside
ip virtual-reassembly
encapsulation ppp
dialer pool 1
dialer-group 1
ppp authentication chap callin
ppp chap hostname xxxxxxxxxxxxxxxxxx
ppp chap password xxxxxxxxxxxxxxxxx
ppp multilink

I have 1 WIC card which is a VDSL compatible EHWIC, the other 2 lines are just plugged into spare ethernet ports on the 2911. The config is correct as you have described but I have a separate set of login credentials for each line with separate diallers.

In reply to a post by helix_quazer:

I have 1 WIC card which is a VDSL compatible EHWIC, the other 2 lines are just plugged into spare ethernet ports on the 2911. The config is correct as you have described but I have a separate set of login credentials for each line with separate diallers.

What does show ppp multilink tell you?

In reply to a post by helix_quazer:

I have 1 WIC card which is a VDSL compatible EHWIC, the other 2 lines are just plugged into spare ethernet ports on the 2911. The config is correct as you have described but I have a separate set of login credentials for each line with separate diallers.

Ok so this is your problem. You will need two more VDSL EHWICs and a single login for all 3 lines. Then it will bond via MLPPP.

Is there any other way to bond other than MLPPP? I still think the set up I have does allow bonding of speeds but I can't find any information on why it woudn't

So 14 days later it looks very much like I was right. Thank you Flash.

How can all 3 lines bond if the radius sees them as 3 unique authentication attempts.

How can you get two of the lines to work when they are plugged into standard Ethernet ports.

The SP will be able to bond at their end. There are several out there that can do this but it ain't cheap.

You need a single authentication, a framed IP and 2 extra EHWICs.

In reply to a post by helix_quazer:

Is there any other way to bond other than MLPPP? I still think the set up I have does allow bonding of speeds but I can't find any information on why it woudn't

You need to provide us a link to what you have.

There are many ways to use multiple lines to carry traffic, only some of them should be called "bonding". BT were offering "Sharedband" at one point, which uses multiple lines or broadband technologies (potentially via multiple ISPs) to carry data between a dedicated CPE and a server at the ISP. MLPPP gives a single session with a single IP to one ISP. Various multi-session parallel load balancing solutions exist. Some VPN kit will split traffic over multiple paths and recombine at the other end.

You can load balance over 3 separate sessions it wont make a different whether they are bonded or not, all you need is the same AD configured to each route. Plugging into the LAN ethernet ports of a router are exactly the same as plugging into a WIC card as you still configure an ATM and a dialler per line.

I tend to be a bit frank at times but the best advice I can give at this stage is that you might want to talk to either a broadband provider that offers a proper bonded service and can support configuration at your end or one who supplies the kit in a managed form.

If you have a requirement to use a particular Cisco device then the good providers may well be able to advise on the set-up and configuration, but that may come at a price.

You've gone from talking about PPP level bonding to talking about load balancing across multiple sessions.

If you're using per-destination load balancing this would explain why you are pegged to a single line's performance. Disable route caching on your LAN interface to switch to per-packet.

besides I doubt bonding 3 FTTC lines is going to work with Sharedband, unless you accept that the max speed across all three FTCC lines using their Power Routers is 80 Mbps, which would equate to max speed of each FTTC being capped att 26.6 Mbps, worse still if you wanted 4 x FTTC lines as I do, the max speed on each would be throttled down to 20 because the router hardware is not up to it

http://uk.store.sharedband.com/index.php?routerinfo=Y

cost wise things start to mount up as well, you need a Power Router for each FTTC line at £149 ex vat, and the monthly line bonding charge of £20 for each FTTC which is capped at 2000 Gbytes data transfer a month for all 4 lines

http://www.sharedband.com/solutions/service-pricing/