21CN :(

Our office currently has two long lines (54dB) and we've found in the past that 21CN destabilises them. Our ISP has said that BT is now putting pressure on them to migrate us to 21CN. If we lose that option our office basically loses its internet connection. What irks me most is that our exchange (Bicester) was upgraded to support FTTC over a year ago but BT chose not to upgrade the cabinet.

We're hopeful of getting fibre to the site at some point but in the meantime it seems we're in danger of losing connectivity.

How do you mean "destabilises"? How does this manifest itself?
I've seen several reports of people have problems with their old routers and being advised to upgrade to routers known to work with 21CN. Is this the problem? What routers are you using?

Two lines, so what you do is migrate one line and once that has shown it works you do the second.

BT Wholesale is putting pressure on in the form that the old 20CN services are more expensive than 21CN and it is just about taking new orders in some areas http://www.thinkbroadband.com/news/5230-bt-wholesale...

As for the destabilises, with perhaps 25% of UK phone lines being this length or more, then we would expect many many more posts with problems. The vast majority are down to things like wiring issues that ADSL never exposed, or using old hardware that never got firmware updates to fix ADSL2+ bugs in it, or people not upgrading firmware.

You don't have to take the 21CN service, you could elect for another LLU provider, or your ISP could make sure that the lines are set to a fixed speed profile on 21CN if you are convinced you will have problems.

In reply to a post by BatBoy:

How do you mean "destabilises"? How does this manifest itself?
I've seen several reports of people have problems with their old routers and being advised to upgrade to routers known to work with 21CN. Is this the problem? What routers are you using?

If I could get the TBBQM graphs for early 2011 to load I could show you a before/after/back again sequence. The best I can do is pick one from earlier this year:

My Broadband Ping

Then today's:

My Broadband Ping

The 2011 sequence was as clear as day. We know from the ISP when the initial migration took place and we can see the reversion. Up until the migration it's a clean graph. Then a red bar as the line reconnects. Then packet loss and drops every few hours. Then exactly to the minute as far as the graph resolution shows we are moved back to 20CN and stability returns.

I didn't ask our ISP for the exact date/time of this recent migration but it was after I said "It's been bad for the last month or so" he responded with "Oh, hang on. Someone migrated you to 21CN recently."

The router is a Cisco 154x and has an Alcatel chipset which the ISP agent said probably doesn't help.

As for migrating one line at a time I don't think that will help. We have a bonded solution.

Edited by Andrue (Wed 10-Oct-12 11:47:14)

Which means the issue may be nothing to do with 21CN ADSL2+ but more to do with the WBC backhaul network and how the ISP manages the traffic/volumes it purchases etc

If latency performance is critical to business then its reconsider the bonded solution and look at the Ethernet options. Very little in the way of latency SLA for WBC.

In reply to a post by MrSaffron:

Which means the issue may be nothing to do with 21CN ADSL2+ but more to do with the WBC backhaul network and how the ISP manages the traffic/volumes it purchases etc

If latency performance is critical to business then its reconsider the bonded solution and look at the Ethernet options. Very little in the way of latency SLA for WBC.

We don't care about latency at all. What we care about is a complete lack of throughput which is what we're getting half a dozen times a day at the moment. Last night's outage caused our overnight synchronisation to fail despite it's supposed ability to handle errors. Our ISP thinks we will get migrated back later today so I'm pretty certain I'll be able to show you a graph for tomorrow that is almost completely clear of red.

So what are the router stats for both lines now then?

Nothing on the current BQM shows any problems with throughput, the yellow and blue simply shows some rises in latency consistent with you using the connection.

http://www.thinkbroadband.com/ping/share/68ee2ecbcb5...
Shows clearly when I went to bed and when I started using the connection this morning.

You can always force your routers to run in G.DMT mode.

In reply to a post by MrSaffron:

So what are the router stats for both lines now then?

I agree strongly here. Go and do it now, before they migrate you back. The line statistics will be important.

Include as much detail as you can - SNR, Bit loading, connection times & error counts. And try to keep some of those over time, before the migration happens.

Just to help later. For now, collecting data is important...

Nothing on the current BQM shows any problems with throughput, the yellow and blue simply shows some rises in latency consistent with you using the connection.

Or worse, the yellow & blue just show what is going on in the ISP network, or the 20CN link, and not your line alone.

I've been doing almost nothing on mine whatsoever. You certainly can't see where I went to bed! My Broadband Ping

In reply to a post by MrSaffron:

So what are the router stats for both lines now then?

I don't know - they manage the router. The agent just agreed that their systems showed the same problems ours do.

Nothing on the current BQM shows any problems with throughput, the yellow and blue simply shows some rises in latency consistent with you using the connection.

Sigh. At least one of us is confused here. To my understanding (based on several years of use but clearly you'd know better) the TBBQM display for our office connection shows it dropping several times a day and for today's graph virtually dead for about two hours around mid-night. The latter being confirmed by the error log from our synchronisation process.

Are you telling me that the TBBQM graph I posted is just 'business as usual' and a perfectly healthy line with no problems? That certainly isn't our experience and for every long red line we see in that graph there will have been a corresponding 'Connection to server lost' from all Outlook clients and/or a plaintive cry from someone in our office asking if the internet connection has failed.

Those are the kinds of things that only seem to happen a lot after our lines are migrated to 21CN and the correlation to the TBBQM seems pretty obvious to me. If that's an invalid assumption then clearly TBBQM is useless and I might as well stop using it.

Edited by Andrue (Wed 10-Oct-12 14:50:32)

I've mentioned before that the packet-loss graph being on a different axis is counter-intuitive and confusing.

If your backup software was busy between 10:30pm and 1:15am doing lots of uploading e.g. using 2 Mbps out of 2 Mbps of upload capacity that you have then

1. I would expect average latency to increase dramatically
2. Some packet loss as routers prioritise your traffic rather than our little ping, which probably gets responded to beyond the 500ms we consider a packet lost at.

Resyncs usually are shown as a period of solid red from top to bottom, as we get zero response during the 30 seconds to a minute this takes. So unless your ADSL modems are able to resync in just 2 to 3 seconds then I'd suggest that the graph is NOT showing a total loss of connection for that period, but a heavily utilised connection.

http://www.thinkbroadband.com/ping/monitors/graph/57... 9pm to midnight on there was me uploading a Gigabyte or more of data. No packet loss, but this varies from router to router, and not sure of how the bonded solution would impact things.

The ISP should have the connection sync history.

BQM is designed as a latency monitoring system, and packet loss detection, but knowing what is happening on the line and in the ISP is very important. We know some users who also set up a BQM to the first hop on their ISP network, to help understand when ISP congestion is an issue.

In reply to a post by MrSaffron:

If your backup software was busy between 10:30pm and 1:15am doing lots of uploading e.g. using 2 Mbps out of 2 Mbps of upload capacity that you have then

1. I would expect average latency to increase dramatically
2. Some packet loss as routers prioritise your traffic rather than our little ping, which probably gets responded to beyond the 500ms we consider a packet lost at.

And yet in the five or six years that's been running we don't normally see that kind of packet loss. Also the process started about half an hour into that outage and normally runs for up to three hours (it's pushing stuff in both directions to sychronize folders between sites.

That chunk of red/blue is not due to the synch process. It's something else that started before the sync, pretty much killed the synch (I think it gave up at about 3%).

In reply to a post by BatBoy:

I've mentioned before that the packet-loss graph being on a different axis is counter-intuitive and confusing.

Perhaps it is but I keep an eye on the graphs as part of my job and in my experience when the red lines meet and overlap the blue lines it means there's a problem. I don't know exactly what it means from a technical POV but I do know that when they do that anyone trying to access the internet or another office will be out of luck. I also know that normally they are few and far between (maybe one a week) but when we get migrated onto 21CN they increase to half a dozen or more every day and are often prolonged outages rather than a quick bounce.

Time will tell but I'm pretty sure that once we're put back on 20CN the red lines will be nothing more than a few pixels at the top of the image.

Blood drips. It doesn't levitate.

And without access to the sync data for the two modems and trying things like turning one off we cannot be sure the issue is NOT ADSL2+ related or the WBC network.

Can your provider not remotely manage the routers to force ADSL only mode on them? This would cure almost all issues where the line was not handling the extra bit loading from ADSL2+ very well, if that is what it is.

Hmm. Is this a problem whose nature is being masked by the bonding?

What is the likelihood that, when one line resyncs, the other does so at the same moment?

If one line resyncs alone, then what happens to the ICMP packets? Are 50% routed to the wrong modem, and lost forever? Or is the routing smart enough so it appears like (just) a loss of 50% capacity?

Either way, a sync on one of the lines is not going to look the same, on the TBB graphs, as "standard" broadband.

@Andrue: What happens when you deliberately turn one modem off, and leave things running on the other one? Has this been tried before?

In reply to a post by Andrue:

Perhaps it is but I keep an eye on the graphs as part of my job

Er, nice job. I prefer f8lure as it handles dynamic IP's, the graphs are on the same axis and you don't get the long grass obscuring the view.

I doubt it's proper bonding using G.Bond as 21CN was never supposed to work bonded. What is it, Sharedband?

In reply to a post by MrSaffron:

Which means the issue may be nothing to do with 21CN ADSL2+ but more to do with the WBC backhaul network and how the ISP manages the traffic/volumes it purchases etc

If latency performance is critical to business then its reconsider the bonded solution and look at the Ethernet options. Very little in the way of latency SLA for WBC.

dont be too quick to blame wiring issues, I certianly dont trust stats provided by openreach.

My experience was this.

adsl1, flaky but ran ok when heavily buffed up with high SNR.
adsl2, same as adsl1 without SRA, with SRA it ran much better and could work on lower SNR.
adsl2+, a disaster, much lower sync than adsl1/2 although with SRA still it wasnt less stable just a much lower sync speed.

I tried various routers, netgear dg384GT (sky router) broadcom 7402s (new and old types), speedtouch 585v4 and a couple of others. All were worse on adsl2+ than adsl2.

With all this said tho, I expect any adsl2+ service to allow people to force an adsl1 or adsl2 connection ie. backwards compatible, I could force adsl1 on ukonline and I could on xilo as well.

So to the OP assuming 21CN allows the router to overide the modulation mode Ithen you will be ok.

Allow them to be migrated, then ask that they be given an up to 8 profile.

Openreach run no hardware for ADSL2+ on 21CN, the MSAN is ran by BT Wholesale, openreach just provide a copper line

In reply to a post by MrSaffron:

Openreach run no hardware for ADSL2+ on 21CN, the MSAN is ran by BT Wholesale, openreach just provide a copper line

openreach do the home visits and probably provide stats from those visits on what they think are home wiring faults.

It is still the case, as long as your router allows you to do it.

I force ADSL2 instead of 2+, simply for stabillity.

In reply to a post by Chrysalis:

openreach do the home visits and probably provide stats from those visits on what they think are home wiring faults.

And your evidence that the stats are wrong is?

In reply to a post by BatBoy:

In reply to a post by Andrue:

Perhaps it is but I keep an eye on the graphs as part of my job

Er, nice job.

Lol, it's not a major part. Mostly I develop software however someone in our office has to be responsible for the connection and I volunteered.

In reply to a post by WWWombat:

Hmm. Is this a problem whose nature is being masked by the bonding?

What is the likelihood that, when one line resyncs, the other does so at the same moment?

If one line resyncs alone, then what happens to the ICMP packets? Are 50% routed to the wrong modem, and lost forever? Or is the routing smart enough so it appears like (just) a loss of 50% capacity?

Either way, a sync on one of the lines is not going to look the same, on the TBB graphs, as "standard" broadband.

That's my theory as well - it explains why the graph goes 'mostly red and blue' rather than entirely red.

@Andrue: What happens when you deliberately turn one modem off, and leave things running on the other one? Has this been tried before?

It's being tested now. So far the results are intriguing. I unplugged just one line and didn't bother to reboot the modem (it's a single modem with dual cards). Throughput has dropped (well duh) but that's all. No errors, no horrible ping.

Edited by Andrue (Thu 11-Oct-12 09:05:45)

In reply to a post by Andrue:

It's being tested now. So far the results are intriguing. I unplugged just one line and didn't bother to reboot the modem (it's a single modem with dual cards). Throughput has dropped (well duh) but that's all. No errors, no horrible ping.

I plugged the line back in, waited a minute then unplugged the other one. That caused a reaction. Killed the connection in fact. I've rebooted the modem now and it's back (on just the other line).

In reply to a post by Andrue:

In reply to a post by Andrue:

It's being tested now. So far the results are intriguing. I unplugged just one line and didn't bother to reboot the modem (it's a single modem with dual cards). Throughput has dropped (well duh) but that's all. No errors, no horrible ping.

I plugged the line back in, waited a minute then unplugged the other one. That caused a reaction. Killed the connection in fact. I've rebooted the modem now and it's back (on just the other line).

By "Killed", I guess you mean that the TBB graph went totally red? Rather than the red/blue combination?

I presume it was done just before your post, and I can see the top to bottom red that I normally associate with loss of connection.

There was a 1 hour segment of mixed red and blue 6:40 to 7:15am today and what looks to be fairly standard busy office until the early evening.

When you unplugged the connection initially did the connection continue to operate? If not then other than more capacity the service does not seem to provide redundancy.

Reconnecting the line should not normally need you to reboot the modem. I do this a lot when swapping around review hardware, and a reboot is very rare, unless I have been tinkering with settings and getting things wrong

In reply to a post by Andrue:

The router is a Cisco 154x and has an Alcatel chipset which the ISP agent said probably doesn't help.

Is this model correct? Is it still a current and supported model?

Cisco had some problems with their lower range (e.g. 8xx & 18xx) routers and those using a HWIC-1-DSL with ADSL2+, for which they released some updates.

Make sure your ISP is using recommended firmware and appropriate hardware for ADSL2+

The following has some more information:
http://www.cisco.com/en/US/prod/collateral/routers/p...
http://www.cisco.com/en/US/prod/collateral/routers/p...
http://www.cisco.com/en/US/docs/ios/12_4t/12_4t11/ds...

Bonding doesn't provide redundancy or resilience, you should use load balancing for that.

In reply to a post by MrSaffron:

There was a 1 hour segment of mixed red and blue 6:40 to 7:15am today and what looks to be fairly standard busy office until the early evening.

In theory there ought to have been nothing running at that time. We're a small office of engineers and I arrive first - at a couple of minutes before 7:15 today. I power cycled the router because I had no connectivity when I got in.

My Broadband Ping

It isn't normal (or wasn't until a month or so ago) for us to get any such blocks. If you look around 2pm you can see the result of me and a colleague applying Windows updates to two new machines that arrived in the office. Quite a bit of blue but not much red.

Edited by Andrue (Thu 11-Oct-12 11:12:02)

In reply to a post by WWWombat:

In reply to a post by Andrue:

In reply to a post by Andrue:

It's being tested now. So far the results are intriguing. I unplugged just one line and didn't bother to reboot the modem (it's a single modem with dual cards). Throughput has dropped (well duh) but that's all. No errors, no horrible ping.

I plugged the line back in, waited a minute then unplugged the other one. That caused a reaction. Killed the connection in fact. I've rebooted the modem now and it's back (on just the other line).

By "Killed", I guess you mean that the TBB graph went totally red? Rather than the red/blue combination?

Yup. I also plugged the original line back in and it didn't help.

So I agree that the 'red+blue' combo doesn't actually mean 'no connection' but it does mean 'unusable connection'. The mystery is why it happens and why it seems tied to 21CN.

I've just spent five minutes downloading Ubuntu (well running the download, anyway). I'm now going to plug the first line back in so that both lines are active. I won't reboot the modem though.

Edit: The router detected and and seamlessly increased our bandwidth back to the dizzy heights of 3.2/0.7.

Edited by Andrue (Thu 11-Oct-12 11:25:42)

Try saturating your upstream too

In reply to a post by MrSaffron:

Try saturating your upstream too

Okay I just did that. Aside from slightly slowing down browsing it didn't really do much.

My Broadband Ping

The red block is because our ISP asked us to turn the router off for 15 minutes to clear some stale sessions. They say there should be no problem reconfiguring our router to ADSL but they want to see how it runs for a few more hours.

I would try forcing ADSL or ADSL2 modulation like a few others here have mentioned

This seems to stabilise ADSL2+/21CN connections for those on longer lines.

In reply to a post by FRS_Plunderer:

In reply to a post by Chrysalis:

openreach do the home visits and probably provide stats from those visits on what they think are home wiring faults.

And your evidence that the stats are wrong is?

your evidence of them been right?

my view is based on a few things.

1 - my personal experience.
2 - aaisp's point of view.
3 - the long line policy from openreach where if a line is long a proper investigation isnt even carried out.

Do Cisco routers of this class really not have the SNMP Line MIB? Do "quality" ISPs really not have support people who know how to access it and plot it with MRTG and the like? We know BTwholesale are incapable of (or unwilling to do) that kind of thing, but then what are they capable of...

Get a look at the line error rate, sync speed, and SNR margin. Raw, rather than several levels of layering away, as the effects seen at IP level inevitably are.

Looking at IP-level figures is, as you have already pointed out, not an ideal way to debug a possible lower-layer (line level) problem, which the TCP layer will do its very bestest to mask (yes I know PING isn't TCP, but...).

We've had some more information now. It looks like it's our router that is the problem. We have a Cisco 1841 and we've found a few articles and forum threads suggesting that it is barely compatible with ADSL2+. Apparently there's a couple of firmware updates that improve stability. Cisco have end of lifed the router and the WICs and are now recommending we upgrade. over £700 for a new router and over £900 for two new WICs.

Pathetic really. I bet there's still some BT Frogs out there working quite happily. We're going to try and push our ISP to update the firmware (they have a service contract) and better yet by the end of this year or before spring we'll have a fibre based service to the office park.

http://stewartandrews.wordpress.com/2012/06/28/cisco...
and
https://supportforums.cisco.com/docs/DOC-6183#/?page=2

Edited by Andrue (Tue 30-Oct-12 17:59:05)