In reply to a post by mixt:

Not trying to punt A&A here (as I was a Zen customer some years ago), but A&A do offer connections on the BE back-haul. This is costly though compared to a *CN connection, so it seems if you want the reliability and quality, you need to pay the money for it.

It's worth pointing out that according to a statement on iDNetters, iDNet are starting to provide a service via Telefonica wholesale (aka BE). AFAICT this service is only in testing at present but it might be worth a look.

Edited by Tacitus (Sun 23-Oct-11 20:39:23)

In reply to a post by mixt:

You mention you administer many circuits, of a business critical nature (I presume). I think it would be fair to say that any business which is relying on the stability of broadband run over a rock bottom standard infrastructure known as 'the phone line' is, for the most part, probably not a business that is serious about paying proper money for a solid connection to the internet. If it really was that important to them, they would be going the SDSL/leased line route.

I never said that it was absolutely end of the earth critical. But clearing up lots of circuits that just don't work anymore and cannot authenticate again after many hours DOES cause major issues, especially when you cannot get updates from the ISP because its own phone system is down... Yes Zen, I'm looking at you.

I know, such technologies are way more expensive, but the reason is simple. Different infrastructure, away from the rest of the commoners and peasants like myself which use the cheap (phone line) based infrastructure.

Actually if you knew that much, you'd also know that leased lines and other similar technologies are generaly delivered over very similar infrastructure and go down. Ask AAISP what happened to its ethernet circuits last week when BT lost the DSL, that's right, the ethernet stuff died too.

That digger goes right through the lines, and goodbye leased line. Same infrastructure. The exchange my DSL is in and my Leased Line loses power (happened recently), both dead. All that extra money, same outcome.

In real terms having extensively tested this, I can get the same overall level of reliability by having 2 or 3 DSL Lines from different ISPs as I can by having a LL + DSL backup. Excep I save a lot of money.

Unless we start talking about true diverse Leased Lines (and often they're not as truely diverse as you believe - I've seen them have common points of failure too)

I understand your frustration by saying it should be Zen sorting this out, but you surely realise what you were signing up to when setting up the accounts. An SLA was probably not even in the terms of service.

It is in the spirit of the service offered that reasonable competence will be used. Killing lines just before the end of the working week to do rebalancing is not the best plan. Having a phone service for support as an ISP that died is pretty poor too. No resiliance by Zen there. Wouldn't matter if I did have a Leased Line from them, I'd STILL have been unable to reach them for support. Great stuff.

So to complain and rant here about that is futile. You would be better protesting with your feet, and moving, either to another provider which doesn't use the BT 21CN network, or SDSL/leased line where appropriate.

Funnily enough I moved many lines away from Zen 2-3 years ago. I do this as a matter of routine, but not every organisation has the money for lots of lines, and although not critical, reading a status page saying issues are resolved, then finding it is far from resolved and even a 20 minute turn it off and on again session leaves lines dead for another hour is just not acceptable - it is massively misleading and could have been avoided.

Plus, most of the reason these load balancing efforts lead to lines dying and failing to come back live with BTs Radius system and how it handles session reconnects - Zen should (as should other ISPs) be hassling BT on this since the way it handles this scenario is substandard and since 21CN comes up fairly regularly as a problem.

If the businesses you are managing can't or are unwilling to do this, then they must pay for that decision by make doing with a less reliable connection. Someone within those organisation needs to assess what such a decision would cost them in the long run (in terms of money they would make/lose for the time they don't have connectivity) when faced with such outages as this one.

It isn't always about money lost.

It is your problem though.

No, it is Zens problem. Right now I pay the money to them. If I choose to spend money with someone else it does cease to be Zens issue, but right now it is down to Zen. You're side stepping the responsibility.

[quote[
For me, it seems A&A provide the best service they can given the flaws of the infrastructure they have to work with.
[/quote]

Actually I'd disagree from personal experience. As much as I like Adrian and co, I don't think A&A offers the best service they can.

I think Zen are the same breed. Although, if load balancing is one of the aftermath issues caused by such outages as these, I agree they should be doing their best to obviate these issues where possible. I think (but maybe a Zen representative can confirm this) that the load balancing is actually done at BT's side, not Zen's, which is why Zen then have to re-distribute users after such an event, and why it is so disruptive to their customers when they have to knock people off and let them reconnect again to spread the load. I'm sure this is something that can be addressed, but maybe it's not a high priority. Either way, if it bothers you that much, maybe it is time to walk and join another provider.

Thanks for the lesson - but seriously, read my historical posts, I do point out if you want to have a "guarantee" you need LLs etc (but this is not the same as reality - an SLA can say anything, but it doesn't make it happen, and provides pretty poor compensation when it doesn't. An SLA is for the most part a box ticking exercise.

As I say, I have significant experience in this field, and have successfully resolved some customers long term reliability issues by moving them AWAY from Leased Lines some times - not because they couldn't afford it, but the service wasn't actually offering any enhanced reliability over DSL - by running them side by side for 6 months, I proved it. They moved to our Dual DSL service and since then I've never heard from them on support. It just works.

My issues are not ultimately with the "mission critical" debate, it is that:

(a) Zen made poor decisions to load balance when they did

(b) This was doubly poor knowing LB's cause stale sessions to happen every single time, and know it leads to death of connection for extended periods. The 20 minute claim is nonsense. All our circuits came back at EXACTLY the same time - that's not symptomatic of a BT stale session issue

(c) Zen's phone service disappeared so you couldn't get through. Bad lack of resiliance there on Zen's part and makes you wonder if in fact the issues are related.

As for an SLA, actually there is an implied SLA of sorts - anyone providing services is obliged to provide the service with reasonable competence and skill (a pretty common and well accepted legal term). That is what I expect. I'm not talking about stupid percentage figures and complicaetd documents full of get out clauses.

On a lighter note, my favourite SLA was one that excluded any sort of failure caused by a third party in any way. When I pointed out that this meant pretty much everything was excluded from the SLA, they didn't get it. Took us a lot of effort to get them to remove this clause. We had to remind them that we have no influence over the choice of suppliers they use, and that the way it had been worded meant absolutely everything was a third party clause. They clearly knew this and hadn't ever paid out on the SLA and nobody had bothered to read the SLA properly. That's why they're basically worthless.

Some very informative rebuttals there - thanks for that! I don't agree with everything you've said, but you raise some valid points which are food for thought.

I never have liked BT's authentication system, radius, PPP sessions, etc. I've seen the problems that framework causes having worked some years ago in a small ISP up in Scotland. Seems the underlying issues still haven't really been addressed 10 years forward, when BT have outages like the recent one. I guess this is why I have the pessimistic view I do about broadband that uses BT infrastructure, and why my general attitude on here is - put up with it, or move on. Cause it's clear it hasn't improved over the last decade (and according to some, 21CN is worse than 20CN!).

Regarding leased lines etc, I'm sure these other more expensive 'circuits' have to be, on the whole, more reliable than base broadband, otherwise why would they cost so much more? And an SLA does mean something, at least in places I have worked in. I do agree with you that if there is a clause which blankets 3rd party outages, then that's just ridiculous (and that's good you flagged this up). I see SLAs as a legal incentive to provide a reliable service. They tend to exist if the loss of service would mean loss of money (so said company compensates you for the outage, or has a guarantee to fix it within a reasonable time frame). Very much a legal thing, and something that should be ridiculed in depth before any SLA agreement is signed between two parties. As for diggers, and 'acts of God', that's just life. Maybe such companies should employ more competent digger operators, who maybe realise the implications of their actions when operating such powerful machinery!

Anyway, not much more to say - so I'll stop now.

In reply to a post by mixt:

put up with it, or move on. Cause it's clear it hasn't improved over the last decade (and according to some, 21CN is worse than 20CN!).

(a) 21CN has been much worse than 20CN for reliability. Without a doubt.

(b) You haven't always got a choice of alternatives to BT 21CN service.

(c) Regarding leased lines etc, I'm sure these other more expensive 'circuits' have to be, on the whole, more reliable than base broadband, otherwise why would they cost so much more?

Mostly historical - it used to be the only option. It remained high. Ask any ISP and they would tell you (if they weren't playing the commercially sensitive game) that they make a LOT more from selling you a 2MB Leased Line over a 2MB SDSL or ADSL.

And an SLA does mean something, at least in places I have worked in

Does it mean when your line fails for a working day and you lose a genuine amount of business, say £10K you get anything like that back? No, at best you would normally get a % of the monthly fee paid, and if you're really lucky, you get a months rental back. Has no bearing on the cost of the failure, and ultimately thus doesn't help you much.

I see SLAs as a legal incentive to provide a reliable service. They tend to exist if the loss of service would mean loss of money (so said company compensates you for the outage, or has a guarantee to fix it within a reasonable time frame).

I see them more as an agreement on what level of service the provider is aiming to achieve. Chances are they'll have it set it just under the real state so as to rarely, or never breach it. It really just sets out a basic understanding and normally a token compensation, but rarely has any other significant impact.

Post deleted by techguy

In reply to a post by therioman:

(a) Zen made poor decisions to load balance when they did

(b) This was doubly poor knowing LB's cause stale sessions to happen every single time, and know it leads to death of connection for extended periods. The 20 minute claim is nonsense. All our circuits came back at EXACTLY the same time - that's not symptomatic of a BT stale session issue

(c) Zen's phone service disappeared so you couldn't get through. Bad lack of resiliance there on Zen's part and makes you wonder if in fact the issues are related.

With respect, my understanding is that BT;s infrastructure is pretty dumb and does not evenly distribute traffic over the service provider's links. (something I'd have thought it being a 21st century network they'd have thought about)

I took the day off work to study for a Microsoft server course I was doing and I was rather annoyed that my connection was down pretty much all day.

However, I'd imagine that the decision was taken to do the load balancing immediately because a large amount of PPP sessions ended up routed over one connection meaning all connections would perform poorly which I'm sure you would have had an issue with?

So the question has to be, what would you have done?

In reply to a post by techguy:

So the question has to be, what would you have done?

Pretty simple, I'd rather slower speeds than no access at all. As it was the load balancing caused an outage far longer than Zen stated on Service Status AND left me with lots of totally dead lines for well over an hour.

What caused an hour of 'dead lines' ? Was it the BT outage? (which for me, only lasted 20 mins) or was it Zen performing the load balancing after the BT outage? What is happening for lines to remain 'dead' for hours? Is it the fact routers land up with one of the 172.* IP addresses from BT, think they are connected, and so don't log back in?

Genuine question, no trolling intended.

In reply to a post by therioman:

In reply to a post by techguy:

So the question has to be, what would you have done?

Pretty simple, I'd rather slower speeds than no access at all. As it was the load balancing caused an outage far longer than Zen stated on Service Status AND left me with lots of totally dead lines for well over an hour.

Personally I'd agree with you and so would a proportion of others I'm sure but there would be an equally large proportion that would hit the phones and forums complaining about the slow speed and wouldn't take 'we're going to rebalance traffic across the links overnight' as an answer as there were probably some business customers trying to pull down large files etc.

Also it being Friday a lot of home customers would also complain.

I think they were in a no win situation TBH, whatever they decided to do would have drawn fire from a section of customers.

It does demonstrate how bad the BT system design is though that it requires operations staff at an ISP to spend hours manually kicking off PPP sessions and monitoring reconnections (which from the behaviour of my router and what I've read elsewhere is what I understand they have to do until the connection reroutes over another link) until they have an equal number of connections running across each hostlink.

Even if BT won't allow ISPs to specify which link a conection comes in over it should be possible for BT's own network operations staff to do it (without charge) following an incident or on requiest of an ISP..

Wake up Ofcom and put the national infrastructure into the hands of a company mutually owned by all service providers that is a non profit and can run it properly and not just to line the pockets of BT shareholders and Directors.

Edited by techguy (Fri 28-Oct-11 01:02:20)

In reply to a post by techguy:

Personally I'd agree with you and so would a proportion of others I'm sure but there would be an equally large proportion that would hit the phones and forums complaining about the slow speed and wouldn't take 'we're going to rebalance traffic across the links overnight' as an answer as there were probably some business customers trying to pull down large files etc.

But the solution there would be disconnect/reconnect - it would "load balance" them (that's what hapens anyhow AFAIK in essense) - except it would be clean and not leave a long period of downtime if the session gets stuck.

I think they were in a no win situation TBH, whatever they decided to do would have drawn fire from a section of customers.

Yeah sure, I know you can't please everyone all the time, but Zen is supposedly a premium brand and premium service, the least impacting way would be the one I suggested.

It does demonstrate how bad the BT system design is though that it requires operations staff at an ISP to spend hours manually kicking off PPP sessions and monitoring reconnections (which from the behaviour of my router and what I've read elsewhere is what I understand they have to do until the connection reroutes over another link) until they have an equal number of connections running across each hostlink.

Maybe so - but Zen have the power to hassle BT over this, we don't. They're BTs customer and they had plenty of time to eval this, and put in requests for