BT Congestion Monitoring

http://status.aa.net.uk/1890

Does this sum up BT and their ability (or inability) to monitor congestion within their exchanges and 21CN network?

Edited by deleted (Mon 03-Mar-14 09:23:15)

Probably.

I'm seeing similar on another exchange, my ISP is trying hard but BT are not easily convinced that their network can possibly be at fault

Hey, we track the congestion we've found on this page as well:

http://community.plus.net/exchange-information/

The complaint is about getting BT to track (and do something about) possible congestion, everyone else seems to find it without any trouble

That could be a useful page, thanks. Even if my exchange isn't on it, but you can see my problem on the BQM links in my sig.

@ Kelly

Plusnet and AAISP are the two proactive ISPs in this area who do a very good job at identifying and publishing capacity issues.

The point I am trying to make is - how are ISPs able to identify capacity problems in someone else's network, yet the people that own and operate that network (i.e. BT Operate/BT Wholesale) seem unable to do so?

Which exchange are you?

I suspect that what one operator calls a capacity problem is what another says is normal behaviour.

Didcot (SMDC)

80/20 FTTC.

probably good answer.

eg. I consider delayed packets (higher latency) as congestion type behavior even tho throughput may not be immediatly affected. Whilst someone else would not consider it worthy of attention.

I think some have commented that they think its unusual that BTw has no SLA for packet loss?

In reply to a post by MrSaffron:

� another says is normal behaviour.

When a BT speedtest on an 80/20 line returns this:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Download speed achieved during the test was - 15.43 Mbps
For your connection, the acceptable range of speeds is 51.67 Mbps-64.58 Mbps .
Additional Information:
IP Profile for your line is - 64.58 Mbps

Upload speed achieved during the test was - 12.25Mbps
Additional Information:
Upstream Rate IP profile on your line is - 20 Mbps
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

That's not normal behaviour


eta- well, it shouldn't be, but recently it is in the evening

Edited by billford (Mon 03-Mar-14 13:39:43)

I would not trust the BT Wholesale speed test as a speed test, too little capacity generally unless it has had a massive upgrade in the last couple of years.

Either no-one wanted to pay for one, or they never even offered the possibility.

Its a rerun of the old BT Central problems, so hardly the first time we've had issues on backhaul networks.

Saying that last Friday night thought it was hitting me bad, but turned out to be wireless congestion/interference locally as flipped to Ethernet and fine.

You may well be right, but it's the only one (afaik) that BT take any notice of.

The tbb tests (not all saved) give similar figures, also backed up by my BQMs since before Christmas- I hope you're not going to say that they're not trustworthy

Indeed.

But do you not think that AAISP service status page is bad publicity for how BT manage and operate their network?

What I find worrying is that end users report speed problems at peak times, an ISP identifies other end users with the same problem and suspects exchange capacity issues, but Openreach engineers are still sent out.

In reply to a post by KellyD:

Hey, we track the congestion we've found on this page as well:

http://community.plus.net/exchange-information/

I bookmark this. Useful to check for my exchange live status but my exchange isn't on it thought. I just wish BT Wholesale should provided all exchange status on BT Wholesale Checker

In reply to a post by AndyHCZ:

� but Openreach engineers are still sent out.

Doubtless it's in the script� shortly after "Have you re-booted your router?"

I trust ours as we can see the network and server loads.

The key indicator of congestion as always with our tester is that the x1 and x6 are very different shape graphs

It has been like that for a decade or more...

Working in support does not mean you have to understand your network map, just to know that the green light saying all is well is still on. The fact that the light is hardwired to the mains and has battery backup is never learnt.

In reply to a post by MrSaffron:

The key indicator of congestion as always with our tester is that the x1 and x6 are very different shape graphs

I'm not sure that helps on mine I'm afraid- Safari on OS X.

In reply to a post by MrSaffron:

not trust the BT Wholesale speed test

But you have to trust it as my ISP (plusnet) say that they will not log a fault for speed if the results are not from the BTW speedtest site.

I have that in writing on a support ticket from PN staff.

In reply to a post by MrSaffron:

I suspect that what one operator calls a capacity problem is what another says is normal behaviour.

BT Wholesale were supposedly trying to avoid being the bottleneck, which given they make their money selling bandwidth makes perfect sense.

Well they could buy a server to run on their own network

Plusnet can't accept them for reporting to BT Wholesale because BTW won't accept any other.

In the case of this thread it's a bit irrelevant, as the BTW tests are showing low speed. It only matters when the BTW ones show full'ish speeds but others don't and the live experience is poor.

Edited by RobertoS (Mon 03-Mar-14 16:22:43)

Yeah with peak pricing one would have thought a self interest to keep scaling network, odds on someone set a trigger for next upgrade and something has delayed the improvements or the accountants rather than the techs are in charge again.

An awful lot of it looks like out and out mess ups. Failing to install hardware in advance of time and now that physical ports have been allowed to saturate having to do the planning, hardware installation, etc, for new cards and fibre.

SVLANs maxing out can be resolved through some rate limit changes, saturating the physical ports, which has happened, means lead times while cards and fibres are installed and whatever work around those is done.

Installing line cards requires a change request, building new physical paths between exchanges and LACs isn't painless either.

BTW Proactive / capacity planning dropped the ball.

Wild card - the one person who knows what to do was probably on holiday.

Can't see much in our data indicating a problem :/

I'm told it could also be a congestion problem with "my" cabinet, or more likely the fibre link to the exchange.

There was some work being done there recently which I thought might be a second cabinet going in, but that now seems unlikely. It's been suggested that it may have been another fibre being blown in- possible, the cabinet was fairly early in the FTTC rollout and BT may have guessed wrongly about the take-up.

Or it could be something else entirely, time will tell

I struggle with the idea that at most 288 people could consistently use close to a gigabit of data for hours on end.

It's not impossible but that'd be a crazy level of usage, Virgin Media can happily share 400Mb between 250+ users pretty much everywhere bar student areas where there are small armies on each modem.

So do I, but something's causing it, and the pattern is both repeatable and consistent with domestic usage.

On weekdays the latency and packet loss start rising (and speeds dropping) at around 5pm, stay high until about 11pm then fairly quickly return to normal. At weekends they normally start rising around midday, but last Saturday was nice and sunny here and the rise didn't start until about 5pm.

Sounds domestic to me� otoh I'm open to other suggestions

It doesn't need 288 people.

Fourteen will do, as long as they aren't all on the same connection.

I think 14 people is a bit optimistic, but 14 households�

I must have slipped a decimal point or two on my first calculations, did them again and it wouldn't take many households on the same cabinet to saturate a gigabit link for hours on end.

And if the junk mail around here is any guide, there ought to be plenty enough who've signed up to BT Sport, Netflix et al, then there's all the broadcast catch-up services�

Yup, eminently possible. I started to see signs of it on the BQMs about mid-December which could also fit.

ps

I'm already getting BT speedtests (just) below the minimum acceptable, and the night is yet young�

~~~~~~~~~~~~~~~~~~~~~~
Download speed achieved during the test was - 51.81 Mbps
For your connection, the acceptable range of speedsis 51.93 Mbps-64.91 Mbps .
Additional Information:
IP Profile for your line is - 64.91 Mbps

Upload speed achieved during the test was - 15.38Mbps
Additional Information:
Upstream Rate IP profile on your line is - 20 Mbps
~~~~~~~~~~~~~~~~~~~~~~

The odds of 5% of households maxing out 70Mb+ simultaneously are tiny. ISPs are still deploying sub-500kbps per subscriber, to saturate a gigabit with 288 connections the average usage needs to be 6 times that.

Again it's not impossible especially if there's no cable and/or a satellite dish ban, but it'd be very unusual.

Openreach are also supposed to deliver a minimum data rate of 20Mb per 40Mb subscriber if I remember rightly, meaning a fully loaded cabinet, even with all 40Mb customers, should be packing 3Gb of backhaul requiring each and every modem on the cabinet to average over 10Mb/s.

It certainly does look like congestion, however Plusnet monitor packet loss and there's relatively little of that.

I don't suppose you know anyone on your cabinet with Sky or TalkTalk FTTC, Bill?

In reply to a post by Ignitionnet:

The odds of 5% of households maxing out 70Mb+ simultaneously are tiny.

Not so sure about that� as I said earlier- BT Sport, Netflix, catchup etc. All bandwidth eaters.

In reply to a post by Ignitionnet:

I don't suppose you know anyone on your cabinet with Sky or TalkTalk FTTC, Bill?

No, I think the cabinet serves a fairly large area and I don't know who else has gone for fibre.

Last estimate I had for numbers was about 18 months ago iirc, and the BT engineer (who had to go to the cabinet to fix the problem I had) reckoned around 70-80 users.

Edited by billford (Mon 03-Mar-14 20:26:06)

In reply to a post by Ignitionnet:

however Plusnet monitor packet loss and there's relatively little of that.

I'm not with Plusnet

There's a lot more than there ought to be on the BQMs, but I'm not overly bothered about that. If something is congested enough to give the speed losses I get then it's inevitable that ping packets are going to get dropped. It doesn't mean that real data is going missing.

In reply to a post by billford:

Not so sure about that� as I said earlier- BT Sport, Netflix, catchup etc. All bandwidth eaters.

They are but trust me it's very unlikely. Have a look at Plusnet's user numbers compared with the total bandwidth consumption on their graphs, 700,120 users into 150Gb of capacity, each customer is allocated 215kbps.

For the sake of argument, and to remove the effect of ADSL if we quadruple the bandwidth usage just to be safe we're still below 1Mb/s per customer, and stats from cable companies indicate that doubling of speed only increases bandwidth usage by around 30%.

It's not impossible but you'd be in a pocket of very heavy users indeed. As I mentioned Virgin Media have been selling 30, 60 and 120 Mb split between 250+ modems on a 300-400Mb pipe for a while. If anything 250 modems is generous and it's more like 500 customers on a 300-400Mb downstream group.

Another thing that's odd is that the maximum latency remains the same, which indicates a maximum buffer size, however the loss is relatively low level and doesn't seem to correspond with the worst of the average latency jump. There was also low-level packet loss until 4:30am.

Monday is the busiest day of the week generally, see what happens. I work with network performance as the day job and this has piqued my interest as it might be an insight into how the Openreach kit manages congestion.

In reply to a post by Ignitionnet:

Monday is the busiest day of the week generally, see what happens. I work with network performance as the day job and this has piqued my interest as it might be an insight into how the Openreach kit manages congestion.

I find it's fairly consistent throughout the week, if anything Friday is the worst. It's only worse at the weekends in terms of duration, not magnitude.

If you want I can send you links to my BQMs for the last week. Or the last four months come to that (tbb don't seem to keep them longer than that), but I'd want a bit of notice

In reply to a post by billford:

In reply to a post by Ignitionnet:

however Plusnet monitor packet loss and there's relatively little of that.

I'm not with Plusnet

There's a lot more than there ought to be on the BQMs, but I'm not overly bothered about that. If something is congested enough to give the speed losses I get then it's inevitable that ping packets are going to get dropped. It doesn't mean that real data is going missing.

Plusnet have a 5% market share so there should be the odd person on the Didcot exchange who uses them

Actually the ping packets do mean that real data is going missing. In fact more 'real data' is going missing than ping packets though obviously the 'real data' gets retransmitted. You're only getting one of these per second while when you hit that download button you're getting thousands of TCP packets inside the link per second.

The PPP frames carrying the pings are also going to be smaller, hence less likely statistically to be dropped. They consume less buffer space and are potentially going to be preferred by routers and switches on the path, depending on the queuing mechanism they are configured to use. The BT BRAS shaper isn't the smartest, hence why AAISP use their own, but the Openreach stuff, who knows?

Congestion slows speeds because of loss. The congestion detection algorithms that are causing your 'real data' to be slow operate on loss. You obviously don't see this because TCP retransmits the data but it's loss, and the TCP protocol's attempts to avoid loss by throttling back, that's causing the lower throughput.

Edited by deleted (Mon 03-Mar-14 20:46:56)

In reply to a post by billford:

I find it's fairly consistent throughout the week, if anything Friday is the worst. It's only worse at the weekends in terms of duration, not magnitude.

If you want I can send you links to my BQMs for the last week. Or the last four months come to that (tbb don't seem to keep them longer than that), but I'd want a bit of notice

I'd settle for which cabinet you're on, thank you

[Removed]

Edited by billford (Mon 03-Mar-14 21:39:19)

Thanks muchly.

yes I was going to mention this but I feel uneasy to disagree with any of your posts now

openreach have a stated min speed from cabinet to exchange per end user, which should mean more than 1 gig backhaul on a fully loaded cabinet.

in my view this means the contention ratio is so low that congestion be extremely unlikely on the cabinet.
As you said isp's contend at much higher levels so congestion would be expected first either at isp or exchange (by this I mean anywhere between exchange and isp).

Edited by Chrysalis (Tue 04-Mar-14 08:55:50)

In reply to a post by Chrysalis:

yes I was going to mention this but I feel uneasy to disagree with any of your posts now

Given you're agreeing with me I think you're quite safe. For now.

Have you seen this blog post - http://revk.www.me.uk/2014/02/bt-21cn-not-fit-for-pu... ?

It would be good if TBB could look at this and possibly do some digging on what was said at LINX on IRC to the UK ISPs. Not sure if this is confidential or not?

Edit: And this one also - http://revk.www.me.uk/2014/02/bt-official-3-packet-l... "3% packet loss is not considered as a fault" [Krishanu Sanyal, Broadband Customer Service Team Manager - BT Wholesale].

Edited by deleted (Fri 07-Mar-14 09:16:13)

sounds like the attitude of engineers as well

"we are right, we refuse to investigate"

so that motto is in all parts of BT it seems.

Here's another one! Nothing to do with congestion..

http://revk.www.me.uk/2014/03/bt-refusing-to-fix-fau...

I think the 21CN network is in its worst state (performance wise) for some time. This all comes on the back of the Chief Network Architect for BT stating recently to all ISPs that "there are no BT exchanges at all that have any congestion".

These are the following exchanges where users are reporting peak time slow downs.

There are a couple of long threads about this - http://community.plus.net/forum/index.php/topic,1247...

https://community.bt.com/t5/BT-Infinity-Speed-Connec...

http://forums.overclockers.co.uk/showthread.php?t=18...

Surely TBB should be looking at this/doing an article? Edit: TBB have done an article - http://www.thinkbroadband.com/news/6360-no-sign-yet-... Much appreciated!

Plusnet

Cuckoo Oak (WNCKO)
Atherstone (EMATTHE)
Runcorn East (LVRNE)
Rossendale (LCROS)
Reading Tilehurst (THTT)

Bangor (WNBG)
Burton On Trent (EMBURTO)
Stepney Green (CLSTE)
Folkestone (NDFOL)
Crouch End (LNCED)

BT Retail - Taken from their Community Forums

Swadlincote (EMSWADL)
Grays Thurrock (EAGRA)
Burton On Trent (EMBURTO)
Gipsy Hill (LSGIP)
Normanton (MYNMN)
Rainham (NDRAI)
Oxford (SMOF)

AAISP - Taken from http://clueless.aa.net.uk/congestion.cgi

Loughton (LNLOU)
Harlow (EAHLW)
Coventry (CMCGF)
Crouch End (LNCED)

Edited by deleted (Tue 18-Mar-14 11:45:56)

My exchange is on the list too. Cuckoo Oak and last night speed: http://www.speedtest.net/my-result/3377537022

Edited by adslmax (Tue 18-Mar-14 18:29:25)

In reply to a post by adslmax:

My exchange is on the list too. Cuckoo Oak and last night speed: http://www.speedtest.net/my-result/3377537022

Plusnet and AAISP seem to disagree, it's not on either of their lists and Plusnet specifically said that they do not see widespread issues on Cuckoo Oak.

Nice bit of work but worried now that people with other issues who posted slow speeds are going to be blaming congestion. Not convinced everyone is suffering congestion and at least one exchange on your list an ISP has specifically said does not show congestion.

[img]http://www.thinkbroadband.com/speedtest/button/13951...[/img]

no issues here

We put some new reporting in this afternoon to try and get a better view of what's going on. BTW now provide us with a list of which customers are on which SVLAN within the BTW network so our quality monitoring can now look for congestion down at the SVLAN level. This should be really useful as we can sometimes look at a big exchange and the data can be a bit inconclusive because packet loss or latency increases for 25 customers gets lost amongst perfect service for 500 others. It was from this data that Folkestone popped up on the report (we think that's fixed now).

I'll take a look at Cuckoo Oak and Didcot tomorrow to see if I can see anything at those sites.

In reply to a post by adslmax:

[img]http://www.thinkbroadband.com/speedtest/button/13951...[/img]

no issues here

But! Two hours ago you posted last night's speed as http://www.speedtest.net/my-result/3377537022

Make your mind up .

another speed test result: [img]http://www.thinkbroadband.com/speedtest/button/13951...[/img]

The question is what happened last night?

In reply to a post by DaveTomlinson:

I'll take a look at Cuckoo Oak and Didcot tomorrow to see if I can see anything at those sites.

Didcot:

My Broadband Ping

In reply to a post by DaveTomlinson:

We put some new reporting in this afternoon to try and get a better view of what's going on. BTW now provide us with a list of which customers are on which SVLAN within the BTW network so our quality monitoring can now look for congestion down at the SVLAN level. This should be really useful as we can sometimes look at a big exchange and the data can be a bit inconclusive because packet loss or latency increases for 25 customers gets lost amongst perfect service for 500 others. It was from this data that Folkestone popped up on the report (we think that's fixed now).

I'll take a look at Cuckoo Oak and Didcot tomorrow to see if I can see anything at those sites.

Dave,

it would be awesome if this data could be sorted so that customers can check for congestion.

It was done on 20CN so fingers crossed it's feasible on 21CN.

In reply to a post by RobertoS:

The question is what happened last night?

exchange capacity work overnight probably the reason behind this.

Edited by adslmax (Tue 18-Mar-14 22:30:08)

I agree with what you've said about people with other issues blaming congestion - we've already kind of seen it:
http://community.plus.net/forum/index.php/topic,1249...

The article by MrSaffron is good because it highlights the TBB tools available to check for problems. Hopefully people will take note.

As for Cuckoo Oak - I went on the info that goldenfibre/adslmax provided. There does seem to have been some capacity upgrades in the early hours of this morning for that area (http://status.zen.co.uk/broadband/maintenance-outage-details.aspx?reference=236432). It is a little bit confusing when BTw put "XXX region". The Much Wenlock exchange itself doesn't appear to be 21CN yet, so the planned works description is a little confusing...

In reply to a post by AndyHCZ:

As for Cuckoo Oak - I went on the info that goldenfibre/adslmax provided. There does seem to have been some capacity upgrades in the early hours of this morning for that area (http://status.zen.co.uk/broadband/maintenance-outage-details.aspx?reference=236432). It is a little bit confusing when BTw put "XXX region". The Much Wenlock exchange itself doesn't appear to be 21CN yet, so the planned works description is a little confusing...

For sure, but Max does not appear to have been disconnected overnight and his graph isn't really changed.

We'll see; maybe there was congestion and other customers were moved to new capacity and then his SVLAN flexed.

Edited by deleted (Tue 18-Mar-14 22:34:23)

No issue here tonight
Dave Tomlinson / AndyH

http://www.thinkbroadband.com/speedtest/button/13951...

In reply to a post by Ignitionnet:

For sure, but Max does not appear to have been disconnected overnight and his graph isn't really changed.

We'll see; maybe there was congestion and other customers were moved to new capacity and then his SVLAN flexed.

No, my fibre haven't been disconnection. That's is correct.
http://www.thinkbroadband.com/ping/share/7f20f6d2e7c...

Edited by adslmax (Tue 18-Mar-14 22:39:01)

This is true, but normally there is a warning like "There will be one service outage to End-Users of approximately XX minutes within the period 00:01 - 06:00." There were quite a few of these region capacity upgrade works in the early hours of this morning.

There were also some planned works for moving customers from congested SVLANS, to non-congested SVLANs. These have to result is short outages...

Yes indeed, Max had none so I was trying to think of ways to fit around that.

SVLANs are rate limited and that rate limit can be changed, though I'm not sure if these are in the PEWs as they shouldn't be service affecting.

SVLAN moves, both customers to different ones or of entire SVLANs to new endpoints, definitely do cause a disconnection.

Why is my graph showing yellow spiked all day? How to kept it low minimum low? Maybe my Fibre router issues or I will try swap Cable router tonight to see if the graph spiked have become low.

Any idea of why is this?

My graph live: http://www.thinkbroadband.com/ping/share/7f20f6d2e7c...

16th March Graph: http://www.thinkbroadband.com/ping/share/0ec85dc7659...

17th March Graph: http://www.thinkbroadband.com/ping/share/91d6e475004...

18th March Graph: http://www.thinkbroadband.com/ping/share/16bac900a99...

My final speed test: http://www.thinkbroadband.com/speedtest/button/13951...

Edited by adslmax (Tue 18-Mar-14 23:34:43)

If it's not affecting usage it doesn't matter. Some routers are slower responding to the pings than others.

I am not so sure

He clearly needs to buy a commercial grade Cisco and configure that.

BT already have case open at Didcot for degraded service, I've added the affected Plusnet customers to the case to help, we can definitely see problems on 3 SVLANs, a 4th shows up on the SVLAN status report as being upgraded and a 5th could be affected but we don't have enough customers on that one to be 100% certain.

I've added the exchange to our exchange page:

http://community.plus.net/exchange-information/

The packet loss we see across our customers pretty much correlates with your TBB graph.

I've just checked Cuckoo Oak for last night and our monitoring doesn't show any problems, unfortunately we only have 1 FTTC customer on that exchange (we have others on the parent exchange Telford Wellington).

Hopefully the PEW you've linked on Zen's site for yesterday morning has fixed that one. This is the description of the PEW:

This Planned Engineering Work is required on the 21C Network within the (Much Wenlock) region for (capacity upgrades). Current circuits are nearing capacity limits, which could result in packet loss and degraded service. We are increasing the capacity of these circuits to reduce the long term likelihood of service degradation. There will be brief service outage between 2 & 20 minutes. This will occur during the window of 02:00 - 03:00.

Thanks Dave, I appreciate the information.

In reply to a post by DaveTomlinson:

I've just checked Cuckoo Oak for last night and our monitoring doesn't show any problems, unfortunately we only have 1 FTTC customer on that exchange (we have others on the parent exchange Telford Wellington).

Dave, if Telford Wellington is the NGA parent won't all the customers use the same SVLANs?

Yeah, all the BTW fibre customers on these exchanges will use the same SVLANs:

CUCKOO OAK
DAWLEY
DONNINGTON
HOLLINSWOOD
NEWPORT, SHROPSHIRE
OAKENGATES
SHIFNAL
STIRCHLEY
TELFORD WELLINGTON

There's currently 6 SVLANs at that exchange.

IMHO early adopters of FTTC will be disappointed with the average speeds they will get in the future as more people switch - it is akin to expecting to always drive at 70MPH on a motorway.

From a BT document I came across:

BT Wholesale Throughput Ranges
The network dimensioning rules are designed to ensure that customers receive an IP throughput based on the product they have purchased, regardless of the activity of other users within this shared part of the network and such that users can achieve a throughput of:
at least 8Mbit/s for 90% of the time over the busiest 3 hour period for the Standard WBC (12Mbit/s for FTTC);
at least 12Mbit/s for 90% of the time over the busiest 3 hour period for Elevated WBC (16Mbit/s for FTTC).

In actuality the average speed experienced in the busiest 3 hour period each day is generally much higher than the speeds quoted above. The IP throughput stated above is always lower than the headline ATM ADSL synchronisation speed due to the protocol overheads introduced by the ATM and PPP layers.

If you want uncontended connectivity you can always stump up for GEA-FTTC at around £400/mo for 80/20...

FTTC doesn't use ATM.

Hi,

Those numbers are out of date now for FTTC/FTTP. The new numbers are:

On 40/2 and 40/10 Standard it's now 20Mbps throughput
On 40/2 and 40/10 Elevated it's now 30Mbps throughput
On 80/20 Standard it's now 40Mbps throughput
On 80/20 Elevated it's now 60Mbps throughput

The Service Level Agreement (SLA) is that users should achieve the above throughputs for 90% of the busy 3 hour period.

There's a percentage calculation if your line doesn't sync at the max rate for the product.

In reply to a post by DaveTomlinson:

BT already have case open at Didcot for degraded service ...a 4th shows up on the SVLAN status report as being upgraded ...

I think that may have been mine... so far tonight my BQMs and speed tests are back to where they ought to be .

http://www.thinkbroadband.com/speedtest/results.html...

Normally by now I'm approaching ADSL2 speeds- from the wrong direction

GEA-FTTC is what you have just described. I'm not sure what you were meaning, at £400pm.

Dave
Any chance you can look at a rather small exchange, SWRSO.
Neighbours and myself have been having trouble for a few months

http://forums.thinkbroadband.com/zen/f/4292019-conte...

Not sure if you have any customers on this exchange (5% of 78 = 3ish)

my TBB bqm
http://www.thinkbroadband.com/ping/share/c8131dbf4cf...


Thanks in advance
Ian

(EDIT ADD MY tbb bqm)

Edited by ian007jen (Thu 20-Mar-14 07:52:47)

Your BQM link seems to have a problem. In Firefox under Win 8.1 it goes to a google results list, and on an iPad it goes to the tbb login screen.

Ian007jen

Your exchange is 20c only so will not be covered by the above posts which were looking at WBC and FTTC backhaul.

Dave may have some other mechanisms to look at your exchange as they used to publish data for 20c congestion. Think you have to be a Plusnet customer to see it through.

I believe Zen have something similar on their portal that you should be able to see as a Zen customer.

As the exchange is so small the backhaul is likely to be physically limited maybe only 34Mb. (28Mb through put after ATM headers etc)

With 78 customers it will never be cost effective to put additional hardware in, BDUK will eventually run something back to another exchange and offload the existing equipment. So 14 of you streaming video at the same time using 2Mb each will cause contention.

There are no dates for FTTC or other at Rhos yet, but Haverfordwest now shows as Mar 14 ( they will have to rush) so you may have some hope by the end ofthe year.

In reply to a post by MrSaffron:

I am not so sure

He clearly needs to buy a commercial grade Cisco and configure that.

Seem to be getting a smidge, just a smidge, of congestion here now; I guess my exchange felt left out.

I blame my campaign here. Signed up lots of people to FTTC quickly and tipped it over the edge.

In reply to a post by DaveTomlinson:

Yeah, all the BTW fibre customers on these exchanges will use the same SVLANs:

CUCKOO OAK
DAWLEY
DONNINGTON
HOLLINSWOOD
NEWPORT, SHROPSHIRE
OAKENGATES
SHIFNAL
STIRCHLEY
TELFORD WELLINGTON

There's currently 6 SVLANs at that exchange.

There is more on the way for this year Dave. Shifnal, Broseley, Much Wenlock, Albrighton and Pattingham in Shropshire for all FTTC roll out phase for this year 2014. So, it will be totally of 11 SVLANs

Edited by adslmax (Thu 20-Mar-14 00:20:46)

In reply to a post by DaveTomlinson:

Hi,

Those numbers are out of date now for FTTC/FTTP. The new numbers are:

On 40/2 and 40/10 Standard it's now 20Mbps throughput
On 40/2 and 40/10 Elevated it's now 30Mbps throughput
On 80/20 Standard it's now 40Mbps throughput
On 80/20 Elevated it's now 60Mbps throughput

The Service Level Agreement (SLA) is that users should achieve the above throughputs for 90% of the busy 3 hour period.

There's a percentage calculation if your line doesn't sync at the max rate for the product.

Very interesting over SLA (quick fixed times by BTw) and throughput minimum accepted range of 60Mbps on the 80/20 instead of 40Mbps.

As mine is range within 40Mbps - For your connection, the acceptable range of speeds is 40 Mbps-77.42Mbps as mine is standard 80/20 (without SLA)

Edited by adslmax (Thu 20-Mar-14 03:15:25)

Your BQM link seems to have a problem

Fixed now

thanks

As the exchange is so small the backhaul is likely to be physically limited maybe only 34Mb.

What i don't understand before December 2013 everything was fine and dandy, TBB speedtest (x1 and x6) both showed full line speed, iplayer streaming in evenings could be carried out HD, it now struggles with SD (very poor quality).

20c congestion

Zen support report no problems, usertools plusnet http://usertools.plus.net/exchanges/?exchange=Rhos&e... noproblems.

Haverfordwest now shows as Mar 14

I read that as March 14th not March 2014, yes seen some fibre cab twins being installed, and openreach engineer said should be enabled by July. (Start date is March 2014)

Ian

The mapping isn't one child exchange to one SVLAN. Each new exchange doesn't get an SVLAN of its own.

down to 9mbit/sec on single threaded transfers, I know everyone here is blaming BTw, but I didnt have these issues on infinity.

not seeing increases in latency on tbb graph but is packetloss.

Edited by Chrysalis (Thu 20-Mar-14 20:56:12)

I am of course in the rather fortunate position of being able to blame BTW as I have 2 ISPs on the go.

http://community.plus.net/forum/index.php/topic,1247...

yeah seen that post after I did mine, thanks for the input.

my new line is on hold as it turns out aaisp dont do talktalk backhaul on fttc yet, but they planning it.

Edited by Chrysalis (Thu 20-Mar-14 21:28:48)

In reply to a post by Chrysalis:

I know everyone here is blaming BTw

And they've come as close to admitting it as they're ever likely to- scroll down to today's update at the end of this news item.

booted of ppp again, twice in 30 mins.

Looks pretty clear-cut.

BT

Plusnet

One thing I find significant in that statement from BTW is that nowhere do they hint that they monitor network congestion etc in any way. Indeed, the final sentence:

In the meantime, we would urge communications providers to contact us should they have any concerns around bandwidth and network performance.

rather indicates that they wait for someone to complain.

Nor do they say that they'll take any notice, but we knew that already

In reply to a post by billford:

One thing I find significant in that statement from BTW is that nowhere do they hint that they monitor network congestion etc in any way.

They did, but it appears they don't now, or if they do no-one's paying attention.

http://revk.www.me.uk/2014/02/bt-21cn-not-fit-for-pu...

They seem to have run their networks too hot, and when extra capacity was required and was more complex than just a configuration change they were stuck.

The real killer has been where capacity requires additional hardware. Some upgrades require additional fibres, others additional line cards or even entirely new chassis.

They've made the same kind of mistake Virgin Media have at times. Not monitoring when a network segment is completely full with hardware resources exhausted.

how can a multi billion pound company simply not monitor a network, that is hard to believe and I believe a smokescreen. Would you deploy millions of pounds worth of infrastructure and then simply not monitor it? Although its possible some parts of it may have been missed on any monitoring.

The tone of revk's blog suggests to me there was a policy change within BT where a decision was made to simply stop spending money on it (seems timed with CEO change).

Am sure voices would have been saying upgrades needed within BT, but other voices that control purchase budgets may have had a voice to input.

Imagine this conversation....

Techie we need to upgrade capacity based on projections
Budget holder when will we hit capacity issues?
Techie middle of march probably.
Budget holder so maybe we can hold off of spending it from this current financial year budget
Techie I would not advise that
Budget holder if it becomes an actual issue we will expedite it.

It may be the projection was for trouble in May/June but an unexpected spike in demand for FTTC has caused usage to rise rapidly, e.g. successful campaigns and as councils talk about the expansion others across the county suddenly find out they can already order.

Or put another way, I ordered my FTTC within days of cab being live, but due to booking engineer install (to fix external wiring) was a few weeks behind the curve and a good number of people beat me to put the first FTTC flag on the map.
.

that sort of thing can happen if you run networks hot eg. if you consider 90% normal rather than trying to keep down to something like 50% utilisation.

I get the impression from revk's blog that BT have changed their attitude for a while now, just he didnt publish it until months after, so what we seeing now is probably after several months of ignoring issues building up.

I dont think there is a sudden spike of demand that happens to be coincidental all around the country.

Edited by Chrysalis (Fri 21-Mar-14 14:49:43)

apparently the loss I only had at peak which gradually appeared then went away at end of peak isnt conclusive.

so nothing escalated.

crazy.

AAISP issued the following service status today (https://status.aa.net.uk/1907):

We have started to see yet more congestion on BT lines last night. This looks again a bit like a link aggregation issue (where one leg of a multiple link trunk within BT is full). The patten is not as obvious this time. Looking at the history we can see that some of the affected lines have had slight loss in the evenings. We did not spot this with our tools because of the rather odd pattern. Obviously we are trying to get this sorted with BT, but we are pleased to confirm that BT are actually providing more data now that shows where each circuit will use network components within their network. We plan to integrate this soon so that we can correlate some of these newer congestion issues and point BT in the right direction more quickly.

Why do I get the feeling BT Wholesale are effectively unable to monitor congestion in their network, so they are providing more information to their customers who do seem able to identify congestion? It could be the case of 'we don't know what we are doing, so we will let someone else help us'.

Edited by deleted (Sat 22-Mar-14 18:38:18)

as I said you dont build a multi million (billion) pound network and then dont monitor it, thats bonkers.

Its the playing dumb game pretending they dont know about it.

Edited by Chrysalis (Sun 23-Mar-14 00:30:35)

In reply to a post by AndyHCZ:

Why do I get the feeling BT Wholesale are effectively unable to monitor congestion in their network, so they are providing more information to their customers who do seem able to identify congestion? It could be the case of 'we don't know what we are doing, so we will let someone else help us'.

It could be that, or it could be that it's simply cheaper to let someone else do it and only react when there's too many complaints to ignore.

Let's face it, the vast majority of broadband customers don't know about tbb, the BQM, what packet loss and congestion (and their symptoms) are, and simply think it's normal.

In reply to a post by billford:

It could be that, or it could be that it's simply cheaper to let someone else do it and only react when there's too many complaints to ignore.

Let's face it, the vast majority of broadband customers don't know about tbb, the BQM, what packet loss and congestion (and their symptoms) are, and simply think it's normal.

This. BT Wholesale are fine with congestion up to a point and in the same manner as Virgin Media appear to like to sit on things until people shout.

To be honest that's a mad approach given, unlike Virgin Media, they make their money by selling bandwidth and make less money if their network isn't delivering. If congestion means they are only sending say 30Gb/s to an ISP instead of 33Gbps they are missing out on 3Gbps of revenue. ISP won't purchase extra if they don't see the need to from current traffic levels.

Edited by deleted (Sun 23-Mar-14 12:57:03)

In reply to a post by Ignitionnet:

BT Wholesale are fine with congestion up to a point

It's one hell of a point if they consider 3% packet loss to be acceptable

In reply to a post by billford:

It's one hell of a point if they consider 3% packet loss to be acceptable

Must be a hang-over from being owned by the GPO.

What I didn't realise, is that virtually the same BT Wholesale statement (issued to ThinkBroadband) was issued to ISPReview nearly a month ago!

http://www.ispreview.co.uk/index.php/2014/02/bt-comm...

So that raises even more question marks because things have got progressively worse in the past two weeks and new exchanges are now impacted. Some Plusnet users have been reporting problems for 2-3 weeks and they are still waiting for an update from BT Wholesale!

In reply to a post by AndyHCZ:

"there are no BT exchanges at all that have any congestion".

Perhaps there is a new term for congestion. Maybe it has been re-defined as "High popularity".

I wasn't aware of that either, but I didn't exactly read it with open-mouthed surprise.

Maybe its a plan to say that they need to charge more per Mbps so they can afford more spare headroom for peak times....

Sounds like they've been taking lessons from Comcast.

In reply to a post by billford:

Sounds like they've been taking lessons from Comcast.

OMG, I hope not. I have friends on Comcast :-/

Congestion in a wholesale network is very poor, as the end user will always assume its their ISP (CP in openreach talk).

In reply to a post by jchamier:

Congestion in a wholesale network is very poor, as the end user will always assume its their ISP (CP in openreach talk).

I've little doubt that BTW are well aware of that.

In reply to a post by MrSaffron:

Maybe its a plan to say that they need to charge more per Mbps so they can afford more spare headroom for peak times....

No way, I wouldn't trust BTW on that one! Never pay BTW more money as they already pre-profits themselves!

In reply to a post by billford:

I've little doubt that BTW are well aware of that.

Hopefully its as the statement in Andrews (updated) article(s) that I've just re-read; BTW hope to have their various issues sorted by next month.

In reply to a post by jchamier:

Hopefully its as the statement in Andrews (updated) article(s) that I've just re-read; BTW hope to have their various issues sorted by next month.

I think that refers to their capacity to take new ADSL connections, not fix their backhaul.

There is no way BTw/TSO are going to be able to fix the ever increasing list of exchanges that have capacity problems in the next few weeks. Some of them will require new fibre links, which is going to take a good 2-3 weeks.

I expect the word "congestion" is something we are going to hear more and more over the next few weeks.

In reply to a post by jchamier:

BTW hope to have their various issues sorted by next month.

Maybe... my really horrible evening congestion and packet loss¹ disappeared like magic last Tuesday, and tonight there are indications of it returning

I'll wait and see.



¹ I recorded BTW speedtests of <4Mbps on an 80/20 FTTC line...

All very very slow tonight on fibre here too. http://www.thinkbroadband.com/speedtest/results.html...

Edited by adslmax (Sun 23-Mar-14 21:18:49)

yes, if you selling bandwidth in a manner that usage doesnt affect revenue then VM's model is more profitable.

But the current issues conflict with how BTw makes money.

On plusnet forums the problems are shown to be very severe tonight.

I had to wait 40 mnutes for a IE update to download, then firther windows update is monitored coming down at 30kB/sec thats "half" the speed of 0.5mbit adsl was back in the year 2000. Make no mistake this is very severe congestion.

I have no doubt BT are aware of the problems before they reported and rather they are deliberatly waiting for complaints before acting. Hopefully They dont take 6+ months to fix like VM does.

How can a isp that charges approx 10x the going rate of bandwidth have congestion.

Edited by Chrysalis (Sun 23-Mar-14 22:03:57)

incidently this gives anyone affected a get out of jail opt out of their contract, its a clear as a sunny day unfit for purpose.

bin slow for me like this last few weeks on EE which i think is a BT Wholesale line?

http://www.thinkbroadband.com/speedtest/button/13956...

http://www.thinkbroadband.com/ping/share/494634eeef4...

it's not affecting gaming yet but some websites are slow to respond and got a couple of page load fails on google search today

I made a post now on plusnet's forums explaining why the 3% been acceptable is so stupid.

in short even packet loss barely above 0% can quite easily affect throughput significantly.

There are two types of fault process. There's one for a single customer fault and one for a common fault affecting multiple customers. The data we've been looking with the various threads has been to try and pinpoint common faults (such as the Nuneaton area, Burton on Trent and Stepney Green area cases on our page and others and the forums). We can see the data on your TBB graph but we can't correlate it to a common fault. With the other exchanges it was really clear because we can see say 5% packet loss at peak across all the customers on a particular SVLAN or group of SVLANs but for your line we can't see that correlation, the shared network level views weren't showing anyone else seeing the same. Apologies if that wasn't clear, Chris is still working on it and we are trying to find a link if we can.

In reply to a post by Chrysalis:

I dont think there is a sudden spike of demand that happens to be coincidental all around the country.

You'd be surprised. We saw a nearly 10% increase in traffic between 8pm and 9pm this Sunday compared to the same time last Sunday which we think was down to just 2 things, El Clasico and no Top Gear.

In reply to a post by DaveTomlinson:

You'd be surprised. We saw a nearly 10% increase in traffic between 8pm and 9pm this Sunday compared to the same time last Sunday which we think was down to just 2 things, El Clasico and no Top Gear.

Wait until 4K streaming come out!

So when you roll out TV on IP, the network will be really stretched then?

Yep, we've already got it on the radar. Netflix and others are already starting to make shows in 4k and I'm sure there's plenty of content that can be converted, not forgetting all the sport. It's what your 80Mbps was made for

Nope, we saw a 10% increase in demand week on week but we still had plenty of headroom available as it was still within our forecast and we budget and plan for demand well ahead of the forecast. We've just done a massive investment in the size of the network with the first part of the new kit (Juniper MX series) coming online later this week giving us considerably more capacity.

Every part of the network has been scaled upwards over the last few months because we know that selling unlimited products and people buying Netflix and Lovefilm and using iPlayer and Sky Go/On Demand will like it, go back for more. Same thing applies today with our TV trial and would apply with any Plusnet TV product. People aren't going to recommend Plusnet if iPlayer on YouView buffers or they only get the standard def video to work on Netflix.

ok understood thanks.

I see as well you have a found a new issue that might be the cause of my congestion, so hopefully thats the cause.

My connection was fine from Wednesday until Saturday, BT managed a whole 4 days� then on Sunday it was nearly back to where it was before, albeit with less severe packet loss.

Do BTW have any sort of clue at all?

Looks like I might be bringing you a fault that's kinda similar but something a bit different, sorry Dave

I was talking about the product mentioned in this blog http://www.timico.co.uk/blog/2013/03/20/ethernet-ove... (it's available from a number of suppliers no just Timico) - my understanding is that delivery from exchange to CP is no different to FTTC or FTTP but the data doesn't travel from exchange to ISP over a Virtual Path.

Thanks, where do Openreach publish that data?

Yes VDSL uses PTM with much lower overhead, I was quoting the original text verbatim.

Yes, I didn't like that blog entry when I first read it last year. That penultimate paragraph is very badly worded, and is what I expect caused you to post what you did. Bog standard FTTC (and in fact FTTP as well) is/are provided using Openreach GEA. See Openreach SIN 498.

The product you mention must be something to do with the backhaul method, as you say. He didn't make that clear.

Indeed, the use of 'GEA' doesn't appear to be at all consistent.

SIN 498 is interesting, in section 1.2:

The GEA-FTTC product will offer the following VDSL2 line rate:
� A peak downstream (from CP to EU) rate of up to 80 Mbit/s,
� A downstream prioritisation rate of 15 Mbit/s or 30 Mbit/s (depening on the product variant purchased) or line rate, whichever is the lower

Does prioritisation imply potential packet loss above these rates?

it could mean one of many things.

its entirely possible there is no prioritisation and those are just fancy words for a guideline to be used for when BTw accept low speed faults.

with the recent BTw congestion throughput was way below those quoted speeds, so packetloss can certianly occur below those speeds, and likewise its normal for no packetloss at all above those speeds.

If there is prioritisation I expect its just higher weighting of packets for the higher product alongside the higher fault threshold.