Loss of Service (LOS) on FTTP

After having ~ 5 months of a rock solid FTTP service, I've noticed that the dreaded red LOS light has appeared on my ONT and the service has been down all morning:

https://s33.postimg.org/ejz4wxqgv/ont.jpg

I've already informed my ISP who said they will get Openreach to investigate ASAP.

Am i right in thinking the fault on the PON will be somewhere between the exchange and the ONT? Can an OR Engineer accidently disconnect my line at the splitter or agg node whilst doing some other work?

Could be and ONT failure, or a Tech could have accidentally damaged/disconnected, or the fibre could have been damaged by a third party - road works, branch hitting if overhead, ... plenty of reasons and once OR get to look they will soon know where it has occurred.

Nothing was on the lunchtime update from Openreach for new FTTP faults.

Is LOS solid red, blinking slowly or rapidly? The same for the PON, or is that off completely?

Oh dear .... that is unusual.

Am i right in thinking the fault on the PON will be somewhere between the exchange and the ONT? Can an OR Engineer accidently disconnect my line at the splitter or agg node whilst doing some other work?

Correct, except the accidental disconnect *shouldn't* happen.

Could be anywhere, if lucky the engineer who attends will have an OTDR to measure to the fault (not many carry them, in fact no one on 'my side' has one, so leg work required.

Have a quick squint at the CSP outside, nothing obvious there is there ?

In reply to a post by AndyHCZ:

Nothing was on the lunchtime update from Openreach for new FTTP faults.

Is LOS solid red, blinking slowly or rapidly? The same for the PON, or is that off completely?

It's a straight red PON Andy, look at the photo BF posted.

Yeah, I just wanted to check if any of the lights are flashing every 0.5 sec or 2 sec as you can tell if the ONT is faulty from this.

I believe on the 4 port Huawei, if the LOS is sold red then PON should intermittently flash on/off every 0.5 sec.

I believe on the 4 port Huawei, if the LOS is sold red then PON should intermittently flash on/off every 0.5 sec.

Never witnessed that .... Never yet met a faulty one either.

Thanks. PON light is completely off and LOS is solid red. Fluidone have arranged for a SFI visit as I guess OR haven�t found anything wrong at their end. CSP outside looks ok so I suspect the ONT has gone tits up

I guess OR haven�t found anything wrong at their end.

Except it should test operationally down, so the remote test doesn't see your ONTE's serial number on the head end in the exchange.

And one would hope the present of the upstream light signal being present or not would be noticed

It's a bit of a weird problem as I don't think the lights are doing what they should.

When I had a fault at the exchange, PON flashed as it tried to connect and then went off.

Have you tried disconnecting the backup unit and leaving the modem off for a few mins, then trying again to see if it connects?

In reply to a post by AndyHCZ:

Have you tried disconnecting the backup unit and leaving the modem off for a few mins, then trying again to see if it connects?

I don�t have a bbu installed (no fva) and yes left ONT off for an hour or so before switching on again but still getting solid red LOS light. SFI visit is booked in for Monday am but I�ll keep rebooting the ONT at regular intervals as I don�t want to be hit with SFI charges in case the ONT magically starts working on Monday am.

Nah. What BF�s is showing is a straight disconnection on the fibre I reckon.

The flashing you are describing is when the ONTE is out of kilter with the head end, or (with the intermittent pause in flashing) is hitting the wrong SASA tray in the exchange.

Yep, but my experience of �one way transmission� between the exch and ONTE would lead me to suggest the PON light on the ONTE would be flashing if that was the case.

Monday?

I'd have thought your money would merit a faster turnaround than that. Maybe not the 6 hour one, but surely a 12 hour, 6 days a week response.

Forum member burakkucat has kindly sent me a scanned copy of the Openreach ONT (Huawei HG8240) user guide which clearly explains what the various LEDs on the ONT mean:

https://s7.postimg.org/legnljluz/page10.jpg

So it appears the ONT is receiving a PON signal but its too weak? As a last resort is it worth doing a hard reset on the ONT?

If anyone wants a peek at the ONT manual (thanks again burakkucat )

https://ufile.io/1sc0v

Edited by deleted (Fri 17-Nov-17 15:20:05)

In reply to a post by WWWombat:

Monday?

I'd have thought your money would merit a faster turnaround than that. Maybe not the 6 hour one, but surely a 12 hour, 6 days a week response.

Perhaps if I had a leased line then it would come with the necessary SLA (eg fix within 24hrs) but I think FoD once installed is treated the same as native FTTP wrt fix times. I can't complain too much though as the Engineer is booked in on the next working day (Mon). I get a decent 4G EE signal indoors so will tether my phone for internet access for a few days.

Nah, straight red means no light .... signal too weak would result in a flashing PON light.

Which can be borne out by unplugging the lead (when the service is working) the resultant red light is clearly not the result of a too weak signal .....*







*DO NOT UNPLUG the lead, you�ve nothing to clean the end with prior to putting it back. As the engineer who visits to demonstrate.

Edited by Zarjaz (Fri 17-Nov-17 15:33:57)

In reply to a post by Zarjaz:

Nah, straight red means no light .... signal too weak would result in a flashing PON light.

Cheers, in that case assuming the ONT isn't faulty the only thing i can think of is that the fibre splice inside the CSP has been disturbed/damaged. We had gale force winds overnight for a short while (North Scotland) so perhaps that's dislodged the CSP's guts.

We had gale force winds overnight for a short while (North Scotland) so perhaps that's dislodged the CSP's guts.

Is any of the feed overhead/through trees ?


What's inside the CSP is fairly secure, the splice itself clips into place...... https://www.dropbox.com/sh/bz5vlj70n92z9m5/AACWDXMoM...

In reply to a post by Zarjaz:

We had gale force winds overnight for a short while (North Scotland) so perhaps that's dislodged the CSP's guts.

Is any of the feed overhead/through trees ?


What's inside the CSP is fairly secure, the splice itself clips into place...... https://www.dropbox.com/sh/bz5vlj70n92z9m5/AACWDXMoM...

Nup, everything goes u/g in ducts. The white EZ-bend cable from the ONT to the CSP remains intact so I'm sure internal cabling is all fine. I guess the mystery will be solved on Monday.

Post deleted by MrSaffron

In reply to a post by bellamob:

... But FTTP loss of service rather than most reliable FTTC is a shocking to be honest ...

What would that be in English?

I thought adslmax was bannneded

Slightly OT, but I'm not sure what indication I would get in the event of loss of service as my ONT doesn't have 'PON' or 'LOS' indicators, just a single 'OPTICAL' LED.

Presumably it would be either solid or flashing red according to the type of faults discussed in this thread.

Good luck for a speedy repair.

Indeeded!

Post removed as user is on suspension for a number of months yet

Might be a dumb question but why don't engineers have easy access to OTDRs?

In reply to a post by Spud2003:

Might be a dumb question but why don't engineers have easy access to OTDRs?

Not enough FTTP deployed to have enough faults to make it worthwhile.

Nope, I think it�s because they are expensive.

In reply to a post by Zarjaz:

Nope, I think it�s because they are expensive.

Indeed. So due to that and the relatively small scale of the rollout no great drive for them. I sure they will be more available when 5 million premises rather than 500k are deployed.

In reply to a post by Rastus:

Slightly OT, but I'm not sure what indication I would get in the event of loss of service as my ONT doesn't have 'PON' or 'LOS' indicators, just a single 'OPTICAL' LED.

That looks like an ECI ONT. Not having had any experience of that particular device, I am unable to answer.

I wonder if Zarjaz can assist?

In reply to a post by Zarjaz:

Nah, straight red means no light .... signal too weak would result in a flashing PON light.

Err but yes but no! Both you and the manual are correct.

We must remember that the manual was either written (or translated) by someone whose first language is not English.

There could be measurable light at the connector but if the intensity is below the minimum threshold for which the HG8240 recognises the light's presence, it will show a solid red LED. It is that particular condition which the Quick Start Guide's author has documented.

In the case when the HG8240 detects the presence of light but the intensity is below that required for service operation, then it will show a flashing red LED.

In reply to a post by baby_frogmella:

Perhaps if I had a leased line then it would come with the necessary SLA (eg fix within 24hrs) but I think FoD once installed is treated the same as native FTTP wrt fix times. I can't complain too much though as the Engineer is booked in on the next working day (Mon). I get a decent 4G EE signal indoors so will tether my phone for internet access for a few days.

Is there a "standard" SLA for "native FTTP" fix times?

For copper, "native WLR" comes with 4 different service maintenance levels, any of which could be chosen by the CP for a copper phone line service (2-days M-F; 1-day M-Sat; 12 hrs M-Sun; 6 hrs, M-Sun). It isn't obvious whether a broadband service added on top acquires the same SLA level in the event of speed issues.

As I understand it, BT Retail defaults to the "1-day, M-Sat" service level now.

I also understand that having FTTC on a line sets the minimum service level to "1-day, M-Sat" too, even if the CP chose less for the phone service.

I don't know what service levels are available for FTTP, but I'd have assumed the same set, and that a CP can choose any of the SLAs for the service they are choosing to sell.

If FTTP follows the same principle as FTTC on copper, then I'd have expected a repair on Saturday under the "1-day M-Sat" service at minimum.

But if I were paying £300pm for a service on FTTP, which everyone is treating as a business service, I would have expected a better response.

I'm also surprised that the fault is being treated as SFI. Even though you only have broadband, surely a PON that cannot connect is as broken as copper could be.

We had a FTTP LOS issue that effected approx 10 houses the other week. Took about 5 days until service was restored to all properties.

One thing I did notice is that even if the issue is external on the Openreach network, your ISP will still say that an Engineer visit to the property is required, although to the customer it would look like they did not turn up as they did not actual need to visit the properties in question. Maybe something that needs to be looked at when solving FTTP network issues.

TBH I don't know the ins & outs of the SLA which applies to my FTTP line - though i'm sure its buried there in the contract t&c's somewhere. All I know is that within 2 hrs of reporting the fault to FluidOne they had arranged for a SFI Engineer visit on Monday - who knows, it may have been on Saturday if an OR Engineer was available.But like i said i have a backup service (4G) in place so its not the end of the world being a few days without FTTP.

My gut feeling is that either the ONT is knackered or the splice inside the CSP has been disturbed (even though its supposed to be secure). The fibre runs all the way to the agg node in relatively new u/g ducting and I haven't seen any heavy machinery digging near the route so unlikely to be a cable break.

Re: SFI visit, I've seen this IDNet article which gives a breakdown of SFI charges in case the customer is liable (ie fault lies with customers equipment). Not sure if its still valid though as article is 7 years old:

http://www.idnetters.co.uk/forums/index.php/topic,23...

Edited by deleted (Sat 18-Nov-17 21:17:14)

In reply to a post by cymru123:

One thing I did notice is that even if the issue is external on the Openreach network, your ISP will still say that an Engineer visit to the property is required, although to the customer it would look like they did not turn up as they did not actual need to visit the properties in question.

Good point, that ties in with what IDNet say about a SFI appointment:

If an engineer attends the exchange and identifies a fault he/she may not attend the premises and no charge will be applied.

Edited by deleted (Sat 18-Nov-17 21:26:37)

I�m not sure why they have termed this repair an SFI task, those are specific the ADSL product.

Considering it is a business contract, I would assume you would get some sort of Business SLA.

I will need to check Cerberus contract. I do know they do offer money back for downtime (this is written in the contract), just curious as time to repair fault.

In reply to a post by cymru123:

We had a FTTP LOS issue that effected approx 10 houses the other week. Took about 5 days until service was restored to all properties.

Ugh - that's bad.

In reply to a post by cymru123:

One thing I did notice is that even if the issue is external on the Openreach network, your ISP will still say that an Engineer visit to the property is required, although to the customer it would look like they did not turn up as they did not actual need to visit the properties in question. Maybe something that needs to be looked at when solving FTTP network issues.

I'm guessing, but copper can be tapped into anywhere there is a joint, so an engineer can perform tests at (say) the PCP and DP at a minimum. He can insert an oscillator, and trace the signal.

For FTTP, I guess it is a much harder job to trace without breaking into the fibre path. Access to the house/business might be needed just to check that the light is reaching that far.

GPON wavelengths aren't visible, so can the OLT port be instructed to generate something visible, that an engineer can actually see?

For the most part, FTTP faults are going to be at the hardware though (ONT/OLT) and Openreach can see this remotely.

When there is a loss of signal at multiple properties (and no alarm bells from monitoring), you need to check if they are on the same DP/splitter/agg node.

Talking to an engineer it's a PITA to find a problem with a fibre path. I believe it's the PTOs that have the kit to find the distance the problem is on the fibre path, but then you have to correspond that to the maps to track down where it is.

I've had 2 outages with my FTTP in 5 years. Both of them were at exchange level and I didn't get an engineer.

What the average Joe does is put a visible light source on at the ONT end (hence access required) and work back, CSP, DP, then the splitter.

�ID� engineers carry OTDR�s, and if the fault isn�t at one of the above, it�ll be passed to them.

Baby F reckons a faulty ONTE or CSP ... my guess is an issue at the splitter or exchange end.

Cos I, for one, am on tenterhooks.

Not seen one of them, there was a post earlier in the thread asking what brand the OP�s ONTE was, and guessing it was an ECI ..... I suspect yours IS an ECI version.

Zarjaz you were right. Openreach chap came earlier on, tried a new ONT but still got the solid red LOS light and then went outside to measure light signal at CSP...nothing, nada, zilch So he thinks there's a fibre break somewhere and right now is having a poke inside the DP. So the mystery continues!

One thing I noticed, when he re-connected the fibre to my ONT, he didn't clean the connector end with alcohol and when I asked him why not he said "naaaaah mate, not necessary to do it again". Surely he should have cleaned it?

Yes, he really should ... that kinda slip shod stuff pee�s me off.

Why did he try a different ONTE ??? Put yer light meter on the end of the lead in, no light end of ...

Doesn�t fill me with confidence TBF

Has he put a visible light source on at the property before he went off to poke about ?

He came back a short while ago and said the problem is at the exchange as he didn't get any light signal even at the last point (splitter/agg node) before the fibre heads off to the exchange. He said a separate team will do the necessary work at the exchange.

Also I think he realised his mistake cos when he came back he removed the fibre connector from the ONT again, cleaned it with his nifty gadget and re-connected it back to the ONT lol

I guess its 'sit tight and wait' time, hopefully OR fix this today.

Which exchange are you on?

In reply to a post by AndyHCZ:

Which exchange are you on?

Inverness Culloden (NSICL), however I understand my FTTP line terminates at the main exchange in town, Inverness Macdhui (NSIMD).

The 'last point' will have been the splitter.

He said a separate team will do the necessary work at the exchange.

That should be an ID engineer.

Also I think he realised his mistake cos when he came back he removed the fibre connector from the ONT again, cleaned it with his nifty gadget and re-connected it back to the ONT lol

That's something I guess.

http://forums.thinkbroadband.com/fibre/t/4573736-re-... .......... I should do this for a living

Out of curiosity what kind of exchange fault would cause a fttp line to suddenly stop working? I don�t think it can be a general (mass) outage as bt/or would fix this within hours, and I imagine BT�s automated systems would pick up such outages without having to rely on end users reporting it to the CPs. Perhaps someone�s unplugged my line at the OLT? ooops

Maybe someone tripped over the cable, wasn't sure where to plug it back in so just shoved it in the nearest available hole?

A broken SFP module in the OLT linecard? If you're the only one on the PON, then there'd be no mass outage.

The fact that's it's not on Openreach's outage report is a fairly good clue that it's only baby_frogmella that's been affected.

The only FTTP issue I can see is this one (and it's a very unusual case):

Ports down, Damaged fibre cable due to rat infestation at customers premises, NGA Ops trying to confirm with council whether the pest control company have now dealt with this to enable fault to be progressed

I�ve had EeZee bend lead in�s chewed through by rodents twice ...

Why, oh why did they make them cheese flavoured [sigh]

That, I�m sorry to say, I don�t know. But the trays where they get spliced through can be busy, so maybe so heavy handed oaf has been working nearby ?

No worries.

Latest update from my ISP

I have checked this with the carrier and they have advised that currently the fault is with the DCOE team as they are investigating the issue.

We have been advised to review back on 22/11/17 as the DCOE team requires 48 working hours for further testing, however, we will keep chasing and will keep you posted.

Who/what is the DCOE team?

Some posts on the plusnet forum make reference to DCOE as the Openreach
Diagnostics Centre of Excellence team.

Exactamundo Craski

Edited by Zarjaz (Mon 20-Nov-17 18:19:21)

But the DCoE are just an internal generic help desk with various sub sections for different parts of the business ...... So one doesn�t �further� a task to them ... they advise, and in the case of the FTTP subdivision have access to the element manager a �tool� for looking into the guts of the exchange equipment .....

I wonder if, after co-op it�s gone on to the 2nd line support team, which would imply summat more �involved� being wrong ......

Smoke �em if ya got �em boys..... I don�t like stuff I can�t fix myself/know someone I can get to fix/or have at least spoken to and understood the next steps. (yes, a Control freak) can they give you any more info you might pass on ?

In reply to a post by baby_frogmella:

Also I think he realised his mistake cos when he came back he removed the fibre connector from the ONT again, cleaned it with his nifty gadget and re-connected it back to the ONT lol

He is probably reading this thread for ideas...........

They'll most likely send it out for a fibre engineer (PTO) to test the fibre fully between the headend and the splitter. If your lucky might be sorted this week

In reply to a post by baby_frogmella:

One thing I noticed, when he re-connected the fibre to my ONT, he didn't clean the connector end with alcohol and when I asked him why not he said "naaaaah mate, not necessary to do it again". Surely he should have cleaned it?

You only need to clean the end of a fibre link if it's dirty. You most certainly don't clean them every time you unplug them. It would be a nightmare otherwise. Now if the link is going to be unplugged for any length of time you can get little caps that go over the end of the cable to protect them from getting dirty. The worst are the little tiny plastic ones that go on LC connectors. They just fall off. The clip on ones are much better. For SC the best ones are the square rubbery plastic jobs that cover the whole end of the SC connector.

Rubbish.

When being taught to splice they showed us, using a microscope, how dirty these connectors get just from brief exposure to the air ....

what was that old phrase about not spoiling the ship for a ha�peth of tar ?

Had another visit from Openreach this morning. This time he appeared to be a 'proper' fibre specialist (ID Engineer?) and carried out all the light tests again. He seems to think the issue is 6 km away in town not far from the headend exchange (NSIMD) and has left his gadget plugged into my fibre whilst he's gone off to investigate. Hopefully this gets fixed today.

https://s8.postimg.org/4pfrq1eyt/Image.jpg

https://s8.postimg.org/l9dgc8m8l/Image2.jpg

Edited by deleted (Tue 21-Nov-17 12:55:08)

As you have a business services, what SLA is in your contract with FluidOne? Are there guaranteed resolution times and a compensation structure?

In reply to a post by ferretuk:

As you have a business services, what SLA is in your contract with FluidOne? Are there guaranteed resolution times and a compensation structure?

I'm on the standard FluidOne SLA which guarantees service uptime of 99.9% each month - i get an automated report each month with uptime/downtime and credits due, if any. If the 99.9% uptime isn't met, I get refunded (pro-rata) the number of days i'm without service. For any issues on their network, they aim to fix them within 5 hrs, however this doesn't apply to issues outside their control (eg Openreach network issues). They offer a better SLA (called SLA+) but this costs more and I'm not sure if they could get OR to fix faults faster on this.

In reply to a post by Zarjaz:

Rubbish.

When being taught to splice they showed us, using a microscope, how dirty these connectors get just from brief exposure to the air ....

what was that old phrase about not spoiling the ship for a ha�peth of tar ?

Because I have been using fibre optic cabling now for over a decade in my day job. So I don't really care what they showed you on a microscope it's just not necessary to constantly clean the ends every time you unplug the cable provided you don't trail it along the floor. Fibre rarely goes bad and it's even more rarely down to dirty ends.

Fibre optic cable is also orders of magnitude more robust than people would have you believe too. The least robust is stuff with ST connectors on the end because well they are heavy lumps of metal that the action of gravity tends to exceed bend radius very quickly if you have the thinner jacketed cable. Thankfully ST connectors are on the way out, as are SC for that matter though for some reason GPON equipment seems to have standardised on it. I presume for increased robustness on simplex cables, LC connectors always seems a bit fragile to me.

Perhaps Openreach take the view that a few seconds of an engineers time to clean the end is better than a 1 in 10,000 (or any other arbitrary number) risk that a bit of dirt might have got in and will disrupt the signal requiring another engineer visit? I would assume it is about avoidance rather than it being an issue in every case.

WOO HOO happy days

Engineer spent nearly all day outside in the heavy rain trying to diagnose the fault...absolutely nothing physically wrong with the fibre. He then re-configured the port at the exchange and bingo! that brought back the connection to life. He thinks one of the exchange planners may have messed up the port settings last Friday hence why my line just went dead.

And thankfully speeds are what they should be
http://www.speedtest.net/my-result/6813234287

In reply to a post by ian72:

Perhaps Openreach take the view that a few seconds of an engineers time to clean the end is better than a 1 in 10,000 (or any other arbitrary number) risk that a bit of dirt might have got in and will disrupt the signal requiring another engineer visit? I would assume it is about avoidance rather than it being an issue in every case.

+1
I was taught the same 20 to 25 years ago, way be for all the gadgets BTOR get to use now days

So I would also spends a few seconds cleaning it when unplugged and plugged back in.

Paul

In reply to a post by baby_frogmella:

WOO HOO happy days

Engineer spent nearly all day outside in the heavy rain trying to diagnose the fault...absolutely nothing physically wrong with the fibre. He then re-configured the port at the exchange and bingo! that brought back the connection to life. He thinks one of the exchange planners may have messed up the port settings last Friday hence why my line just went dead.

And thankfully speeds are what they should be
http://www.speedtest.net/my-result/6813234287

Nice, now some other person has lost their fibre connection and has reported it to their ISP LOL.

Paul

Nice, now some other person has lost their fibre connection and has reported it to their ISP LOL.

Maybe OpenReach are setting up a new FTTP timeshare business. You pay lots of money but only get to use it for 2 weeks a year...

Nice one.
I'm sure it was an inconvenience it being off for a few days but on the positive side it will remind you just how awesome it is to have such a good connection!

Very pleased to hear its fixed .....

Not so pleased to hear that some twit had borked it in the first place.

In reply to a post by ian72:

Perhaps Openreach take the view that a few seconds of an engineers time to clean the end is better than a 1 in 10,000 (or any other arbitrary number) risk that a bit of dirt might have got in and will disrupt the signal requiring another engineer visit? I would assume it is about avoidance rather than it being an issue in every case.

+ 1

In reply to a post by ian72:

Perhaps Openreach take the view that a few seconds of an engineers time to clean the end is better than a 1 in 10,000 (or any other arbitrary number) risk that a bit of dirt might have got in and will disrupt the signal requiring another engineer visit? I would assume it is about avoidance rather than it being an issue in every case.

Not really, if there is a problem it will be immediately obvious due to lack of link when plugged in. Admittedly there was lack of link to begin with in this case, but that was not down to dirt on the connector. I can assure you that nobody who works with fibre patch leads on a daily basis cleans them every time they unplug them. If when plugging in you get link you are golden.

Fibre Optic Association?
Cisco?
LanShack
Arista?

They are the top 4 google results when searching for "should i clean fibre connection". It would seem the general guidance is to always clean fibre connectors before making or remaking a connection - and there appear to be many more.

So what if your 'it's OK not to clean that" connector is just one small part of many CBA connections , that adds up to issues in the future.

The original ADSL engineer installs were hugely 'over engineered' BUT you don't see those punters on here moaning about not getting the promised speed on their VDSL installations.

I know an FTTP only estate where we left going back for faults after installation, every single one due to a duff splice somewhere. Bumped into the installation gang and watched as the lead engineer told his two trainees " Look you don't have to clean with an alcohol wipe, it's good enough to use your fingers if they are dry" Root cause identified.