Huawei modem firmware breaking VPN ? (RevK blog)

This is an interesting finding, see the RevK's blog:
http://revk.www.me.uk/2013/11/bt-huawei-fttc-modem-b...

I don't appear to be having any issues with VPN since the new firmware was pushed. Maybe it's because I have Sky Fibre because looking at the blog it seems to be affecting PPPoE connections to the modem.

Me neither, and I use a UDP based VPN.
http://www-03.ibm.com/software/products/us/en/mobile...

In reply to a post by simon194:

I don't appear to be having any issues with VPN since the new firmware was pushed. Maybe it's because I have Sky Fibre because looking at the blog it seems to be affecting PPPoE connections to the modem.

Yep; if Sky Fibre is still using MER then it's immune to this issue.

Reading that blog, will all BT vdsl modem get the firmware update, or just people using BT infinity? I have a BT vdsl modem installed by BT but my ISP is Entanet - will that get a forced updated?

Nick

Fortunately I'll be immune as my work VPN uses GRE, however those who use IPSEC-based VPNs have potential for issues.

In reply to a post by Lethe:

Reading that blog, will all BT vdsl modem get the firmware update, or just people using BT infinity? I have a BT vdsl modem installed by BT but my ISP is Entanet - will that get a forced updated?

Nick

The modems are managed by Openreach, not BT Retail, who supply BT Infinity.

All Huawei VDSL modems will get the update.

In reply to a post by Ignitionnet:

In reply to a post by Lethe:

Reading that blog, will all BT vdsl modem get the firmware update, or just people using BT infinity? I have a BT vdsl modem installed by BT but my ISP is Entanet - will that get a forced updated?

Nick

The modems are managed by Openreach, not BT Retail, who supply BT Infinity.

All Huawei VDSL modems will get the update.

Yes, sorry, I meant Openreach (but it's all BT to me )

OK, since I had fttc 15 days ago line has been stable except looking at my router logs I got a disconnect/reconnect at 02:05 Monday morning - of course, I dunno if it was the vdsl modem rebooting (it's a black box as you know) but just a PPPoE log, so I wonder if that is what it was.

Thanks for reply.

Nick

In reply to a post by Ignitionnet:

In reply to a post by Lethe:

Reading that blog, will all BT vdsl modem get the firmware update, or just people using BT infinity? I have a BT vdsl modem installed by BT but my ISP is Entanet - will that get a forced updated?

Nick

The modems are managed by Openreach, not BT Retail, who supply BT Infinity.

All Huawei VDSL modems will get the update.

The ISPs can upgrade but OR have the final ruling.

Ryan, can you explain that a bit more? What does OR have the final ruling on, exactly?

In reply to a post by Lethe:

Yes, sorry, I meant Openreach (but it's all BT to me )

That's the problem Ofcom has created. Its horrible naming; but you'll find most regular posters on these forums will correct posters as its vitally important to separate the two.

OK, since I had fttc 15 days ago line has been stable except looking at my router logs I got a disconnect/reconnect at 02:05 Monday morning - of course, I dunno if it was the vdsl modem rebooting (it's a black box as you know) but just a PPPoE log, so I wonder if that is what it was.

2am approx is when Openreach can do exchange work, and it may have just been a session drop/recycle.

In reply to a post by BatBoy:

Ryan, can you explain that a bit more? What does OR have the final ruling on, exactly?

I don't think that makes sense, as the ISP generally doesn't see the OR modem at all, they only see the PPPoE termination device which is usually the consumer router. (may be ISP supplied).

2am approx is when Openreach can do exchange work Probably true. It is also a common time for DLM to force a non-emergency cab-modem resync; eg if it has finally decided you can have a little more speed.

In reply to a post by ryant704:

The ISPs can upgrade but OR have the final ruling.

The ISPs have no access to the OR modems. Their management interfaces are only reachable from Openreach's systems and there are no routes for them to go elsewhere.

Interesting.

Thanks guys and gals.

Nick

The ISPs can push firmware updates but if OR decide they don't want to allow that update they have the right to refuse it.

It's in the BT SIN 498.

Edited by deleted (Wed 06-Nov-13 22:01:28)

I think you're referring to this bit

CPs or their EUs will be responsible for maintaining the firmware of their modems and monitoring their connectivity and performance, typically via a TR-069 interface using CPE WAN Management Protocol (CWMP).

The CP provided modem and filtering devices must meet the requirements of this specification in order to provide reliable operation and to avoid harm to other VDSL2 lines sharing the same cable binder.

Openreach reserves the right to withhold or limit service where potential violation of the Access Network Frequency Plan (ANFP) or impact to an other customers� service is detected

In reply to a post by ryant704:

The ISPs can push firmware updates but if OR decide they don't want to allow that update they have the right to refuse it.

It's in the BT SIN 498.

I think you're thinking here about the sections of SIN 498 about the yet to be launched "CP Provided Modem Product Variant" - i.e. FTTC without the Openreach modem.


The situation for the Openreach modems is reported in section 1.2.1 of SIN 498: "There will be occasional firmware upgrades which will involve reasonable volumes of traffic (M bytes). Openreach will report to CPs when these are scheduled across the GEA network."

In other words, it's at Openreach's discretion when to roll out firmware upgrades to the modems, and the ISPs are told when these upgrades are happening.

Doesn't that section now apply with the advent of the Homehub 5?

In reply to a post by BatBoy:

Doesn't that section now apply with the advent of the Homehub 5?

BT Retail and EE have started supplying equipment with integrated VDSL2 modems, but, outside any ongoing pilots, FTTC remains solely in its "modem and full engineer install" variant at present. As such, Openreach engineers should always be installing the modems and connecting to the router's Ethernet WAN port, and no non-Openreach equipment should be connected, including hacked Openreach modems.

In practice, engineers may well dispense with the modem if presented with an ISP supplied integrated VDSL2 router, and quite a few customers have hacked their modem or connected their own privately bought VDSL2 equipment. Theoretically, Openreach could find out and disable these customers' DSLAM ports, but there's been no reports here of them doing so.


It remains to be seen whether Openreach's policies change once CP supplied modem FTTC launches. It is possible Openreach will start monitoring connected equipment and turning off ports connected to equipment other than Openreach modems with the latest firmware and ISP supplied equipment approved by Openreach that is on an approved firmware version.

Though we've been talking about the upcoming new variants loosely as wires only FTTC, at least so far as the SIN goes, the policy appears to be that only Openreach approved equipment supplied by ISPs is allowed, and that the normal pattern will be a degree of remote management by the ISP (the first two paragraphs of section 2.4 of SIN 498 refer).

In particular, Openreach may toughen up their policies on disabling ports if they launch vectoring, with any failure by the connected equipment to engage in vectoring correctly leading to the DSLAM port being disabled.


Unfortunately, despite the mention in section 2.4 of SIN 498, there's no reference I can find to the terms for CP supplied modem FTTC on those parts of the Openreach web site that can be accessed by members of the public, other than the very limited information on the CPE Enablement page. For ISPs to be launching integrated VDSL2 routers, the process must have moved on from the situation on that page, and I suspect a full launch of CP supplied modem FTTC is a few months away at most. I thought I saw January being suggested somewhere on the forums, but until the service is announced, it's not official.

In reply to a post by David_W:

In practice, engineers may well dispense with the modem if presented with an ISP supplied integrated VDSL2 router, and quite a few customers have hacked their modem or connected their own privately bought VDSL2 equipment. Theoretically, Openreach could find out and disable these customers' DSLAM ports, but there's been no reports here of them doing so.

It appears that part of the recent update re-locked those BTOR modems which had been unlocked by people.
(and removed the webserver GUI within the modem)
Whether this removing users access to the stats was deliberate or whether it was some side effect of whatever the update was really all about is unknown.

There are also some reports on forums I've seen of 3rd parties VDSL combined units performing worse on sync etc speeds when compared to the data from an unlocked BTOR modem.

I agree with you about the necessity for vectoring to work for all the modems to be working correctly - and the necessity to disconnect those which are interfering with the vectoring process to the detriment of nearby lines.
I personally believe that the refusal by BTOR to give out control of interleaving to ISP's is part of this. They do not want external parties having the ability to unilaterally change the conditions for one line - possibly at the expense of a worse performance then being achieved on adjacent lines in the cable bundle.

Edited by deleted (Thu 07-Nov-13 08:34:06)

In reply to a post by zom22:

I personally believe that the refusal by BTOR to give out control of interleaving to ISP's is part of this. They do not want external parties having the ability to unilaterally change the conditions for one line - possibly at the expense of a worse performance then being achieved on adjacent lines in the cable bundle.

Activation or otherwise of interleaving won't in itself change the RF conditions on a particular line. Vectoring doesn't care about whether a line is interleaved or not, that's higher up in the protocol stack than it's measuring performance.

thats the point I been trying to make also, thank you for agreeing with me,

I respect DLM does more than just adjust interleaving and banding, as I have seen numerous power mask changes on my line, one happened the mornign after I turned tr069 back on. (a negative change was a further cutback).

But as you said interleaving and banding wont affect crosstalk.

Edited by Chrysalis (Thu 07-Nov-13 20:53:18)

We've updated the Status post with some more details of the problem we are seeing.

"
Here is an update with some a more specific explanation as to what the problem we are seeing is:

On WBC FTTC, we can send a UDP packet inside the PPP and then drop the PPP a few seconds later. After the PPP re-establishes, UDP packets with the same source and destination IP and ports won't pass, the do not reach the LNS at the ISP.

Further to that, it's not just one src+dst IP and port tuple which is affected. We can send 254 UDP packets using different src+dest ports before dr weop the PPP. After it comes back up, all 254 port combinations will fail. It is worth noting here that this cannot be reproduced on an FTTC service which allocates a dynamic IP which changes each time PPP re-established.

If we send more than 254 packets, only 254 will be broken and the others will work. It's not always the first 254 or last 254, the broken ones move around between tests.

So it sounds like the modem (or, less likely, something in the cab or exchange) is creating state table entries for packets it is passing which tie them to a particular PPP session, and then failing to flush the table when the PPP goes down.

This is a little crazy in the first place. It's a modem, it shouldn't even be aware that it's passing PPPoE frames, let along looking inside them to see that they are UDP.

This only happens when using an Openreach Huawei HG612 modem that we suspect has been recently remotely and automatically upgraded by Openreach in the past couple of months. Further - a HG612 modem with the 'unlocked' firmware does not have this problem. A HG612 modem that has probably not been automatically/remotely upgraded does not have this problem.

Side note: One theory is that the brokenness is actually happening in the street cab and not the modem. And that the new firmware in the modem which is triggering it has enabled 'link-state forwarding' on the modem's Ethernet interface.
"
https://status.aa.net.uk/1854

Have you checked to see if the problem is present on both eci and huawie street cabs?

Edited by R0NSKI (Tue 12-Nov-13 15:33:37)

it sounds a bit bizzare for an 'accidental bug'.

I've seen weirder.

We're not sure, as we can't tell ourselves what kit is in the cab. We have been in contact with BT regarding this though, and they are investigating too.

In this case the customer themselves might be handy as the source of information - the 3 cabinets BT use look quite different and it's easy enough to tell ECI from Huawei. Each exchange area / all exchanges connecting back to a single NGA handover all have the same DSLAM manufacturer.

I believe it's also possible to tell which type of cabinet is in use by the band plans being used, but to know this you will of course need an unlocked modem.

But as Ignitionnet says, the cabs do look different depending on the make.

Also, I'm pretty sure a certain poster on these forums could tell you what make of cab it is if given the exchange and cab number, even if that was by pm.

there is also a command on the hg to identify the remote chipset.

broadcom = huawei

iftn=eci

command

'xdslcmd info --vendor'

# xdslcmd info --vendor
xdslcmd: ADSL driver and PHY status
Status: Showtime
Retrain Reason: 0
Max: Upstream rate = 27342 Kbps, Downstream rate = 68016 Kbps
Path: 0, Upstream rate = 20000 Kbps, Downstream rate = 68965 Kbps

ChipSet Vendor Id: IFTN:0xb203
ChipSet VersionNumber: 0xb203
ChipSet SerialNumber: 5501843597