Register (or login) on our website and you will not see this ad.
|
|
Good Morning,
I've recently migrated to Zen, from BT, and am facing an annoying problem - head banging against a brick-wall has started. For the last 3 years my current setup has functioned fantastically on BT, since the migration to Zen things seem to have gone a bit wonky - the Zen aspect may or may not be related.
I have an OpenBSD 5.7 router connected to either an HG612/ECI/TG589 (bridge) modem, via a switch, I encounter the same problem with both. The problem? PPPoE drops frequently after between 1 - 15 minutes of connected time and reconnects, then repeats, the modem sync is not dropping. The router has an OpenVPN (UDP) VPN connection that routes all traffic to the OpenVPN server in the DC. I should add, I still have another line still with BT with the exact same setup and this does not encounter the problem and has been up for some 70 days.
Between migrating from BT -> Zen, the only thing that changed on the OpenBSD router was the PPPoE username/password. From the moment the migration occurred, this problem started occurring.
Thing's I have ruled out:
- Cabling, no errors on switch ports but all cables have been replaced
- Not HG612 or ECI related, problem happens with both (also the Zen TG589 in bridge mode). Initially thought it could be the HG612 bug with UDP/VPNs, however the modem is unlocked and running the latest release. The trick of unplugging and reseating the eth cable doesn't make any difference.
- OpenBSD config, there is minimal kernel PPPoE config same setup works with BT and continues to work
- OpenBSD OS versions (tried 3 different images)
- Rolled back RFC4638 patches, i.e for MTU 1500. The Max Payload is negotiated successfully during the connection process, so I don't believe this is the issue but have tried without anyway.
- LCP echo/replies are all being sent and responded to in a timely manner, there are no ignore/dropped echos/replies before the 'term-req' is received'
- Line itself is stable when using the TG589 in full router mode, i.e PPP does not disconnect as it does in bridge
Enabled debug on the OpenBSD pppoe interface Zen are sending 'term-req' - verified by checking the MAC address sending the term-req - from some forum searches, here and on Kitz, this seems to have come up a number of times with Zen but with no one really getting to the bottom of it?
Jun 28 21:15:56 rtr00 /bsd: pppoe0 (8864) state=3, session=0x2eb output -> 84:26:2b:a2:3c:da, len=139
Jun 28 21:15:56 rtr00 /bsd: pppoe0 (8864) state=3, session=0x2eb output -> 84:26:2b:a2:3c:da, len=139
Jun 28 21:15:56 rtr00 /bsd: pppoe0: lcp input(opened): <term-req id=0x0 len=4 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-3f-75-80-e0-83-59-1b-c3>
Jun 28 21:15:56 rtr00 /bsd: pppoe0: lcp opened->stopping
Jun 28 21:15:56 rtr00 /bsd: pppoe0: phase terminate
Jun 28 21:15:56 rtr00 /bsd: pppoe0: ipcp down(opened)
Jun 28 21:15:56 rtr00 /bsd: pppoe0: ipcp opened->starting
Jun 28 21:15:56 rtr00 /bsd: pppoe0: ipcp close(starting)
Jun 28 21:15:56 rtr00 /bsd: pppoe0: ipcp starting->initial
Jun 28 21:15:56 rtr00 /bsd: pppoe0: lcp send terminate-ack
Jun 28 21:15:56 rtr00 /bsd: pppoe0: lcp output <term-ack id=0x0 len=4>
Jun 28 21:15:56 rtr00 /bsd: pppoe0 (8864) state=3, session=0x2eb output -> 84:26:2b:a2:3c:da, len=12
Jun 28 21:16:06 rtr00 /bsd: pppoe0: lcp TO(stopping) rst_counter = 0
Jun 28 21:16:06 rtr00 /bsd: pppoe0: lcp stopping->stopped
Jun 28 21:16:06 rtr00 /bsd: pppoe0: phase dead
Jun 28 21:16:06 rtr00 /bsd: pppoe0: timeout
Jun 28 21:16:06 rtr00 /bsd: pppoe0: disconnecting
Jun 28 21:16:06 rtr00 /bsd: pppoe0: lcp down(stopped)
Jun 28 21:16:06 rtr00 /bsd: pppoe0: lcp stopped->starting
Jun 28 21:16:06 rtr00 /bsd: pppoe0: phase establish
Jun 28 21:16:06 rtr00 /bsd: pppoe0 (8863) state=1, session=0x0 output -> ff:ff:ff:ff:ff:ff, len=18
Jun 28 21:16:06 rtr00 /bsd: pppoe0: Down event (carrier loss), taking interface down.<7>pppoe0: lcp close(starting)
Jun 28 21:16:06 rtr00 /bsd: pppoe0: lcp starting->initial
Jun 28 21:16:06 rtr00 /bsd: pppoe0: phase dead
Jun 28 21:16:06 rtr00 /bsd: pppoe0 (8863) state=2, session=0x0 output -> 84:26:2b:a2:3c:da, len=38
Jun 28 21:16:06 rtr00 /bsd: pppoe0: session 0x2ee connected
Jun 28 21:16:06 rtr00 /bsd: pppoe0: lcp open(initial)
Jun 28 21:16:06 rtr00 /bsd: pppoe0: lcp initial->starting
Jun 28 21:16:06 rtr00 /bsd: pppoe0: phase establish
Jun 28 21:16:06 rtr00 /bsd: pppoe0: lcp up(starting)
Jun 28 21:16:06 rtr00 /bsd: pppoe0: lcp starting->req-sent
Jun 28 21:16:06 rtr00 /bsd: pppoe0: lcp output <conf-req id=0x5 len=14 05-06-d2-4f-8e-af-01-04-05-d4>
Jun 28 21:16:06 rtr00 /bsd: pppoe0 (8864) state=3, session=0x2ee output -> 84:26:2b:a2:3c:da, len=22
Jun 28 21:16:06 rtr00 /bsd: pppoe0: lcp input(req-sent): <conf-req id=0xa7 len=19 01-04-05-d4-03-05-c2-23-05-05-06-1a-89-24-a4-00-00-00-00-00-00-00-00-00-00-00-20-85-80-10-ff-1c-cb-c5>
LCP echo request/reply going back and forth, no sign of any timeouts on my end, 4 seconds after the last request/reply the term-req is received from the ISP/BT/Zen.
Jun 29 13:20:42 rtr00 /bsd: pppoe0: lcp input(opened): <echo-req id=0x16 len=8 55-fc-dc-3b-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-b1-d0-1d-2b-00-00-00-00>
Jun 29 13:20:42 rtr00 /bsd: pppoe0: got lcp echo req, sending echo rep
Jun 29 13:20:42 rtr00 /bsd: pppoe0: lcp output <echo-reply id=0x16 len=8 9b-cc-13-a7>
Jun 29 13:20:45 rtr00 /bsd: pppoe0: lcp input(opened): <echo-req id=0x17 len=8 55-fc-dc-3b-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-31-92-e8-9b-00-00-00-00>
Jun 29 13:20:45 rtr00 /bsd: pppoe0: got lcp echo req, sending echo rep
Jun 29 13:20:45 rtr00 /bsd: pppoe0: lcp output <echo-reply id=0x17 len=8 9b-cc-13-a7>
Jun 29 13:20:55 rtr00 /bsd: pppoe0: lcp input(opened): <echo-req id=0x18 len=8 55-fc-dc-3b-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-54-f8-2b-14-00-00-00-00>
Jun 29 13:20:55 rtr00 /bsd: pppoe0: got lcp echo req, sending echo rep
Jun 29 13:20:55 rtr00 /bsd: pppoe0: lcp output <echo-reply id=0x18 len=8 9b-cc-13-a7>
Jun 29 13:20:58 rtr00 /bsd: pppoe0: lcp input(opened): <echo-req id=0x19 len=8 55-fc-dc-3b-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-44-34-28-75-00-00-00-00>
Jun 29 13:20:58 rtr00 /bsd: pppoe0: got lcp echo req, sending echo rep
Jun 29 13:20:58 rtr00 /bsd: pppoe0: lcp output <echo-reply id=0x19 len=8 9b-cc-13-a7>
Jun 29 13:21:01 rtr00 /bsd: pppoe0: lcp input(opened): <echo-req id=0x1a len=8 55-fc-dc-3b-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-fc-ff-39-26-00-00-00-00>
Jun 29 13:21:01 rtr00 /bsd: pppoe0: got lcp echo req, sending echo rep
Jun 29 13:21:01 rtr00 /bsd: pppoe0: lcp output <echo-reply id=0x1a len=8 9b-cc-13-a7>
Jun 29 13:21:05 rtr00 /bsd: pppoe0: lcp input(opened): <term-req id=0x12 len=4 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-fc-08-46-fa-00-00-00-00>
Jun 29 13:21:05 rtr00 /bsd: pppoe0: lcp opened->stopping
Jun 29 13:21:05 rtr00 /bsd: pppoe0: lcp send terminate-ack
Jun 29 13:21:05 rtr00 /bsd: pppoe0: lcp output <term-ack id=0x12 len=4>
.. If anyone has any suggestions, or seen anything similar previously, I'm all ears  Gut feeling is that there is a general incompatibility somewhere, one forum post I found (chap on Zen again) tried 4-5 different routers/modems and still saw the problem. Although, I accept there are many users without this problem using non-Zen-supplied hardware.
I currently have my second line in a migration state from BT, which I think I'm going to have to put on hold until I get to the bottom of this.
Edited by deleted (Tue 30-Jun-15 08:21:07)
|
|
|
Quick suggestion - try reducing MTU to 1492 and see if it makes any difference.
I have an entirely stable setup with Zen using HG612, Draytek 2830 and Linux/Windows network all using MTU 1492. I have seen problems in the dim and distant past with MTU 1500.
--
Brian
Zen Fibre 2 - 80/20 sync
|
|
|
Hi there,
Thanks for the reply - going back to 1492 was one of the first things I tried, including going back to a version of OpenBSD that had no support for RFC4638 at all. Same problem occurred.
At the moment I'm testing using a TCP OpenVPN tunnel on the router, rather than UDP, so far PPP has stayed up for over an hour - it never did this with UDP. But need to complete more tests to see if this is a fluke and/or it reoccurs when going back to UDP - or if someone/somewhere at BT/Zen noticed/fixed something in the background
Thanks,
Edited by deleted (Tue 30-Jun-15 13:28:26)
|
|
Register (or login) on our website and you will not see this ad.
|
|
|
|
Let me know how you get on. I will be interested to see if using UDP was a simple cause.
|
|
|
Sounds like you may have possibly been hit by this bug:
http://www.revk.uk/2013/11/bt-huawei-fttc-modem-bug-...
|
|
|
Sounds like you may have possibly been hit by this bug:
http://www.revk.uk/2013/11/bt-huawei-fttc-modem-bug-...
Hi there,
Thanks for the reply - however, I ruled this one out fairly early on, it happens/happened with the ECI modem and the TG589 Also, when you used to disconnect/reseat the eth cable on an impacted HG612 it worked around it, until PPP reset (i.e should be more than a few minutes). Not to mention my HG612 is unlocked and running the latest and greatest firmware.
The actual 'term-req' is/was coming from the remote MAC address, not local, confirmed with tcpdump.
Cheers,
Edited by deleted (Wed 01-Jul-15 12:08:28)
|
|
|
|
When I had (potentially still have) this problem it just randomly went away, whilst I was trying a number of things - my setup went back to exactly what it was to start with, i.e router/modem/etc, problems disappeared. What was 100% clear from my diagnostics, is that it was Zen that was sending the 'term-req' and pulling the PPP session down - the problem being, no one from Zen could tell me why.
I did notice that at times when I was getting disconnected I was going through a gateway that was suffering high latency, dsl4 at the time seemed to be the worst for me, there are other threads relating to this on TBB. Shortly after the PPP session died after Zen killed it from their end, most of the time, I ended up on a different gateway and it would stay up for a bit longer. Then it would drop again, but dsl4 seemed to be consistently bad for me for the first 3-4 days of my line transferring over to Zen.
Current PPP session is just under 3 days, with 4.5 days modem sync. Am I happy since migrating to Zen? Not overly to be honest, I wasted considerable time looking into this and going through the support-script but with no clear outcome - if I'd wanted that, I could have stayed with BT! With hindsight, I wish I hadn't bothered and just stayed with BT. Having had an AA FTTC line previously I would class that as a premium service, Zen? Slightly better than BT, but not justifiably so with the cost - especially as BT would have given a good deal to retain my custom.
Time will tell if the service Zen provide changes my mind...
|
|
|
|
That is a far cry from the comments we used to get from people about our service and I am sorry you feel this way.
Is DSL4 still one of the main culprits? Would you mind sending me some more info about it? I appreciate you have been through all this before, however working where I am, I can liaise directly with Networks to see if we can find the issue.
|
|
|
Today has been absolutely terrible for me - PPP has been bouncing all over the place since 14:00, every 1 - 20 minutes - I spoke to a chap at Zen earlier who seems to know his stuff, they did some work on DSL4 and apparently moved 6,000 lines to rebalance things. I have to say, I'm still not overly happy - still periods of latency etc, I think their rudimentary load balancing is no longer fit for purpose (if it ever was)
https://f8lure.mouselike.org/index.asp?DoAction=View...
However, still no resolution on the PPP reseting. Calmed down for 4-5 days, then gradually got worse, now I'm back to where I was. My 2nd BT FTTC line continues to be rock solid PPP up time of around 15 days (last reset was me).
|
|
|
For completeness sake, as I haven't posted proof of the MAC sending the term-req yet - tcpdump from the pppoedev interface, LCP echo-request/reply goes back and forth, nothing dropped, then the remote end sends the 'term-req' @21:49:22.681482.
94:de:80:8f:a4:23 is the local MAC of the pppoedev interface, 84:26:2b:a2:3c:da is/was the remote for this session
| Text | 1
23
45
67
89
1011
1213
1415
1617
1819
2021
2223
2425
2627
2829
3031
3233
3435
3637
3839
4041
42 | 21:49:12.583483 84:26:2b:a2:3c:da 94:de:80:8f:a4:23 8864 60: PPPoE-Session
code Session, version 1, type 1, id 0x0264, length 10 LCP: Echo-Request, Magic-Number=1419402018
21:49:12.583505 94:de:80:8f:a4:23 84:26:2b:a2:3c:da 8864 30: PPPoE-Session code Session, version 1, type 1, id 0x0264, length 10
LCP: Echo-Reply, Magic-Number=211993925721:49:15.624809 84:26:2b:a2:3c:da 94:de:80:8f:a4:23 8864 60: PPPoE-Session
code Session, version 1, type 1, id 0x0264, length 10 LCP: Echo-Request, Magic-Number=1419402018
21:49:15.624828 94:de:80:8f:a4:23 84:26:2b:a2:3c:da 8864 30: PPPoE-Session code Session, version 1, type 1, id 0x0264, length 10
LCP: Echo-Reply, Magic-Number=211993925721:49:18.604635 84:26:2b:a2:3c:da 94:de:80:8f:a4:23 8864 60: PPPoE-Session
code Session, version 1, type 1, id 0x0264, length 10 LCP: Echo-Request, Magic-Number=1419402018
21:49:18.604655 94:de:80:8f:a4:23 84:26:2b:a2:3c:da 8864 30: PPPoE-Session code Session, version 1, type 1, id 0x0264, length 10
LCP: Echo-Reply, Magic-Number=211993925721:49:22.681482 84:26:2b:a2:3c:da 94:de:80:8f:a4:23 8864 60: PPPoE-Session
code Session, version 1, type 1, id 0x0264, length 6 LCP: Terminate-Request
21:49:22.681509 94:de:80:8f:a4:23 84:26:2b:a2:3c:da 8864 26: PPPoE-Session code Session, version 1, type 1, id 0x0264, length 6
LCP: Terminate-Ack21:49:32.742741 94:de:80:8f:a4:23 84:26:2b:a2:3c:da 8864 32: PPPoE-Session
code Session, version 1, type 1, id 0x0265, length 12 LCP: Configure-Request, Magic-Number=-82992445[|lcp]
21:49:32.787984 84:26:2b:a2:3c:da 94:de:80:8f:a4:23 8864 60: PPPoE-Session code Session, version 1, type 1, id 0x0265, length 21
LCP: Configure-Request, Max-Rx-Unit=1500, Auth-Prot CHAP/MD5, Magic-Number=127734709, Vendor-Ext21:49:32.787987 84:26:2b:a2:3c:da 94:de:80:8f:a4:23 8864 60: PPPoE-Session
code Session, version 1, type 1, id 0x0265, length 12 LCP: Configure-Ack, Magic-Number=-82992445, Vendor-Ext
21:49:32.788012 94:de:80:8f:a4:23 84:26:2b:a2:3c:da 8864 41: PPPoE-Session code Session, version 1, type 1, id 0x0265, length 21
LCP: Configure-Ack, Max-Rx-Unit=1500, Auth-Prot CHAP/MD5, Magic-Number=127734709[|lcp]21:49:33.367008 84:26:2b:a2:3c:da 94:de:80:8f:a4:23 8864 60: PPPoE-Session
code Session, version 1, type 1, id 0x0265, length 21 LCP: Configure-Request, Max-Rx-Unit=1500, Auth-Prot CHAP/MD5, Magic-Number=127734709, Vendor-Ext
21:49:33.367041 94:de:80:8f:a4:23 84:26:2b:a2:3c:da 8864 41: PPPoE-Session code Session, version 1, type 1, id 0x0265, length 21
LCP: Configure-Ack, Max-Rx-Unit=1500, Auth-Prot CHAP/MD5, Magic-Number=127734709[|lcp] |
Resetting on average of every 5 minutes at the moment.
Edited by deleted (Wed 08-Jul-15 23:05:59)
|
|
|
|
Beginning to again suspect Zen do something funky when the traffic seen is all UDP (or even primarily UDP), the moment I enable an OpenVPN UDP tunnel PPP starts getting reset again - between 1 and 20 minutes - traffic is generally pretty low, 10-100K/S, apart from the odd burst when rsync backups happen. So, I shut down the OpenVPN tunnel on the Zen routing domain and then loop downloaded a 100M speedtest file from Linode, so all traffic is TCP (other than maybe the odd DNS request).....PPP stays up.
It is interesting that the days I saw latency problems there were no PPP resets, after they did maintenance on DSL4 (confirmed) latency improved but the PPP resets started again. I have absolutely nothing concrete to back this up, but I'm beginning to wonder if there is something load-balancing-wise on their end that was disabled for a few days, latency gets worse, they re-enable it and I'm back to the PPP resets when most traffic is UDP. Are they potentially seeing it as P2P/Torrent traffic and trying to move me around to cope with the additional load of Wimbledon?!
Whilst it could be a coincidence, this started again yesterday afternoon about 14:00 when latency started to increase again - this was when Andy Murray was playing, things went mad last time he was playing in the tournament.
|
|
|
That is indeed super strange.
What happens if you bring the PPP interface up and send no IP traffic what so ever on it? Do Zen still disconnect you due to some time out being reached due to no traffic being sent?
This is not something I would expect them to do, but possibly this scenario is never reached nowadays because of general traffic noise that most networks now generate (NTP syncing, DNS lookups etc).
Edited by deleted (Thu 09-Jul-15 13:42:01)
|
|
|
|
Hi there,
If I connect the PPPoE interface and send no traffic, I mean no traffic as it is in its own routing domain with no daemons/services started, it stays up.
If I then loop a wget download from Linode, HTTP/TCP, it still stays up....
My conclusions (with nothing really to back them up, other than what I've seen) is that either they're trying to shift around heavy UDP traffic, or they're ignoring the LCP echo request/reply for the health check and check instead on seeing TCP traffic... for which there would be none at all when the OpenVPN tunnel is up.
Cheers,
|
|
|
Perhaps?
https://aastatus.net/1854 (and revK's link).
|
|
|
|
Right.
Ok, next couple of tests I would now try are:
1) Bring the interface up fresh and send just a single UDP packet out of the connection. Wait for it to drop/remain up.
2) Bring the interface up fresh and send just a single TCP packet out of the connection. Wait for it to drop/remain up.
It is indeed looking like what you are suggesting. It may just be the case that you'll have to cronjob some netcat open port test (nc -z ip port on Linux) to generate some kind of TCP traffic so they don't terminate the link. I would be interested to know the outcome of the above 2 tests though.
|
|
|
Perhaps?
https://aastatus.net/1854 (and revK's link).
No, this is different - no ports are blacklisted and the reset *is* coming from the remote MAC, not just dropped. I'm also not using an HG612 at the moment.
I have however tested with HG612 (unlocked and fully updated, so would/should be patched), ECI modem, ECI modem running OpenWRT and a TG589v3.
Edited by deleted (Thu 09-Jul-15 13:57:17)
|
|
|
It is indeed looking like what you are suggesting. It may just be the case that you'll have to cronjob some netcat open port test (nc -z ip port on Linux) to generate some kind of TCP traffic so they don't terminate the link. I would be interested to know the outcome of the above 2 tests though.
If that is the case, I will be seeking contract cancellation and likely just going to A&A again....after the amount of time spent on this in the last week or so, I'll deal with the prices.
To be fair to Zen, I finally have a chap on the case who seems to have a natural geeky-curiosity-for-the-unexplained...something I was struggling with originally with people I spoke to @Zen, especially when told to wait and see if it happens again.
Am going for a walk for a bit, will run some of the tests mentioned above later on.
Edited by deleted (Thu 09-Jul-15 19:11:37)
|
|
|
Interface was up with an infinite loop downloading a Linode speedtest file over HTTP for over 5 hours, within 30 minutes of re-enabling the OpenVPN UDP interface (so all traffic out to the ISP was UDP) the PPP bouncing started again.
Disclaimer: Yes, I accept it could be coincidence....again 
Edited by deleted (Thu 09-Jul-15 21:05:06)
|
|
|
Latency was good through the Andy Murray match - PPP was reasonably stable throughout the day, even with a UDP OpenVPN tunnel, stayed up for 5 hours, has just gone down hill in the last 30 minutes with at least 4 term-reqs from the remote end.
May or may not be related, problems started more or less 24 hours after I restarted the OpenVPN UDP tunnel - restarted it yesterday evening about 9PM-ish, had one reset shortly after doing so, then remained stable-ish (i.e 4-5 hours) until now.
Edited by deleted (Fri 10-Jul-15 21:33:58)
|
|
|
|
PPP continued to reset pretty much every 5-6 minutes for the last hour, bringing down the OpenVPN UDP tunnel and restarting the infinite loop HTTP (TCP) download from Linode.
2nd FTTC line (BT) at the same premises, running exactly the same OpenVPN UDP tunnel, up for 4 days on PPP - was a manual restart even then.
|
|
|
Looped HTTP download (no OpenVPN UDP tunnel) ran without a PPP drop for around an hour, starting the OpenVPN UDP tunnel again I get a PPP reset within 5-6 minutes. PPP has been reset 4 times so far, again every 5-6 minutes.
Either there is a new chipset bug somewhere (that impacts both HG612, TG589, ECI but only on Zen lines), or Zen (?) have something running to analyse the protocols over the line potentially over a 24 hour period, running every 5 minutes, then resetting if there is X amount/percentage of UDP vs TCP traffic. Others with similar problems probably see the resets as almost random, however as I can/have tested with 100% either UDP or TCP (but never both at the same time) can exacerbate the problem. I do however suspect that this is only done at certain times, or perhaps when certain load is seen in general on their network.
The above is about the only conclusion I can draw from what I have seen.... But maybe I'm looking for conspiracy that isn't there.
Edited by deleted (Fri 10-Jul-15 23:14:57)
|
|
|
|
With the UDP OpenVPN tunnel up resets are almost like clockwork this evening - started at around 14:20, next at 16:20 then... you guessed it..18:20. Every 2 hours almost to the minute, give or take a couple.
|
|
|
Zen (?) have something running to analyse the protocols over the line potentially over a 24 hour period, running every 5 minutes, then resetting if there is X amount/percentage of UDP vs TCP traffic.
Why would they do this?
--
Brian
Zen Fibre 2 - 80/20 sync
|
|
|
Zen (?) have something running to analyse the protocols over the line potentially over a 24 hour period, running every 5 minutes, then resetting if there is X amount/percentage of UDP vs TCP traffic.
Why would they do this?
Hypothetically, to move/balance suspected P2P/Torrent users - again, I have nothing to back this up, but I would imagine the large majority of UDP traffic over an ISP such as Zen will be torrent traffic... Just not in this case.
|
|
|
Zen (?) have something running to analyse the protocols over the line potentially over a 24 hour period, running every 5 minutes, then resetting if there is X amount/percentage of UDP vs TCP traffic.
Why would they do this?
Hypothetically, to move/balance suspected P2P/Torrent users - again, I have nothing to back this up, but I would imagine the large majority of UDP traffic over an ISP such as Zen will be torrent traffic... Just not in this case.
As far as I'm aware, UDP is mostly only used with Bittorrent for connecting to the trackers. This is probably relatively low traffic levels. The actual downloads themselves take place over TCP.
Dropping UDP traffic makes no sense. Typically real time traffic is transported via UDP, such as VoIP. Throttling this would cause massive issues for anyone using something like this. Traffic shaping is done by dropping TCP packets, as these actually scale back upon congestion being encountered using Sliding Window. Dropping UDP packets would typically not cause UDP to scale back what it is sending, and therefore not cause the desired effect of causing traffic flows to slow down the sending of data.
Incidentally, 84:26:2b:a2:3c:da is an Alcatel-Lucent MAC address.
A job spec for them does not list Alcatel-Lucent as a required vendor. They do list Ericsson (formally Redback) however, which are a LNS vendor. It is likely that they are terminating your session on these.
I know (but can't provide evidence) that BT run Alcatel-Lucent's in their network. It looks like it is actually them that are pulling the session down for whatever reason. It would be interesting to find out what Zen are seeing as the session termination cause for your drops.
|
|
|
As far as I'm aware, UDP is mostly only used with Bittorrent for connecting to the trackers. This is probably relatively low traffic levels. The actual downloads themselves take place over TCP.
This used to be the case before uTP was developed. Torrent traffic, both tracker and data transfer, now happens across UDP (using uTP transport protocol, which is BitTorrent specific protocol). That has timing information and all sorts of clever tricks to detect congestion, allowing clients to throttle back (at the application layer of the network stack) in order to not flood network links.
Dropping UDP packets would typically not cause UDP to scale back what it is sending, and therefore not cause the desired effect of causing traffic flows to slow down the sending of data.
Again, per my above comment, uTP does scale back when detecting dropped packets, as it then sees that as a congested link. The protocol is specifically designed to make all this possible.
Further reading here: https://en.wikipedia.org/wiki/Micro_Transport_Protocol
Edited by deleted (Tue 14-Jul-15 15:36:18)
|
|
|
... nested quotes trimmed ...
Why would they do this?
Hypothetically, to move/balance suspected P2P/Torrent users - again, I have nothing to back this up, but I would imagine the large majority of UDP traffic over an ISP such as Zen will be torrent traffic... Just not in this case.
Dropping UDP traffic makes no sense.
I didn't say they were dropping UDP traffic, far from it, more that the ratio of UDP vs TCP was a potential catalyst for the PPP session being terminated from the remote end.
But, I have nothing to prove this conclusively - only that when I was seeing the regular resets I could more or less make it happen (or stop) simply by changing the predominant protocol. On Saturday night for example I would have been able to set my watch by it and it stopped pretty much bang on midnight.
|
|
|
As far as I'm aware, UDP is mostly only used with Bittorrent for connecting to the trackers. This is probably relatively low traffic levels. The actual downloads themselves take place over TCP.
This used to be the case before uTP was developed. Torrent traffic, both tracker and data transfer, now happens across UDP (using uTP transport protocol, which is BitTorrent specific protocol). That has timing information and all sorts of clever tricks to detect congestion, allowing clients to throttle back (at the application layer of the network stack) in order to not flood network links.
I was aware of uTP, but I wasn't aware of how far deployed it is. I've just examined traffic on the network I operate (which isn't a typical network) and I'm seeing a ratio of roughly 5000:1 in favour of TCP for bittorrent transfers (That we can detect)
Dropping UDP packets would typically not cause UDP to scale back what it is sending, and therefore not cause the desired effect of causing traffic flows to slow down the sending of data.
Again, per my above comment, uTP does scale back when detecting dropped packets, as it then sees that as a congested link. The protocol is specifically designed to make all this possible.
Further reading here: https://en.wikipedia.org/wiki/Micro_Transport_Protocol
Hence my use of the word typically. The client could (and allegedly does for uTP) implement parts of TCP that would make it detect congestion and scale back. This is not typical however. In general dropping UDP would not cause the protocol using it to scale back, and would not alleviate congestion. Zen dropping UDP traffic to limit traffic going across their network would be daft.
Additionally Zen claim not to do any traffic shaping.
|
|
|
You seem to be caught up on dropping UDP traffic, I did not say that is what they did - you've misunderstood. The PPP/PPPoE session itself is reset, please check the tcpdump earlier on in this thread.
Edited by deleted (Tue 14-Jul-15 17:36:38)
|
|
|
|
Dropping a PPP session makes even less sense (from a traffic management perspective). You're just going to reconnect and continue your UDP transfer.
While it does sound like something is going in, I would doubt it is intentionally malicious.
|
|
|
Dropping a PPP session makes even less sense (from a traffic management perspective). You're just going to reconnect and continue your UDP transfer.
While it does sound like something is going in, I would doubt it is intentionally malicious.
Not really, not when they had DSL gateways that were confirmed as being overloaded and they manually moved 6,000 odd connections from one. Also from research this is how Zen load balance, quite often having to manually terminate the PPP session to force/hope it connects to another gateway.
I don't recall saying it was malicious, I'd just like it to stop. I suspect, but cannot confirm, it is a very rudimentary load balancing metric, one that I can exacerbate due to the nature of my traffic (again as I said earlier in this thread).
Edited by deleted (Tue 14-Jul-15 17:54:54)
|
|
|
Dropping a PPP session makes even less sense (from a traffic management perspective). You're just going to reconnect and continue your UDP transfer.
While it does sound like something is going in, I would doubt it is intentionally malicious.
Not really, not when they had DSL gateways that were confirmed as being overloaded and they manually moved 6,000 odd connections from one. Also from research this is how Zen load balance, quite often having to manually terminate the PPP session to force/hope it connects to another gateway.
I don't recall saying it was malicious, I'd just like it to stop. I suspect, but cannot confirm, it is a very rudimentary load balancing metric, one that I can exacerbate due to the nature of my traffic (again as I said earlier in this thread).
Not routinely during the middle of the day.
While I do think you should get Zen to investigate, and fix whatever the problem is you're having, I am certain that you are barking up the wrong tree with your load balancing theory.
Did you find out what Zen see as the termination cause?
|
|
|
Dropping a PPP session makes even less sense (from a traffic management perspective). You're just going to reconnect and continue your UDP transfer.
While it does sound like something is going in, I would doubt it is intentionally malicious.
Not really, not when they had DSL gateways that were confirmed as being overloaded and they manually moved 6,000 odd connections from one. Also from research this is how Zen load balance, quite often having to manually terminate the PPP session to force/hope it connects to another gateway.
I don't recall saying it was malicious, I'd just like it to stop. I suspect, but cannot confirm, it is a very rudimentary load balancing metric, one that I can exacerbate due to the nature of my traffic (again as I said earlier in this thread).
Did you find out what Zen see as the termination cause?
That is not as simple as you may think, despite multiple escalations - but yes, that was the very first question I asked that they cannot answer.
I'm going to duck out of this for now, unless I have something else to add and/or a response from them.
Edited by deleted (Tue 14-Jul-15 18:12:03)
|
|
|
|
Was advised today that it had been escalated to 'one of the main NOC guys' - just checking a few things now, LCP echo requests seem to be coming every 10 seconds (this matches my BT FTTC line), checking my earlier dumps/captures they were every 3 seconds...
Now, from the logs the term-req was sent around 4 seconds after the last echo LCP request/response - could this be the problem? One going missing at 3 seconds, then resetting at 4 - either some loss somewhere (but modem sync remains), or an overloaded device generating the echo requests, or simply a little to frequent for real world use?
Will see what Zen come back with tomorrow and I'll raise it with them then.
|
|
|
|
Post deleted by iMiMiMx
|
|
|
|
Nope - LCP echo request interval still seems to be 10 seconds this morning, but I've had 3 term-reqs in the last few minutes from around 08:30 - modem sync still rock solid.
|
|
|
|
LCP echo interval going to down to 3 seconds is apparently expected behaviour, if the previous LCP echo request on the 10 second interval did not receive a response - the interval is lowered temporarily - so currently, Zen advise the term-req is being sent because the Zen end doesn't think that mine responded. However from my dumps earlier it did, these dumps are as late in the process/flow as I can take them so they're 'on the wire' up to the modem. No real data on why I could reproduce the issues at certain times, almost on-demand, with UDP traffic/tunnels but not TCP.
Interesting/confusing aspect, even though I receive and reply to the 10 second interval LCP echo request which is shown by the tcpdump, the Zen end thinks mine didn't respond and so lowers the interval to 3, my end receives again and responds. If the link were dead, then surely I would not expect to receive the LCP echo on the 3 second interval, twice, and respond to it both times? Or, if there is a fault the fault only sometimes impacts the upstream - to allow me to receive and see their echo request, even after the remote end thinks my end did not respond. Whilst coincidences do happen, the problem starting from the word go on the migration should add some weight against it being a line fault (and no sync drop, ever).
After a bit of a slow start investigating at the required low level, finally, Zen are hopefully coming into their own and showing why they have the reputation they do. The last few days/week I've certainly been more encouraged by the technical updates coming back and their desire to continue investigating.
|
|
|
Pages in this thread: 
Print Thread
