Ops - routing broken/glitch?

Anyone awake in Zen NMC land...

Some sort of routing glitch has just struck.

From London lines on your network we get no route to some yahoo address space.

[root@xs ~]# traceroute -I uk.messenger.yahoo.com
traceroute to uk.messenger.yahoo.com (98.139.200.98), 30 hops max, 40 byte packets
1 gateway-to-the-void.goth.int (172.16.0.1) 0.261 ms 0.228 ms 0.261 ms
2 losubs.subs.dsl3.th-lon.zen.net.uk (62.3.84.25) 7.904 ms 8.350 ms 9.088 ms
3 ae0-116.cr1.th-lon.zen.net.net (62.3.84.193) 9.712 ms 10.186 ms 9.198 ms
4 ge-3-0-0-0.cr2.kp-leeds.zen.net.uk (62.3.80.77) 15.134 ms 15.113 ms 15.687 ms
5 * * *
6 * * *
7 * * *
8 * * *
9 * * *


It's working from another wholesale link;

[root@helios ~]# traceroute -I uk.messenger.yahoo.com
traceroute to uk.messenger.yahoo.com (98.139.200.98), 30 hops max, 40 byte packets
1 pointer.goth.int (x.x.x.x) 4.495 ms 4.915 ms 4.921 ms
2 79.135.121.80 (79.135.121.80) 35.154 ms * *
3 * * *
4 * * *
5 * * *
6 * gi0-0-71-gradwell.bdr-rt2.thdo.ncuk.net (80.249.97.21) 32.350 ms 34.441 ms
7 ge-3-3-0.pat1.tc2.yahoo.com (195.66.226.129) 74.237 ms 43.928 ms 43.441 ms
8 UNKNOWN-66-196-65-X.yahoo.com (66.196.65.0) 45.449 ms 32.368 ms ge-0-1-0.pat1.the.yahoo.com (66.196.65.28) 30.819 ms
9 UNKNOWN-216-115-96-X.yahoo.com (216.115.96.30) 114.010 ms as-0.pat1.nyc.yahoo.com (66.196.65.13) 115.277 ms 109.799 ms
10 ae-2.pat2.bfz.yahoo.com (216.115.100.74) 128.579 ms ae-2.pat1.bfz.yahoo.com (216.115.100.26) 126.766 ms ae-2.pat2.bfz.yahoo.com (216.115.100.74) 129.457 ms
11 ae-4.msr2.bf1.yahoo.com (216.115.100.73) 122.908 ms ae-3.msr1.bf1.yahoo.com (216.115.100.29) 124.131 ms ae-4.msr2.bf1.yahoo.com (216.115.100.73) 124.083 ms
12 xe-4-0-0.clr1-a-gdc.bf1.yahoo.com (98.139.232.81) 121.501 ms 121.389 ms xe-7-0-0.clr2-a-gdc.bf1.yahoo.com (98.139.128.19) 125.148 ms
13 et-17-1.fab2-1-gdc.bf1.yahoo.com (98.139.128.39) 127.404 ms et-18-25.fab8-1-gdc.bf1.yahoo.com (98.139.128.67) 130.034 ms 129.328 ms
14 po-10.bas2-1-prd.bf1.yahoo.com (98.139.129.31) 122.874 ms po-16.bas2-1-prd.bf1.yahoo.com (98.139.129.127) 129.052 ms po-10.bas2-1-prd.bf1.yahoo.com (98.139.129.31) 123.886 ms
15 myc1.msg.vip.bf1.yahoo.com (98.139.200.98) 127.605 ms 123.345 ms 122.788 ms

Same problem here. Don't have any connectivity to many sites at the moment.

Edited by deleted (Mon 25-Apr-11 18:23:16)

A "me too" reply over here.

Same here.

Looks like so far it's

98.136.0.0/14
69.147.64.0/18
129.82.0.0/16

I see routes for them via LINX anyway, and as I've said other wholesale providers.

Ok I see other posts since I updated this - might be useful to start getting a list of prefixes that people can't reach.

Edited by IamQ (Mon 25-Apr-11 18:32:21)

I'm not sure exactly the full range of IPs that I can't get to but I do know I can't get to these (they respond from my external server in The Netherlands):

70.84.222.0/24
72.233.89.0/24
208.100.20.0/24

Edited by deleted (Mon 25-Apr-11 18:36:51)

Same here - been iffy for the last couple of hours.

pablo@ubuntu-64:~$traceroute -I uk.messenger.yahoo.com
traceroute to uk.messenger.yahoo.com (98.139.200.98), 30 hops max, 60 byte packets
1 router (192.168.1.1) 0.783 ms 1.312 ms 1.694 ms
2 losubs.subs.dsl1.mbr-roch.zen.net.uk (62.3.82.17) 22.521 ms 24.874 ms 26.557 ms
3 ae0-136.cr1.mbr-roch.zen.net.uk (62.3.80.137) 28.592 ms 29.592 ms 31.705 ms
4 ae2-0.cr1.kp-leeds.zen.net.uk (62.3.80.70) 35.225 ms 36.879 ms 38.387 ms
5 ge-3-1-0-0.cr2.kp-leeds.zen.net.uk (62.3.80.74) 40.406 ms 42.779 ms 43.980 ms
6 * * *
7 * * *
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
pablo@ubuntu-64:~$

Same problem here
imgur.com (173.231.140.218)

Got another range:
204.152.204.0/24

I tried to call up support but completely forgot it's a bank holiday today. Thought it was still Sunday. Doh...

"Me too". Can't connect to Dropbox.com or Twitter.com

*for traceroute, dropbox fell at the first hop after my router.

Edited by ian_c (Mon 25-Apr-11 18:59:01)

This has just been posted to the Zen status pages.

I have routing back - though it's a bit of a mess at the moment.

http://status.zensupport.co.uk/index.php?serviceid=2...

"Transit & Peering

The following new Transit & Peering incident has been identified. Further information is detailed below.
New York Link issues(New)
Outage - Emergency maintenance

Start: Mon, 25/04/2011 19:16

End:

Update: Mon, 25/04/2011 21:00

Clear:

Our engineers have discovered an issue with our New York link. This has been raised to our third party supplier who are investigating.

Some customers may notice a change to the way their traffic is routed, however no impact to service is expected.

Zen regret any inconvenience this may cause.

Posted: Mon, 25/04/2011 19:17 by Mark McKinlay"

Edited by IamQ (Mon 25-Apr-11 19:24:17)

Yup, I'm managing to get to my destination again at least so huzzah!

Ditto.

Oh ye of little faith.

Had it been most other ISPs the problem would have remained until tomorrow probably.

Credit where credit is due I think.

Just remember it does take time to track down the root cause of a problem.

Edited by techguy (Mon 25-Apr-11 21:32:50)

In reply to a post by techguy:

Oh ye of little faith.

Had it been most other ISPs the problem would have remained until tomorrow probably.

Credit where credit is due I think.

Just remember it does take time to track down the root cause of a problem.

Maybe - but then if I had lost half our eBGP sessions to the USA I'd of known within the BGP time out interval and iBGP or another eBGP peer would of fixed everything and pushed it another way without intervention. I'd of also got and sent alerts to my NMC team and my on-call team, it would of also auto updated the status portal & tickers.

Also true I have seen some ISP's drop all there BGP on a Saturday morning and its not back and running till the Monday - but frankley those people shouldn't be ISP's and any customer/user who goes with someone *that* shonky has only them selves to blame...

Credit where it's due - but this is business as usual stuff that should of automatically been picked up and flagged to the 'outages' page and or any other user based platform (RSS, Twitter etc) I don't take much and I've had conversations with people in Zen land about this sort of thing before.

Remember this would of broken *lots* of people over the entire network (including leased, core and access customers.)

Still no update on the fault either, but traffic to the AS's I posted the other day is now flowing the via London and being handed off that way.

Is Zen still having issues - its unusable this evening, like wading through treacle.

Am supposed to be working but having to resort to a 3G connection.

Err not sure TBH. Not noticed any problems at home this evening and none of my remote sessions have died.

Not done much downloading only light browsing.

Sorry can't be much more help.

USA traffic is still going via London anyway - that might have an impact depending where you are, though I'd not expect it to be much.

Ah no - turned out to be a router firmware issue!!

Latest firmware (.48) version for the netgear dgnd3300v2 router gradually slowed it to a crawl before killing any and all netgear connections.

I wondered why when I restarted all would be fine again for a litle while... doh!

Roll on tmw - fttc day!

In reply to a post by peaks:

Ah no - turned out to be a router firmware issue!!

Latest firmware (.48) version for the netgear dgnd3300v2 router gradually slowed it to a crawl before killing any and all netgear connections.

I wondered why when I restarted all would be fine again for a litle while... doh!

Roll on tmw - fttc day!

Ahh that's ok then. Are you by chance using the router as a DNS relay ? That was known to slow them all down and crash a few.