NTP DDoS attacks

I got bit this week... was on holiday and when getting home, my Internet allowance was used up - 27GB in 2 days.

To cut a long story short, it was my fault - years ago I had a server with the NTP pool. Trouble was, I never changed the conf, so it was always open.

Anyway, after sorting that out, and locking down the door, I was still being hit by 100's of requests a minute.

The only way out was to get an IP change - sorted.

Here is a mail I got - please read to make sure you don't get hit by this - your ISP may not be so good to help out:

==================================

The following free online tool can be used to check your IP address:
http://support.ntp.org/ntpq.php

Administrators:
1. If you run ntpd, upgrading to the latest version, which removes the
"monlist" command that is used for these attacks; alternately, disabling the
monitoring function by adding "disable monitor" to your /etc/ntp.conf file.
2. Setting the NTP installation to act as a client only. With ntpd, that can
be done with "restrict default ignore" in /etc/ntp.conf; other daemons
should have a similar configuration option. More information on configuring
different devices can be found here:
https://www.team-cymru.org/ReadingRoom/Templates/sec...
3. Adjusting your firewall or NTP server configuration so that it only
serves your users and does not respond to outside IP addresses.

If you don't mean to run a public NTP server, we recommend #1 and #2. If you
do mean to run a public NTP server, we recommend #1, and also that you
rate-limit responses to individual source IP addresses -- silently
discarding those that exceed a low number, such as one request per IP
address per second. Rate-limit functionality is built into many
recently-released NTP daemons, including ntpd, but needs to be enabled; it
would help with different types of attacks than this one.

Fixing open NTP servers is important; with the 400x+ amplification factor of
NTP DRDoS attacks -- one 40-byte-long request usually generates 18252 bytes
worth of response traffic -- it only takes one machine on an unfiltered 1
Gbps link to create a 450+ Gbps attack!

============================================

Nick

I received the same email earlier and was a bit disapponted to discover that the router IOS I updated about three weeks ago to the latest version released on 22 Nov 2013 was vulnerable.

As yet there's no fix or workaround beyond disabling NTP on the router

Edited by caffn8me (Fri 14-Feb-14 17:51:21)

Does disabling NTP on a router prevent a network behind the router from accessing NTP servers in the www?

In reply to a post by Lethe:

https://www.team-cymru.org/ReadingRoom/Templates/sec...

Cheers for the heads up...
You might want to remove that pesky . from the URL above
http://www.team-cymru.org/ReadingRoom/Templates/secu... works better than yours

No, not it my case. What should happen is that you trust everything on local network, so requests to NTP server on the Internet will be allowed to reply - but cold queries to NTP server will be dropped.

Nick

In reply to a post by camieabz:

Does disabling NTP on a router prevent a network behind the router from accessing NTP servers in the www?

Not in this case. All it does is stops the timestamp on router logs being accurate as the router's own NTP service is not running. Machines inside the router still synchronize with external NTP servers without any problem.

If you are using your router as a firewall, disabling inbound NTP - (TCP and UDP port 123) should still not affect internal synchronization. I wouldn't imagine inbound NTP would be open unless explicitly configured.

Surely toy can just adjust the acl on the external interfaces to prevent inbound ntp from the internet in general?

First of all though, does the router open port 123 anyway?

Use the check here:

http://support.ntp.org/ntpq.php

and also do a scan to see what ports are open:

http://www.t1shopper.com/tools/port-scan/

Nick

Interesting� my setup is that this computer (the "main" one on the LAN, an iMac) uses ntpd to sync with one of the NPL public time servers, everything else on the LAN (including the router) gets its time from this computer (at least, those that give me a choice of server). All using IPv4.

Your first link shows that ntpd is accessible from the internet on this machine over IPv6 but returns zeroes over IPv4. Presumably that's due to not using NAT with IPv6- the link sees the router over IPv4, this machine over IPv6.

The second link only seems to know about IPv4 and says that port 123 on the router is not open.

So the OS X firewall appears to pass incoming IPv6 ntpd requests, not sure it's worth worrying about?

In reply to a post by ionic:

Surely toy can just adjust the acl on the external interfaces to prevent inbound ntp from the internet in general?

Not according to Cisco. Because NTP uses UDP it's quite easy to spoof the source address which can still result in the router being used for a DDoS attack. I'm sure they'll eventually come up with a fix.

Workaround:
There are no workarounds other than disabling NTP on the device.
<snip>

Warning: Because the feature in this vulnerability utilizes UDP as a
transport, it is possible to spoof the sender's IP address, which may defeat
access control lists (ACLs) that permit communication to these ports from
trusted IP addresses.

Blimey - from here:

http://support.ntp.org/bin/view/Support/AccessRestri...

6.5.1.1.2. Blocking Unauthorized Access

If your ntpd is publicly accessible, do you really need to block all connections from unauthorized hosts?

If the answer is "No", skip to 6.5.1.1.3. Allow Queries?

If the answer is "Yes" use the following default restriction (and keep in mind that you will have to add restrict lines for every authorized server and client host/subnet as described in 6.5.1.2.1. If you used 'restrict default ignore'):

IPV4: restrict default ignore
IPv6: restrict -6 default ignore

What I did was firewall off port 123 (after the horse has bolted) and added this to ntp.conf:

restrict 12.34.56.0 mask 255.255.255.0 nomodify notrap

which allows my local network access (not real IP there) but nobody else.

Maybe you need something like it but with IPV6 addresses.

Nick
EDIT: I pasted wrong info

Edited by deleted (Sat 15-Feb-14 12:23:52)

Hmm, I need to do some reading, that's obvious!

Then I need to try to understand what I'm reading, that's likely to be the tricky bit

Thanks for that.

Sorry, just edited above post - have another read.

Nick

IPV6 port scanner.

http://ipv6.chappell-family.com/ipv6tcptest/

Nick

I think what I said before still applies- I don't know a lot about IPv4 and even less about IPv6

lethe / caffn8me - Cheers for that. Just wondering if routers out there are default 'on' state, and whether that's necessary for normal operation.

Bill - Unless you think you're getting DDos attacks, I wouldn't worry.

If you want to see what it looks like... gulp:

http://linicks.net/ntpddos.png

Nick

In reply to a post by camieabz:

.

Bill - Unless you think you're getting DDos attacks, I wouldn't worry.

No, no, noooooo. This is what happened to me, and reading up, the bot[s] mark open NTP servers and collate for a while. Then *BANG* your server gets hit into the DDoS.

Now, at this stage, you shut it down/firewall/whatever... but the rest of the bot network STILL keep hitting you non-stop 100's of hits a minute with 1000's of IP's.

It is unrelenting - and even at this stage, I was still using up about 3MB an hour on this. The modem light doesn't stop flashing for a second.

Hence why I had to get an IP change (ISP will park my old IP for a few weeks, hoping it will fade away).

Nick

Edited by deleted (Sat 15-Feb-14 16:14:16)

In reply to a post by camieabz:

Bill - Unless you think you're getting DDos attacks, I wouldn't worry.

That's more or less the conclusion I've come to- I've had a google around and looked at the files that OS X appears to use (it's got a separate .conf file for restrictions) and decided that meddling is a higher risk approach than hoping I don't get attacked

The iMac runs with a dynamic IPv6 address anyway (though the router is fixed), and with a /48 allocation a reboot should slow them down a bit while they try to find where I've gone

Bill, don't get complacent:

http://blog.cloudflare.com/understanding-and-mitigat...

Nick

I'm not complacent about it, but everything I've seen so far relates to IPv4 and, as I said, port 123 is blocked by the router on that protocol.

For IPv6 I suspect the risk is extremely low, certainly lower than the risk of what I could do by messing with things I don't understand. If someone can come up with a step-by-step guide how to block incoming IPv6 ntp queries on OS X then I'll think about it.

Also, unless it was a typo, you said earlier that it was costing you 3MB/hour- frankly, I doubt if I'd even notice that. I use nearly 20x that just streaming Radio 3

The 3MB an hour was AFTER I blocked the attack. The attack is relentless.

Here are someT logs:

During attack before I got home to fix it:

Text

1 23 45 67 89 1011 1213 1415 1617 18

04:31:10        145.6 MB        4.69 GB 04:42:04        153.54 MB       4.84 GB04:51:43        159.94 MB       4.99 GB 05:02:05        170.82 MB       5.16 GB05:09:19        205.57 MB       5.36 GB 05:21:16        210.54 MB       5.57 GB05:29:15        159.34 MB       5.72 GB 05:42:09        142.66 MB       5.86 GB05:51:28        192.65 MB       6.05 GB 06:01:38        203.06 MB       6.25 GB06:11:51        200.13 MB       6.44 GB 06:20:55        206.91 MB       6.65 GB06:31:38        177.75 MB       6.82 GB 06:38:35        172.82 MB       6.99 GB06:52:16        131.23 MB       7.12 GB 07:01:38        120.75 MB       7.23 GB07:11:27        156.79 MB       7.39 GB 07:21:50        133.41 MB       7.52 GB

Logs after I locked it down, but still getting hit:

Text

1 23 45 67 89 1011 1213 1415 1617 1819

03:36:16        679.19 KB       19.5 MB 03:46:26        632.98 KB       20.12 MB03:57:02        619.59 KB       20.72 MB 04:06:11        665.82 KB       21.37 MB04:16:33        640.04 KB       22 MB 04:26:04        803.64 KB       22.78 MB04:36:48        701.18 KB       23.47 MB 04:46:42        710.06 KB       24.16 MB04:56:10        795.03 KB       24.94 MB 05:06:25        750.56 KB       25.67 MB05:15:55        678.12 KB       26.33 MB 05:26:26        564.38 KB       26.88 MB05:36:05        403.87 KB       27.28 MB 05:45:53        528.09 KB       27.79 MB05:56:13        632.77 KB       28.41 MB 06:06:37        632.51 KB       29.03 MB06:13:57        620.63 KB       29.63 MB 06:25:45        707.88 KB       30.33 MB06:36:01        798.03 KB       31.11 MB

Seriously, it is worth looking into keeping it secure.

AND as I stated, once you get hit, no matter what you do, the ATTACK requests will not STOP!

Nick

In reply to a post by Lethe:

AND as I stated, once you get hit, no matter what you do, the ATTACK requests will not STOP!

There is a strong chance they will stop after a few days when people get bored..

In reply to a post by billford:

So the OS X firewall appears to pass incoming IPv6 ntpd requests, not sure it's worth worrying about?

For now, there is a good chance the attack is IPv4 based so you probably don't need to interrupt your weekend plans to get this to fixed, but it might be worth investigation at some point.

In reply to a post by Pipexer:

In reply to a post by Lethe:

AND as I stated, once you get hit, no matter what you do, the ATTACK requests will not STOP!

There is a strong chance they will stop after a few days when people get bored..

Unfortunately, they are not people - but bots, spoofed IP's and other mechanisms. I turned my modem off for 12 hours, and as soon as I plugged it in, off they go again. No matter what I did, non-stop incessant pounding on port 123.

My last post here - it was just a heads up to what will happen. If nobody agrees with me, then fair enough - I just hope nobody else gets attacked due to an open NTP server even if you *think* you will not.

Nick