Yesterday I posted a post mortem on an outage we had Saturday. The outage was caused when we applied an overly aggressive rate limit to traffic on our network while battling a determined DDoS attacker. In the process of writing it I mentioned that we'd seen a 65Gbps DDoS earlier on Saturday. I've received several questions since that all go something like: "65Gbps DDoS!? Who launches such an attack and how do you defend yourself against it?!" So I thought I'd give a bit more detail.
What Constitutes a Big DDoS?
A 65Gbps DDoS is a big attack, easily in the top 5% of the biggest attacks we see. The graph below shows the volume of the attack hitting our EU data centers (the green line represents inbound traffic). When an attack is 65Gbps that means every second 65 Gigabits of data is sent to our network. That's the equivalent data volume of watching 3,400 HD TV channels all at the same time. It's a ton of data. Most network connections are measured in 100Mbps, 1Gbps or 10Gbps so attacks like this would quickly saturate even a large Internet connection.
At CloudFlare, an attack needs to get over about 5Gbps to set off alarms with our ops team. Even then, our automated network defenses usually stop attacks without the need of any manual intervention. When an attack gets up in the tens of Gigabits of data per second, our ops team starts monitoring the attack: applying filters and shifting traffic to ensure the attacked customer's site stays online and none of the rest of our network is affected.
So You Want to Launch a DDoS
So how does an attacker generate 65Gbps of traffic? It is highly unlikely that the attacker has a single machine with a big enough Internet connection to generate that much traffic on its own. One way to generate that much traffic is through a botnet. A botnet is a collection of PCs that have been compromised with a virus and can be controlled by what is known as a botnet herder.
Botnet herders will often rent out access to their botnets, often billing in 15 minute increments (just like lawyers). Rental prices depend on the size of the botnets. Traditionally, email spammers purchased time on botnets in order to send their messages to appear to come from a large number of sources. As email spam has become less profitable with the rise of better spam filters, botnet herders have increasingly turned to renting out their networks of compromised machines to attackers wanting to launch a DDoS attack.
To launch a 65Gbps attack, you'd need a botnet with at least 65,000 compromised machines each capable of sending 1Mbps of upstream data. Given that many of these compromised computers are in the developing world where connections are slower, and many of the machines that make up part of a botnet may not be online at any given time, the actual size of the botnet necessary to launch that attack would likely need to be at least 10x that size. While by no means unheard of, that's a large botnetand using all its resources to launch a DDoS risks ISPs detecting many of the compromised machines and taking them offline.
Amplifying the Attacks
Since renting a large botnet can be expensive and unwieldy, attackers typically look for additional ways to amplify the size of their attacks. The attack on Saturday used one such amplification technique called DNS reflection. To understand how these work, you need to understand a bit about how DNS works.
When you first sign up for an Internet connection, your ISP will provide you with a recursive DNS server, also known as a DNS resolver. When you click on a link, your computer sends a lookup to your ISP's DNS resolver. The lookup is asking a question, like: what is the IP address of the server for cloudflare.com? If the DNS resolver you query knows the answer, because someone has already asked it recently and the answer is cached, it responds. If it doesn't, it passes the request on to the authoritative DNS for the domain.
Typically, an ISP's DNS resolvers are setup to only answer requests from the ISP's clients. Unfortunately, there are a large number of misconfigured DNS resolvers that will accept queries from anyone on the Internet. These are known as "open resolvers" and they are a sort of latent landmine on the Internet just waiting to explode when misused.
DNS queries are usually sent via the UDP protocol. UDP is a fire-and-forget protocol, meaning that there is no handshake to establish that where a packet says it is coming from actually is where it is coming from. This means, if you're an attacker, you can forge the header of a UDP packet to say it is coming from a particular IP you want to attack and send that forged packet to an open DNS resolver. The DNS resolver will reply back with a response to the forged IP address with an answer to whatever question was asked.
To amplify an attack, the attacker asks a question that will result in a very large response. For example, the attacker may request all the DNS records for a particular zone. Or they may request the DNSSEC records which, often, are extremely large. Since resolvers typically have relatively high bandwidth connections to the Internet, they have no problem pumping out tons of bytes. In other words, the attacker can send a relatively small UDP request and use open resolvers to fire back at an intended target with a crippling amount of traffic.
Mitigating DNS Reflection Attacks
One of the great ironies when we deal with these attacks is we'll often get an email from the owner of the network where an open resolver is running asking us to shut down the attack our network is launching against them. They're seeing a large number of UDP packets with one of our IPs as the source coming in to their network and assume we're the ones launching it. In fact, it is actually their network which is being used to launch an attack against us. What's great is that we can safely respond and ask them to block all DNS requests originating from our network since our IPs should never originate a DNS request to a resolver. Not only does that solve their problem, but it means there's a smaller pool of open resolvers that can be used to target sites onCloudFlare's network.
There have been a number of efforts to clean up open resolvers that are currently active. Unfortunately, it is slow going and the default installation of many DNS clients still has them open by default. While we actively reach out to the worst offenders to protect our network, to protect the Internet generally there will need to be a concerted effort to clean up open DNS resolvers.
In terms of stopping these attacks, CloudFlare uses a number of techniques. It starts with our network architecture. We use Anycast which means the response from a resolver, while targeting one particular IP address, will hit whatever data center is closest. This inherently dilutes the impact of an attack, distributing its effects across all 23 of our data centers. Given the hundreds of gigs of capacity we have across our network, even a big attack rarely saturates a connection.
At each of our facilities we take additional steps to protect ourselves. We know, for example, that we haven't sent any DNS inquiries out from our network. We can therefore safely filter the responses from DNS resolvers: dropping the response packets from the open resolvers at our routers or, in some cases, even upstream at one of our bandwidth providers. The result is that these types of attacks are relatively easily mitigated.
What was fun to watch was that while the customer under attack was being targeted by 65Gbps of traffic, not a single packet from that attack made it to their network or affected their operations. In fact, CloudFlare stopped the entire attack without the customer even knowing there was a problem. From the network graph you can see after about 30 minutes the attacker gave up. We think that's pretty cool and, as we continue to expand our network, we'll get even more resilient to attacks like this one.