CloudFlare's I'm Under Attack Mode (IUAM) is elegantly simple. When a site is under an application layer (Layer 7) distributed denial of service (DDoS) attack, the mode will return a challenge page to a visitor. The challenge requires the visitor's browser to answer a math problem which takes a bit of time to compute. Once successfully answered, the browser can request a cookie and won't be challenged again.

IUAM has been incredibly successful at stopping Layer 7 attacks, but it's had a dirty little secret since it was first launched. While we'd suggested that the math problem the browser had to solve would be computationally complex, in reality it was incredibly simple: literally adding together two single-digit integers.

Several people over the last 6 months had written to us to let us know about this "critical vulnerability." They explained how easy it would be for an attacker to reverse engineer the math problem and create malware that could bypass the protection. Internally, we had a bet on how long it would take for some bad guy to actually do so. My money was on "never."

When Lee and I created Project Honey Pot back in 2004 we spent hundreds of engineering hours designing traps that were so random they were hard to identify. Even then, I secretly worried that an enterprising bad guy would recognize some pattern in the traps and be able to avoid them. We watched carefully for 9 years and no one ever took the time to do so. It was great, on one hand, since it meant that Project Honey Pot kept tracking attackers but, on the other, it meant it was never causing them enough trouble that they'd spend the engineering effort to defeat us. Lee and I learned the lesson: don't over-engineer too early.

Which brings me back to IUAM. This morning we got word from the great folks over at ESET that they'd detected malware specifically designed to bypass CloudFlare's IUAM. Called OutFlare -- how cool is it that we have malware named after us!! -- the malware reads our IUAM page, finds the simple math problem, and calculates the answer. It is hardly rocket science, but it was actually pretty thrilling to the whole CloudFlare team that we'd been so successful at stopping attackers that at least one of them took the time to reverse engineer this protection.

Unlike me, some other engineers on CloudFlare's team had a suspicion that this day would come. They therefore had, waiting in the wings, code to increase the complexity of IUAM's challenges. The malware pulls the math equation off the page and computes the answer before posting back. The solution was easy: obfuscate the equation and run through some other tricks that make it hard to find the answer if you're not actually rendering the Javascript.

Today, after getting word that the simple version of IUAM had been reverse engineered by the OutFlare's malware, we pushed an update. If you're using IUAM there's nothing you need to do to take advantage of the new protection, we've already updated the protection rendering the OutFlare malware obsolete.

Going forward, we have plans if this scheme gets cracked. Specifically, we have an IUAM version that relies on a field of mathematics known as "proof of work" problems. These are difficult to compute answers for but easy to verify. A recent example of such a proof of work problem which has captured the imagination of much of the tech community is Bitcoin. The electronic currency requires a significant amount of computational time to find the answer to a problem, but once found each answer ("coin") is easy to verify.

In Bitcoin's case, the difficulty of the question is adjusted upward over time to compensate for increasing computing power and to control currency inflation. We can use the same premise to increase the "work" that an attacker needs to do when we detect a Layer 7 attack against a CloudFlare customer.

In these situations there's always a question of whether there will be an arms race between the attackers writing the malware and the good guys offering protection. In this case there may be, but I like our odds in such a war. As today's example demonstrated, because CloudFlare is deployed as a service and we can update our systems to adjust to new threats in realtime we have an asymmetrical advantage. Pity the poor malware writer who now has to reverse engineer the new IUAM protection and push a code change to all his bots. If he comes up with something effective, we'll just adapt again — instantly.

The history of such arms races suggests you should bet on the cloud to win. In the spam wars, spammers and anti-spam software makers were locked in an arms race that it looked like neither would win from the mid-90s through the mid-2000s. Then something changed: new services like MXLogic, MessageLabs, CloudMark, and Postini started delivering anti-spam not as software but as a "cloud" service. Not only were these services easier to install and administer than previous anti-spam software or appliances, they could also adjust to spammers in realtime. The result has been that today these services have largely won the spam wars.

One more thing with regard to OutFlare. While the malware was able to read and pass the simple math challenge, that is only one layer of IUAM's protection. On the server side, CloudFlare still tracked all requests and, for devices that created a statistically high number of connections, we automatically imposed rate limits and other mitigation techniques. In other words, even without the fix we made, our customers were protected from the attack.

Thanks again to our friends at ESET for alerting us to the new OutFlare malware. We'll keep our eyes open to any new variants and, as they inevitably arise, we'll continue to adapt to ensure that all CloudFlare customers are always a step ahead of the web's nastiest threats.

Yesterday I posted a post mortem on an outage we had Saturday. The outage was caused when we applied an overly aggressive rate limit to traffic on our network while battling a determined DDoS attacker. In the process of writing it I mentioned that we'd seen a 65Gbps DDoS earlier on Saturday. I've received several questions since that all go something like: "65Gbps DDoS!? Who launches such an attack and how do you defend yourself against it?!" So I thought I'd give a bit more detail.

A 65Gbps DDoS is a big attack, easily in the top 5% of the biggest attacks we see. The graph below shows the volume of the attack hitting our EU data centers (the green line represents inbound traffic). When an attack is 65Gbps that means every second 65 Gigabits of data is sent to our network. That's the equivalent data volume of watching 3,400 HD TV channels all at the same time. It's a ton of data. Most network connections are measured in 100Mbps, 1Gbps or 10Gbps so attacks like this would quickly saturate even a large Internet connection.

At CloudFlare, an attack needs to get over about 5Gbps to set off alarms with our ops team. Even then, our automated network defenses usually stop attacks without the need of any manual intervention. When an attack gets up in the tens of Gigabits of data per second, our ops team starts monitoring the attack: applying filters and shifting traffic to ensure the attacked customer's site stays online and none of the rest of our network is affected.

So how does an attacker generate 65Gbps of traffic? It is highly unlikely that the attacker has a single machine with a big enough Internet connection to generate that much traffic on its own. One way to generate that much traffic is through a botnet. A botnet is a collection of PCs that have been compromised with a virus and can be controlled by what is known as a botnet herder.

Botnet herders will often rent out access to their botnets, often billing in 15 minute increments (just like lawyers). Rental prices depend on the size of the botnets. Traditionally, email spammers purchased time on botnets in order to send their messages to appear to come from a large number of sources. As email spam has become less profitable with the rise of better spam filters, botnet herders have increasingly turned to renting out their networks of compromised machines to attackers wanting to launch a DDoS attack.

To launch a 65Gbps attack, you'd need a botnet with at least 65,000 compromised machines each capable of sending 1Mbps of upstream data. Given that many of these compromised computers are in the developing world where connections are slower, and many of the machines that make up part of a botnet may not be online at any given time, the actual size of the botnet necessary to launch that attack would likely need to be at least 10x that size. While by no means unheard of, that's a large botnetand using all its resources to launch a DDoS risks ISPs detecting many of the compromised machines and taking them offline.

Since renting a large botnet can be expensive and unwieldy, attackers typically look for additional ways to amplify the size of their attacks. The attack on Saturday used one such amplification technique called DNS reflection. To understand how these work, you need to understand a bit about how DNS works.

When you first sign up for an Internet connection, your ISP will provide you with a recursive DNS server, also known as a DNS resolver. When you click on a link, your computer sends a lookup to your ISP's DNS resolver. The lookup is asking a question, like: what is the IP address of the server for cloudflare.com? If the DNS resolver you query knows the answer, because someone has already asked it recently and the answer is cached, it responds. If it doesn't, it passes the request on to the authoritative DNS for the domain.

Typically, an ISP's DNS resolvers are setup to only answer requests from the ISP's clients. Unfortunately, there are a large number of misconfigured DNS resolvers that will accept queries from anyone on the Internet. These are known as "open resolvers" and they are a sort of latent landmine on the Internet just waiting to explode when misused.

DNS queries are usually sent via the UDP protocol. UDP is a fire-and-forget protocol, meaning that there is no handshake to establish that where a packet says it is coming from actually is where it is coming from. This means, if you're an attacker, you can forge the header of a UDP packet to say it is coming from a particular IP you want to attack and send that forged packet to an open DNS resolver. The DNS resolver will reply back with a response to the forged IP address with an answer to whatever question was asked.

To amplify an attack, the attacker asks a question that will result in a very large response. For example, the attacker may request all the DNS records for a particular zone. Or they may request the DNSSEC records which, often, are extremely large. Since resolvers typically have relatively high bandwidth connections to the Internet, they have no problem pumping out tons of bytes. In other words, the attacker can send a relatively small UDP request and use open resolvers to fire back at an intended target with a crippling amount of traffic.

One of the great ironies when we deal with these attacks is we'll often get an email from the owner of the network where an open resolver is running asking us to shut down the attack our network is launching against them. They're seeing a large number of UDP packets with one of our IPs as the source coming in to their network and assume we're the ones launching it. In fact, it is actually their network which is being used to launch an attack against us. What's great is that we can safely respond and ask them to block all DNS requests originating from our network since our IPs should never originate a DNS request to a resolver. Not only does that solve their problem, but it means there's a smaller pool of open resolvers that can be used to target sites onCloudFlare's network.

There have been a number of efforts to clean up open resolvers that are currently active. Unfortunately, it is slow going and the default installation of many DNS clients still has them open by default. While we actively reach out to the worst offenders to protect our network, to protect the Internet generally there will need to be a concerted effort to clean up open DNS resolvers.

In terms of stopping these attacks, CloudFlare uses a number of techniques. It starts with our network architecture. We use Anycast which means the response from a resolver, while targeting one particular IP address, will hit whatever data center is closest. This inherently dilutes the impact of an attack, distributing its effects across all 23 of our data centers. Given the hundreds of gigs of capacity we have across our network, even a big attack rarely saturates a connection.

At each of our facilities we take additional steps to protect ourselves. We know, for example, that we haven't sent any DNS inquiries out from our network. We can therefore safely filter the responses from DNS resolvers: dropping the response packets from the open resolvers at our routers or, in some cases, even upstream at one of our bandwidth providers. The result is that these types of attacks are relatively easily mitigated.

What was fun to watch was that while the customer under attack was being targeted by 65Gbps of traffic, not a single packet from that attack made it to their network or affected their operations. In fact, CloudFlare stopped the entire attack without the customer even knowing there was a problem. From the network graph you can see after about 30 minutes the attacker gave up. We think that's pretty cool and, as we continue to expand our network, we'll get even more resilient to attacks like this one.

These checks take about 5 seconds to perform, and during that time the visitor's (or attacker's) web browser could be performing other work. Part of the verification takes the form of JavaScript that CloudFlare delivers to the browser. Currently, that JavaScript only performs the verification checks, but it could do more. After the checks the visitor is forwarded on to the web site.

In the past, many distributed computing efforts have harnessed the power of machines across the Internet to do collaborative work. The SETI@Home project looks for extraterrestrial life, Folding@Home looks at protein folding to help research into drugs and diseases, and GIMPS is looking for particular prime numbers. Wikipedia has a long list of such projects.

We think that I'm Under Attack mode version 2.0 could be an "I'm Doing Some Good" mode by including a distributed computation in the JavaScript that's delivered as part of dealing with attacks. The project would need to be able to broken down into chunks that run 5 seconds at a time, and be written in JavaScript. It could be run across all web sites that are under attack and in the browsers of legitimate and attacking users potentially using the resources of evil doers for a good purpose.

The end users wouldn't see any difference from the way I'm Under Attack Mode works today, but a little bit of compute power that's not being used while checks for malicious behavior are made could be put to gooduse. Put together, many thousands of machines could be working on a distributed computing project without any effort on the part of end users. And without any extra impact on web site owners.

The hard question to answer is... which project?

Rather than come up with our own ideas we'd like to throw this open to the community for suggestions. The best (and most implementable) solution will be picked by CloudFlare and implemented to start turning a bad situation into a good one.

Make suggestions in the comments below.

CloudFlare provides a broad level of protection from a wide range of attacks. We do this while minimizing false positives or annoyances to legitimate customers. CloudFlare didn't begin as a DDoS mitigation service, but we've rapidly found that we are good at protecting sites from these attacks. Today we're offering a new security mode to make our DDoS protection even better.

In the OSI model, traditional DDoS attacks targeted the Layer 4. The so called "transport" layer of the network stack specifies the protocol (e.g., TCP or UDP). These attacks flood an interface with garbage traffic in order to overwhelm it's resources in one way or another. Usually, the attack fills up the capacity of a network switch or overwhelms a server's network card or CPU's ability to handle the traffic.

CloudFlare has largely mitigated these attacks by building out significant capacity across our network. We have fat pipes and lots of machines to absorb floods of traffic. We also make broad use of the Anycast protocol which has the effect of scattering the load of a distributed attack across multiple data centers, reducing the exposure of potential single point of failure. The result is that no packets from a traditional Layer 4 attack will ever reach a site behind CloudFlare.

A new breed of attacks targets Layer 7, the "application" layer. These attacks focus on specific characteristics of web applications that present bottlenecks. For example, the so-called Slow Read attack sends packets very slowly across multiple connections. Since Apache opens a new thread for each connection, and since connections are maintained as long as there is some traffic being sent, you can overwhelm a web server by exhaust its thread pool relatively easily.

CloudFlare has protections in place against many of these attacks, and in real world experiences we generally reduce the HTTP attack traffic by about 90%. For most attacks and most of our customers, this has been enough to keep them online. However, the 10% of traffic that gets through our traditional protections can still be overwhelming to either customers with limited resources or in the face of very large attacks. We wanted to help in these cases too, so today we're announcing something new.

Introducing "I'm Under Attack Mode." The name is pretty self-explanatory: it's a new security level you can set for your site when you're under attack. The effect is that we will add an additional set of protections to stop potentially malicious HTTP traffic from being passed to your server. While we perform a number of additional checks, the only thing noticeable to legitimate visitors to your site is that when they first arrive they'll see an interstitial page for about 5 seconds while checks are complete. Think of it as a challenge where the tests are automatic and visitors never need to fill in a CAPTCHA.

After verified as legitimate by the automated tests, visitors are able to browse your site unencumbered and won't see typically the test page again. Javascript and cookies are required for the tests and recording the fact that the tests were correctly passed. We've also designed the new checks to not block search engine crawlers, your existing allowlists, and other pre-vetted traffic. As a result, enabling I'm Under Attack Mode will not negatively impact your SEO or known legitimate visitors. What's also cool is that data on attack traffic that doesn't pass the automatic checks is fed back into CloudFlare's system to further enhance our traditional protections.

While CloudFlare did not start as a DDoS mitigation service we have realized this is an area where we can provide a lot of benefit in an easy and affordable way. I'm Under Attack Mode is the first of several new features we'll be releasing over the coming month to offer a full gauntlet of DDoS protection. Stay tuned.