CloudFlare's I'm Under Attack Mode (IUAM) is elegantly simple. When a site is under an application layer (Layer 7) distributed denial of service (DDoS) attack, the mode will return a challenge page to a visitor. The challenge requires the visitor's browser to answer a math problem which takes a bit of time to compute. Once successfully answered, the browser can request a cookie and won't be challenged again.
2 + 2 = Surprisingly Effective
IUAM has been incredibly successful at stopping Layer 7 attacks, but it's had a dirty little secret since it was first launched. While we'd suggested that the math problem the browser had to solve would be computationally complex, in reality it was incredibly simple: literally adding together two single-digit integers.
Several people over the last 6 months had written to us to let us know about this "critical vulnerability." They explained how easy it would be for an attacker to reverse engineer the math problem and create malware that could bypass the protection. Internally, we had a bet on how long it would take for some bad guy to actually do so. My money was on "never."
Good News/Bad News: I Lost the Bet
When Lee and I created Project Honey Pot back in 2004 we spent hundreds of engineering hours designing traps that were so random they were hard to identify. Even then, I secretly worried that an enterprising bad guy would recognize some pattern in the traps and be able to avoid them. We watched carefully for 9 years and no one ever took the time to do so. It was great, on one hand, since it meant that Project Honey Pot kept tracking attackers but, on the other, it meant it was never causing them enough trouble that they'd spend the engineering effort to defeat us. Lee and I learned the lesson: don't over-engineer too early.
Which brings me back to IUAM. This morning we got word from the great folks over at ESET that they'd detected malware specifically designed to bypass CloudFlare's IUAM. Called OutFlare -- how cool is it that we have malware named after us!! -- the malware reads our IUAM page, finds the simple math problem, and calculates the answer. It is hardly rocket science, but it was actually pretty thrilling to the whole CloudFlare team that we'd been so successful at stopping attackers that at least one of them took the time to reverse engineer this protection.
Proof of Work
Unlike me, some other engineers on CloudFlare's team had a suspicion that this day would come. They therefore had, waiting in the wings, code to increase the complexity of IUAM's challenges. The malware pulls the math equation off the page and computes the answer before posting back. The solution was easy: obfuscate the equation and run through some other tricks that make it hard to find the answer if you're not actually rendering the Javascript.
Today, after getting word that the simple version of IUAM had been reverse engineered by the OutFlare's malware, we pushed an update. If you're using IUAM there's nothing you need to do to take advantage of the new protection, we've already updated the protection rendering the OutFlare malware obsolete.
Going forward, we have plans if this scheme gets cracked. Specifically, we have an IUAM version that relies on a field of mathematics known as "proof of work" problems. These are difficult to compute answers for but easy to verify. A recent example of such a proof of work problem which has captured the imagination of much of the tech community is Bitcoin. The electronic currency requires a significant amount of computational time to find the answer to a problem, but once found each answer ("coin") is easy to verify.
In Bitcoin's case, the difficulty of the question is adjusted upward over time to compensate for increasing computing power and to control currency inflation. We can use the same premise to increase the "work" that an attacker needs to do when we detect a Layer 7 attack against a CloudFlare customer.
Arms Race? Bet on the Cloud
In these situations there's always a question of whether there will be an arms race between the attackers writing the malware and the good guys offering protection. In this case there may be, but I like our odds in such a war. As today's example demonstrated, because CloudFlare is deployed as a service and we can update our systems to adjust to new threats in realtime we have an asymmetrical advantage. Pity the poor malware writer who now has to reverse engineer the new IUAM protection and push a code change to all his bots. If he comes up with something effective, we'll just adapt again — instantly.
The history of such arms races suggests you should bet on the cloud to win. In the spam wars, spammers and anti-spam software makers were locked in an arms race that it looked like neither would win from the mid-90s through the mid-2000s. Then something changed: new services like MXLogic, MessageLabs, CloudMark, and Postini started delivering anti-spam not as software but as a "cloud" service. Not only were these services easier to install and administer than previous anti-spam software or appliances, they could also adjust to spammers in realtime. The result has been that today these services have largely won the spam wars.
One More Thing
One more thing with regard to OutFlare. While the malware was able to read and pass the simple math challenge, that is only one layer of IUAM's protection. On the server side, CloudFlare still tracked all requests and, for devices that created a statistically high number of connections, we automatically imposed rate limits and other mitigation techniques. In other words, even without the fix we made, our customers were protected from the attack.
Thanks again to our friends at ESET for alerting us to the new OutFlare malware. We'll keep our eyes open to any new variants and, as they inevitably arise, we'll continue to adapt to ensure that all CloudFlare customers are always a step ahead of the web's nastiest threats.