Heuristics and Rules: Why We Built a New Old WAF

We just rolled out an update to CloudFlare's Web Application Firewall (WAF). Previously, CloudFlare's WAF has received criticism from people who have tested it and found that it didn't behave as traditional WAFs are expected to. That contrasted with the real world experience of users who saw our WAF virtually eliminate actual web threats. Seemingly paradoxically, all of the following are correct: the criticisms weren't wrong and CloudFlare's original WAF was working as designed. I'll explain below, but first some history.

I started working in the Internet security space in 1998. At the time, I was interested in the emerging problem of email spam. The state of the art technology in 1998 for battling spam was an open source project called SpamAssassin. SpamAssassin was a rules-based email filter engine. At a simplistic level, it had a list of things that were likely to appear in spam messages (e.g., the word "viagra"), each with a score. When a new message arrived, it was run through the engine, each hit incremented the score, and a message was considered "spam" if the score exceeded a certain threshold.

This rules-based approach had at least two major flaws. First, it was very difficult to configure because every organization's needs were different. For instance, while "viagra" may have been an indication of spam in many cases, if you were setting up SpamAssassin for the Pfizer corporation (which manufactures the drug) then "viagra" may actually have been an indication of non-spamminess.

Second, the bad people had access to the same standard rule sets as the good people. That meant it was easy for them to craft messages that could bypass the filters. Remember spam messages that spelled "viagra" like "v1@6r@"? It turns out that there are 1,300,925,111,156,286,160,896 different ways you can spell the word "viagra". Since the rules based approach to SpamAssassin was brittle, it was easy for attackers to get around the rules.

In 2002, before he started Y Combinator, Paul Graham wrote an influential essay called "A Plan for Spam." The essay suggested that the rules-based approach for email filtering was headed down the wrong path. Instead, Paul argued anti-spam systems should be based on heuristics. Specifically, he outlined a Bayesian statistics-based approach where spam filters could be self-learning based on a statistical calculation of "normal" adjusted for an individual user's feedback marking messages as "spam" and "not spam."

In 2003, after an overwhelming response to his essay, Paul organized the first MIT Anti-Spam Conference. From that conference, many of the companies that would go on to largely solve the email spam problem (at least from the perspective of end users) would emerge. Paul turned out to be right and whatever email program you're using today it is almost guaranteed that a heuristic engine is making the decision on what ends up in your spam folder.

I attended the MIT Anti-Spam Conference for several years and, in 2005, Paul invited me to give a talk on Project Honey Pot which, in many ways, was the open source project that served as the initial inspiration for CloudFlare. The MIT Anti-Spam Conference was also where I first met John Graham-Cumming, who now works on CloudFlare's team and was the principal architect behind our WAF update. More on that in a second, but first a bit about the state of traditional WAFs today.

A decade after the first MIT Spam Conference, the archetype of the traditional WAF is an open source project known as ModSecurity. Most commercial WAFs are based around the same fundamental design. At a simplistic level, ModSecurity has a list of things that are likely to appear in different types of web-based attacks, each with a score. When a new web request arrives, it is run through the ModSecurity engine, each hit increments a score, and then the request is considered a threat if the score exceeds a certain threshold.

Sound familiar? It should, it's the same rules-based approach that SpamAssassin used. It also suffers from the same challenges. It is difficult to configure on a per-organization basis, partially explaining why the market for traditional WAFs has remained remarkably small. Moreover, bad people have access to the standard rule sets meaning they can craft attacks that easily bypass them.

At CloudFlare, we thought we'd skip to the inevitable end of the story and build a WAF based on heuristics from the beginning. Rather than rely on brittle rules, CloudFlare was designed to look at the totality of traffic across all our customers' sites and then use that data to stop actual attacks. We also tailored the protection on a site-by-site basis. For example, a chunk of SQL posted to a site like Stack Exchange may not be a threat while a chunk of SQL posted to a cooking blog may be. At scale, if enough websites used our service, we knew we could provide an extremely high level of effective protection driven by data and heuristics.

That approach has worked. We have more than a million happy users who report that CloudFlare protects them from a wide range of attacks. However, over the last four years we've also realized there is a place for rules.

Some of this is purely marketing. Heuristic-based approaches are notoriously difficult to test. Sign up a new site for CloudFlare and start posting a laundry list of SQL injection attacks from a connection with browsing behavior across other CloudFlare sites that indicates it is not malicious and our systems will tend to come to the conclusion that this is a site covering SQL injection attacks and let the posts through. That actually isn't wrong, but it isn't very useful when you're trying to run a test to see that we block actual attacks. It also complicates tests for "check-the-box" security audits like PCI.

Beyond mere testing, rules also can be good when applied appropriately. We've seen several instances where a new attack will emerge and we deploy a rule to quickly thwart it. While we've had the ability to do that ourselves, we haven't previously allowed our customers, partners (e.g., hosting providers), or trusted security consultants to do the same.

We spent the last 6 months building a traditional, rules-based WAF to augment CloudFlare's existing next generation, heuristics-based WAF. The new WAF is included by default with every Pro and better plan ($20/month for the first site, $5/month for each additional site). In the process, we also took the opportunity to rearchitect how some of our security systems work in order to provide more visibility and flexibility. You'll see that increasingly reflected in our UI over the next few months.

When we set out to add a traditional WAF, our design goal was to build a system that was fully user configurable, accepted the existing ModSecurity rule configuration language, added less than 1ms of latency to requests when processing the full OWASP ruleset or its equivalent, would update worldwide within less than 30 seconds of a user configuration change, provided data and feedback on rules as they were triggered, and would scale to handle the billions of requests CloudFlare processes every day. I'm excited to say we exceeded our goal.

The new rules-based WAF is available now and augments the existing heuristics–based WAF. Over the next few weeks we'll be publishing a series of technical blog posts on what went into building a rules-based WAF that is fast, flexible, and can scale to handle CloudFlare's level of traffic. In the meantime, feel free to try out the WAF and let us know what you think.