Mitigating Bot Attacks against Cloudflare

The word “bots” on the Internet is a fairly loaded one. My earliest ‘bot’ experience was on IRC, where bots were quite helpful in making sure your favorite channel didn’t get taken over by malicious users and allowed for fun games of trivia. Around five years ago, “bots” were often referencing text chats in combination with AI and messaging platforms/apps as a new way to interact with customers. Today most of the connotations around bots on the Internet, particularly in the security space, are negative, and we have a number of vendors offering new ways to detect and block bots.

In its most simple form, a bot is an automated piece of software that replaces human interaction. In the examples above, this is done, so we can scale a process to be faster or more extensive than a single manual action. Search Engine bots exist because it is impossible (or at the very least, impractical) to crawl the Internet one curl at a time. The benefit of scale can be used for both good and for bad, by attacking a property on the Internet. Bots are used for attacks at scale — they can be deployed to attack an improperly configured API, to take a site down, or take a list of pwned credentials and see which work on a login endpoint before exfiltrating data.

The global pandemic has caused a global shortage of microchips. In 2020, Microsoft, Nvidia, and Sony all launched high demand, low supply gaming devices or video cards that remain scarce months later. This type of environment is ripe for all manner of bot activity — from the hobbyist trying to gain access to one of these items for their own use, to the hoarder who is actively trying to buy as many as possible at retail price for resale on the much higher priced open market. A bot will scrape for inventory and then script through the add to cart/buy process much faster than a legitimate user behind a browser. This can lead to frustration at the very least, but can also cause a Layer 7 DDoS on the application which compounds the issue. Nvidia was forced to manually and retroactively cancel sales to bots during one of its releases as a result. While gaming is a fun distraction, this same set of circumstances can also unfortunately affect vaccine distribution.

The current generation of Cloudflare Bot Management was released in 2019. The product was originally borne out of an internal experiment to see if we could predict whether a given request would solve a challenge using our network data and machine learning (ML) models. This eventually led to our first Bot Score and full-fledged Bot Management product, which has gone through a number of enhancements and iterations and has been an extremely successful product in our customers’ security toolkit. Today we enhanced it further for our Pro and Business plans with Super Bot Fight Mode.

During the initial internal development stages, we discovered a problem on one of our own customer-facing sites. For reasons that are still not entirely clear (but likely malicious), attackers would put junk data into forms on various Cloudflare landing pages where legitimate users would enter their information to register for an event or promotion, or to be contacted by our sales team. Initially this was a nuisance, as it would cause our teams to sort through and delete bad data. We also had to go back and identify the legitimate submissions. As the problem grew, it became so significant that it impacted our back end provider (where the forms were being correlated) and entered into our CRM. It turned from nuisance into a form of application DDoS which had to be carefully stopped to avoid false positives, i.e., blocks of legitimate submissions.

Word of the problem reached our engineering team, and it became clear that this was a great place to launch our first customer and dogfood our product. At first, attackers were not attempting to hide very much. They used out-of-the-box scripts and oftentimes didn’t even bother to change obvious signals like the User Agent header or spread the attack over a significant number of IPs or ASNs. As we added heuristics into the system and tuned the ML model, the attackers adjusted their tactics. Over time, our full array of tools became necessary as the attackers continued to adapt. An example of a more recent attack is below; during the attack, automated traffic increased to 7x over normal for roughly 30 minutes. Scaling would have likely been an issue for our downstream provider, where we ultimately sent API calls on form submission. During this attack, our Anomaly Detection (which is the basis for our new API abuse detection) did the majority of the heavy lifting, accounting for 42% of the detections.

We also battle bots on our dashboard across many of our user-facing APIs. A common pattern of abuse we have seen over the years is that a malicious user signs up numerous domains, often from free TLDs, and often spread across multiple accounts. The user takes these domains and engages in a form of Search Engine Optimization (SEO) spam. It is believed that one of the ways you increase your search ranking is to have many sites linking to your domain. SEO spammers sign up large numbers of domains and create thousands of hostname records within and then charge unscrupulous website owners to crosslink from each of these domains.

SEO shenanigans aside, this presents two technical issues for our systems. First: if enough of these domains and records are created at the same time, it can cause DNS Pump, our system designed to push customer DNS records to 200+ locations in a manner of seconds, to slow down. We pride ourselves on making DNS propagation seem like magic and our customers are used to and rely on this speed. Second, there is the practical cost of storing these records on each of our edge servers for no purpose other than to possibly trick search engines into thinking a site is popular.

This type of abuse only works through scale. An SEO spammer must be able to sign up a lot of domains in an efficient and automated fashion in order to sell their services. So we threw Bot Management at the problem. Originally we slowed the problem down through Rate Limiting as well as preventing a given IP from signing up numerous accounts over a short period of time, but attackers persisted. We challenged new account signups that appeared automated through our Bot Management product and the problem immediately went away.

Another part of Cloudflare’s dashboard that deals with bots is our billing system. Attackers will use our payment card processing system to test stolen credit card numbers for validity, which allows them to use or resell the card for use on another, more expensive transaction at a different site. Once again, speed and scale are key here. An attacker does not want to test one or two card numbers — they want to test dozens or hundreds and automate the process. Our billing engineering team uses Bot Management to trigger challenges or blocks when a user adds or changes a payment method to stop this attack. The billing team also passes the bot score to our third party payment provider’s own fraud detection system (via Cloudflare Workers), which incorporates the score into its analysis for manual transaction reviews.

Our Bot Management products are successful because of the sheer number of customers and amount of traffic on our network. With approximately 25M Internet properties protected by Cloudflare, we are uniquely positioned to gather these signals and interpret them into actionable intelligence. Today’s Super Bot Fight Mode release extends this capability to Pro and Business but will also provide a boost across all Bot Management users. As Pro and Business zones start to challenge and block bots based directly on bot signals, this data more directly trains our models versus the inferred data we use for non bot-related challenges and blocks. This diversity of signal and scale of data on a global platform gives us confidence in our ability not just to block bots today, but in the future as well.

We are excited to get early access feedback from our Enterprise customers on our API detection and abuse models. APIs present different challenges than those posed by web-facing properties, but our experience in building our Anomaly Detection platform for Bot Management allows us to use related techniques to identify API endpoints and detect anomalies within automated traffic. The inclusion of these new techniques with our existing security portfolio is part of our commitment to providing our customers with the best tools on a single platform, no matter what traffic they have on their domain. The combination of all of our tools allows for flexible response as well — customers can block DDoS attacks and definite bots, challenge likely bots, and use our rate limiting to target suspicious API traffic.