Millions of customers around the world use Cloudflare to keep their applications safe by blocking bot traffic to their website. We block an average of 336 million requests per day for self-service customers using a service called Super Bot Fight Mode. It is a crucial part of how customers keep their websites online.

While most customers use Cloudflare’s Verified Bot directory to securely allow good, automated traffic, some customers also like to write their own localized integration scripts to crawl and update their website, or perform other necessary maintenance functions. Because these bots are only used on a single website, they don’t fit our verified bot criteria the way a Google or Bing crawler does. This makes Super Bot Fight Mode difficult to manage for these types of customers.

Previously, Super Bot Fight Mode ran as an independent service on our global network and other Cloudflare security services were unable to affect its configuration. To solve this, we’ve rewritten Super Bot Fight Mode behind the scenes. It’s now a new managed ruleset in the new WAF, just like the OWASP Core Ruleset or the Cloudflare Managed Ruleset. This doesn’t change the interface, but brings Super Bot Fight Mode closer to where customers are managing their other security exceptions.

As we speak, the WAF team is carefully migrating all self-serve customers from our old Firewall Rules system to a new system. This new system, called Custom Rules, simplifies the exception process in the rules you write with no other changes or loss of functionality. In the old system we had two separate actions, “allow” and “bypass”. In the new Custom Rules, there’s only one action called “skip”. Rules that “skip” traffic can skip the rest of your custom rules (just like an “allow” rule would) and other Cloudflare services. As Cloudflare customers are given the “Skip” action, you will be able to see the option available to “skip” Super Bot Fight Mode. Here’s an example:

While we spoke to customers about their use cases for skipping Super Bot Fight Mode, one use-case kept popping up that didn’t quite fit the rest: WordPress Loopback requests. As many people know, as part of WordPress’ self-diagnostic capabilities, a WordPress site will make automated requests back to itself over the Internet to confirm its reachability and functionality. These loopback diagnostics can come from dozens of different community developed plugins, each implementing loopback requests slightly differently. To help accommodate an ever-growing diversity in diagnostic tools used in WordPress, we have added a simple configuration option to securely allow these loop-back requests.

In the future, we will be integrating this feature with the Cloudflare WordPress plugin to make it even easier to use WordPress with Cloudflare.

Self-serve customers with Custom Rules can create “Skip” rules to create exceptions for Super Bot Fight Mode today. We are currently rolling out Custom Rules to all of our customers. If you do not see this option available now, you should expect to see it in the next several weeks. If the lack of flexibility has prevented you from using Super Bot Fight Mode in the past, please log into the Cloudflare dashboard and try it with these new skip rules!

While we’ve added flexibility to customers’ Super Bot Fight Mode deployments, we know that Free plan customers want the same level of customization that self-serve customers do. Now that our migration of Super Bot Fight Mode to the new WAF is complete, we plan to do the same for the original Bot Fight Mode to allow more free customers than ever before to join us in the fight against bots.

Once a year, we pull data from our Bot Fight Mode to determine the number of trees we can donate to our partners at One Tree Planted. It's part of the commitment we made in 2019 to deter malicious bots online by redirecting them to a challenge page that requires them to perform computationally intensive, but meaningless tasks. While we use these tasks to drive up the bill for bot operators, we account for the carbon cost by planting trees.

This year when we pulled the numbers, we saw something exciting. While the number of bot detections has gone significantly up, the time bots spend in the Bot Fight Mode challenge page has gone way down. We’ve observed that bot operators are giving up quickly, and moving on to other, unprotected targets. Bot Fight Mode is getting smarter at detecting bots and more efficient at deterring bot operators, and that’s a win for Cloudflare and the environment.

We’ve seen two changes this year in the Bot Fight Mode results. First, the time attackers spend in Bot Fight Mode challenges has reduced by 166%. Many bot operators are disconnecting almost immediately now from Cloudflare challenge pages. We expect this is because they’ve noticed the sharp cost increase associated with our CPU intensive challenge and given up. Even though we’re seeing individual bot operators give up quickly, Bot Fight Mode is busier than ever. We’re issuing six times more CPU intensive challenges per day compared to last year, thanks to a new detection system written using Cloudflare’s ruleset engine, detailed below.

When Bot Fight Mode launched, we highlighted one of our core detection systems:

“Handwritten rules for simple bots that, however simple, get used day in, day out.”

Some of them are still very simple. We introduce new simple rules regularly when we detect new software libraries as they start to source a significant amount of traffic. However, we started to reach the limitations of this system. We knew there were sophisticated bots out there that we could identify easily, but they shared enough overlapping traits with good browser traffic that we couldn’t safely deploy new rules to block them safely without potentially impacting our customers’ good traffic as well.

To solve this problem, we built a new rules system written on the same highly performant Ruleset Engine that powers the new WAF, Transform Rules, and Cache Rules, rather than the old Gagarin heuristics engine that was fast but inflexible. This new framework gives us the flexibility we need to write highly complex rules to catch more elusive bots without the risk of interfering with legitimate traffic. The data gathered by these new detections are then labeled and used to train our Machine Learning engine, ensuring we will continue to catch these bots as their operators attempt to adapt.

We’ve heard from Bot Fight Mode customers that they need more flexibility. Website operators now expect a significant percentage of their legitimate traffic to come from automated sources, like service to service APIs. These customers are waiting to enable Bot Fight Mode until they can tell us what parts of their website it can run on safely. In 2023, we will give everyone the ability to write their own flexible Bot Fight Mode rules, so that every Cloudflare customer can join the fight against bots!

Source: One Tree Planted

We're also pleased to report the second tree planting project from our 2021 bot activity is now complete! Earlier this year, Cloudflare contributed 25,000 trees to a restoration project at Victoria Park in Nova Scotia.

For our second project, we donated 10,000 trees to a much larger restoration project on the eastern shoreline of Kumirmari island in the Sundarbans of West Bengal, India. In total, the project included more than 415,000 trees along 7.74 hectares of land in areas that have been degraded or deforested. The types of trees planted included Bain, Avicennia officianalis, Kalo Bain, and eight others.

The Sundarbans are located on the delta of the Ganges, Brahmaptura, and Meghna rivers on the Bay of Bengal, and are home to one of the world's largest mangrove forests. The forest is not only a UNESCO World Heritage site, but also home to 260 bird species as well as a number of threatened species like the Bengal tiger, the estuarine crocodile, and Indian python. According to One Tree Planted, the Sundarbans are currently under threat from rising sea levels, increasing salinity in the water and soil, cyclonic storms, and flooding.

The Intergovernmental Panel on Climate Change (IPCC) has found that mangroves are critical to mitigating greenhouse gas (GHG) emissions and protecting coastal communities from extreme weather events caused by climate change. The Sundarbans mangrove forest is one of the world's largest carbon sinks (an area that absorbs more carbon than it emits). One study suggested that coastal mangrove forests sequester carbon at a rate of two to four times that of a mature tropical or subtropical forest region.

One of the most exciting parts of this project was its focus on hiring and empowering local women. According to One Tree Planted, 75 percent of those involved in the project were women, including 85 women employed to monitor and manage the planting site over a five-month period. Participants also received training in the seed collection process with the goal of helping local residents lead mangrove planting from start to finish in the future.

Thanks to every Cloudflare customer who’s enabled Bot Fight Mode so far. You’ve helped make the Internet a better place by stopping malicious bots, and you’ve helped make the planet a better place by reforesting the Earth on bot operators’ dime. The more domains that use Bot Fight Mode, the more trees we can plant, so sign up for Cloudflare and activate Bot Fight Mode today!

Working with our partners at One Tree Planted (OTP), Cloudflare was able to support the restoration of 20 hectares of land at Victoria Park in Nova Scotia, Canada. The 130-year-old natural woodland park is located in the heart of Truro, NS, and includes over 3,000 acres of hiking and biking trails through natural gorges, rivers, and waterfalls, as well as an old-growth eastern hemlock forest.

The planting projects added red spruce, black spruce, eastern white pine, eastern larch, northern red oak, sugar maple, yellow birch, and jack pine to two areas of the park. The first area was a section of the park that recently lost a number of old conifers due to insect attacks. The second was an area previously used as a municipal dump, which has since been covered by a clay cap and topsoil.

Our tree commitment began far from the Canadian woodlands. In 2019, we launched an ambitious tool called Bot Fight Mode, which for the first time fought back against bots, targeting scrapers and other automated actors.

Our idea was simple: preoccupy bad bots with nonsense tasks, so they cannot attack real sites. Even better, make these tasks computationally expensive to engage with. This approach is effective, but it forces bad actors to consume more energy and likely emit more greenhouse gasses (GHG). So in addition to launching Bot Fight Mode, we also committed to supporting tree planting projects to account for any potential environmental impact.

As soon as Bot Fight Mode is enabled, it immediately starts challenging bots that visit your site. It is available to all Cloudflare customers for free, regardless of plan.

When Bot Fight Mode identifies a bot, it issues a computationally expensive challenge to exhaust it (also called “tarpitting"). Our aim is to disincentivize attackers, so they have to find a new hobby altogether. When we tarpit a bot, we require a significant amount of compute time that will stall its progress and result in a hefty server bill. Sorry not sorry.

We do this because bots are leeches. They draw resources, slow down sites, and abuse online platforms. They also hack into accounts and steal personal data. Of course, we allowlist a small number of bots that are well-behaved, like Slack and Google. And Bot Fight Mode only acts on traffic from cloud and hosting providers (because that is where bots usually originate from).

Over 550,000 sites use Bot Fight Mode today! We believe this makes it the most widely deployed bot management solution in the world (though this is impossible to validate). Free customers can enable the tool from the dashboard and paid customers can use a special version, known as Super Bot Fight Mode.

Now, the hard part: how can we translate bot challenges into a specific number of trees that should be planted? Fortunately, we can use a series of unit conversions, similar to those we used to calculate Cloudflare’s total GHG emissions.

We started with the following assumptions.

Table 1.

Next, we selected a high-traffic day to model the rate and duration of bot challenges on our network. On May 23, 2021, Bot Fight Mode issued 2,878,622 challenges, which lasted an average of 50 seconds each. In total, bots spent 39,981 hours engaging with our network defenses, or more than four years of challenges in a single day!

We then converted that time value into kilowatt-hours (kWh) of energy based on the rate of power consumed by our generic server listed in Table 1 above.

39,981 (hours) x .2 (kWh/hour) = 7,996 (kWh)

Once we knew the total amount of energy consumed by bad bot servers, we used an emissions factor (the amount of greenhouse gasses emitted per unit of energy consumed) to determine total emissions.

7,996 (kwh) x 338.52 (gCO2e/kwh) = 2,706,805 (gCO2e)

If you have made it this far, clearly you like to geek out like we do, so for the sake of completeness, the unit commonly used in emissions calculations is carbon dioxide equivalent (CO2e), which is a composite unit for all six GHGs listed in the Kyoto Protocol weighted by Global Warming Potential.

The last conversion we needed was from emissions to trees. Our partners at OTP found that a mature tree absorbs roughly 21 kgCO2e per year. Based on our total emissions that translates to roughly 47,000 trees per server, or 840 trees per CPU core. However, in our original post, we also noted that given the time it takes for a newly planted tree to reach maturity, we would multiply our donation by a factor of 25.

In the end, over the first two years of the program, we calculated that we would need approximately 42,000 trees to account for all the individual CPU cores engaged in Bot Fight Mode. For good measure, we rounded up to an even 50,000.

We are proud that most of these trees are already in the ground, and we look forward to providing an update when the final 25,000 are planted.

"Planting trees will benefit species diversity of the existing forest, animal habitat, greening of reclamation areas as well as community recreation areas, and visual benefits along popular hiking/biking trail networks."  - Stephanie Clement, One Tree Planted, Project Manager North America

Reforestation is an important part of protecting healthy ecosystems and promoting biodiversity. Trees and forests are also a fundamental part of helping to slow the growth of global GHG emissions.

However, we recognize there is no single solution to the climate crisis. As part of our mission to help build a better, more sustainable Internet, Cloudflare is investing in renewable energy, tools that help our customers understand and mitigate their own carbon footprints on our network, and projects that will help offset or remove historical emissions associated with powering our network by 2025.

Want to be part of our bots & trees effort? Enable Bot Fight Mode today! It’s available on our free plan and takes only a few seconds. By the time we made our first donation to OTP in 2021, Bot Fight Mode had already spent more than 3,000 years distracting bots. That is enough time to watch Stanley Kubrick’s The Shining more than 10 million times.

Help us defeat bad bots and improve our planet today!

—-For more information on Victoria Park, please visit https://www.victoriaparktruro.caFor more information on One Tree Planted, please visit https://onetreeplanted.orgFor more information on sustainability at Cloudflare, please visit www.cloudflare.com/impact

Almost half of the Internet’s traffic is powered by bots. Bots have scoured the net for years, relentlessly hacking into bank accounts, scooping up Bruno Mars tickets, and scraping websites for data. The problem is so widespread that we launched Bot Fight Mode in 2019 to fight back. Since then, over 150,000 individuals and small businesses have used the product, and we’ve received countless requests for more functionality. More analytics, more detections, and more controls.

Introducing Super Bot Fight Mode.

Beginning immediately, any Cloudflare user with a Pro or Business site can take new action against bots. We’ve added advanced features in the dashboard and some exciting updates to analytics. Free customers will retain all the benefits they've enjoyed with Bot Fight Mode, and our Enterprise Bot Management product will continue to push the needle on innovation.

Our bot solutions have a new home. The features we discuss in this blog post go beyond a single toggle, so we created a hub for bot protection. Head to the Firewall app and select the “Bots” subtab to get started.

The new hub is live for all users, including those with Enterprise Bot Management.

First up: we’re bringing our popular Bot Report to the Pro plan. Here, you can see a breakdown of your bot traffic, updated in real time to help you spot attacks.

The Bot Report includes three traffic types:

1. Likely automated traffic may have come from bad bots. We use heuristics, machine learning, and other techniques to spot these requests. In most cases, this traffic will hurt your site without providing anything useful in return.

2. Likely human traffic is legitimate and important. Ideally, the vast majority of your traffic matches this type.

3. Verified bot traffic comes from good bots on the Internet. We have verified search crawlers like Google and payment notification services like PayPal. Most users choose to allow this traffic.

All of this data is available via GraphQL as well. So if you are looking to routinely monitor bot traffic, the API will help you do so.

Pro users can also do more to stop bots — select “Configure Super Bot Fight Mode” to add protection. Highlights include:

• The option to challenge or block traffic from “definitely automated” sources. Note that this will only affect the traffic we are most confident comes from bots.

• The option to enable JavaScript Detections to identify headless browsers and other actors on the Internet.

• The option to include or exclude verified bots from protection.

If your site interacts with Slack, for example, you can exclude verified bots to help Slackbot do its job. Or if you notice an increase in ad fraud, try challenging automated traffic and watch the results.

Bot Analytics is now included with the Business plan.

We originally launched Bot Analytics to give our Enterprise users more visibility. Since the launch, however, Business users have asked us for many of the same insights. And because Cloudflare has always tried to democratize technology (as we’ve done with Firewall Events and other products), this is something we had to do.

Business users can access a new version of Bot Analytics; one that is designed to work with the mitigation tools described below. Users can view traffic by type, adjust the time frame, and filter by different attributes like IP address or user agent.

Another perk: Bot Analytics shows how we categorize traffic. Scroll to “requests by detection source” to understand which engine flagged a particular request. If you want to learn more about our detection engines, check out our blog post on the topic.

Of course, we also added new mitigation features. While Pro users can defend against “definitely automated” traffic, Business users can also target “likely automated” traffic. What’s the difference? The latter includes requests scored by our machine learning engine. These requests often come from sophisticated bots — the ones that evade simple security tools by rotating IPs or convincingly imitating humans.

Perhaps your site suffers from inventory hoarding. You list items for sale, but they are almost immediately claimed by bots. Understandably, your customers are upset (and so are you!). Go ahead and use Bot Analytics to pinpoint the attacker, and if the attack falls under “likely automated,” consider blocking this traffic.

We also realize that different sites may have different sensitivities to bot traffic. Users can respond appropriately by issuing a challenge, blocking entirely, or doing nothing at all.

These features are all included in the Cloudflare Business plan. Once you enable mitigation, check your Firewall Events tab to watch traffic get blocked or challenged.

For those with more advanced security needs, Bot Management remains the gold standard. And it’s only getting better.

Unlike Bot Fight Mode, Bot Management is built directly into the Firewall. This means that users can restrict their bot protection to a particular path (like a /login endpoint). Bot Management also includes granular bot scores, which users can pair with other attributes to produce more powerful protection. It even includes Anomaly Detection, which we use to recognize outlier patterns on your site.

We also continue to improve Bot Management. For example, just moments ago, we announced early access to API Abuse Detection. This announcement follows months of research and development. We’re using unsupervised learning to map out APIs, identify legitimate user flows, and keep out bad bots. The end result: Cloudflare will be able to protect your mobile apps (without an SDK) and secure your API endpoints (without any provided schema). Read more about the early access period.

These features (and countless others) will continue to guard the Internet’s largest sites. If you think you need Bot Management, let us know.

Cloudflare’s goal has always been to help build a better Internet. This mission extends to every part of the Internet — and to every person who uses it.

Today’s introduction of Super Bot Fight Mode was born from this mission, particularly from the idea that we are stronger as a united front against bots. Each website we protect is one that bots will waste their resources on. At Cloudflare, we are actively fighting back, and unleashing new challenges that will disincentivize bot operation with tarpitting.

We encourage you to enable Super Bot Fight Mode today. Cloudflare now offers bot protection with every plan (including Free), so there’s no excuse not to try it! Test the new features and let us know what you think.