Rate Limiting: Delivering more rules, and greater control

With more and more platforms taking the necessary precautions against DDoS attacks like integrating DDoS mitigation services and increasing bandwidth at weak points, Layer 3 and 4 attacks are just not as effective anymore. For Cloudflare, we have fully automated Layer 3/4 based protections with our internal platform, Gatebot. In the last 6 months we have seen a large upward trend of Layer 7 based DDoS attacks. The key difference to these attacks is they are no longer focused on using huge payloads (volumetric attacks), but based on Requests per Second to exhaust server resources (CPU, Disk and Memory). On a regular basis we see attacks that are over 1 million requests per second. The graph below shows the number of Layer 7 attacks Cloudflare has monitored, which is trending up. On average seeing around 160 attacks a day, with some days spiking up to over 1000 attacks.

A year ago, Cloudflare released Rate Limiting and it is proving to be a hugely effective tool for customers to protect their web applications and APIs from all sorts of attacks, from “low and slow” DDoS attacks, through to bot-based attacks, such as credential stuffing and content scraping. We’re pleased about the success our customers are seeing with Rate Limiting and are excited to announce additional capabilities to give our customers further control.

There are times when you clearly know that traffic is malicious. In cases like this, our existing Block action is proving effective for our customers. But there are times when it is not the best option, and causes a negative user experience. Rather than risk a false negative, customers often want to challenge a client to ensure they are who they represent themselves to be, which is in most situations, human not a bot.

Firstly, to help customers more accurately identify the traffic, we are adding Cloudflare JavaScript Challenge, and Google reCaptcha (Challenge) mitigation actions to the UI and API for Pro and Business plans. The existing Block and Simulate actions still exist. As a reminder, to test any rule, deploying in Simulate means that you will not be charged for any requests. This is a great way to test your new rules to make sure they have been configured correctly.

Secondly, we’re making Rate Limiting more dynamically scalable. A new feature has been added which allows Rate Limiting to count on Origin Response Headers for Business and Enterprise customers. The way this feature works is by matching attributes which are returned by the Origin to Cloudflare.

One of the things that really drives our innovation is solving the real problems we hear from customers every day. With that, we wanted to provide some real world examples of these new capabilities in action.

Each of the use cases have Basic and Advanced implementation options. After some testing, we found that tiering rate limits is an extremely effective solution against repeat offenders.

Credential Stuffing Protection for Login Pages and APIs. The best way to build applications is to utilise the standardized Status Codes. For example, if I fail to authenticate against an endpoint or a website, I should receive a “401” or “403”. Generally speaking a user to a website will often get their password wrong three times before selecting the “I forgot my password” option. Most Credential Stuff bots will try thousands of times cycling through many usernames and password combinations to see what works.

Here are some example rate limits which you can configure to protect your application from credential stuffing.

Basic:Cloudflare offers a “Protect My Login” feature out the box. Enter the URL for your login page and Cloudflare will create a rule such that clients that attempt to log in more than 5 times in 5 minutes will be blocked for 15 minutes.

With the new Challenge capabilities of Rate Limiting, you can customize the response parameters for log in to more closely match the behavior pattern for bots you see on your site through a custom built rule.

Logging in four times in one minute is hard - I type fast, but couldn’t even do this. If I’m seeing this pattern in my logs, it is likely a bot. I can now create a Rate Limiting rule based on the following criteria:

.table-with-last-column-right-aligned tr td:last-child { text-align: right; }

RuleID

URL

Count

Timeframe

Matching Criteria

Action

1

/login

4

1 minute

Method: POSTStatus Code: 401,403

Challenge

With this new rule, if someone tries to log in four times within a minute, they will be thrown a challenge. My regular human users will likely never hit it, but if they do - the challenge insures they can still access the site.

Advanced:And sometimes bots are just super persistent in their attacks. We can tier rules together to tackle repeat offenders. For example, instead of creating just a single rule, we can create a series of rules which can be tiered to protect against persistent threats:

.table-with-last-column-right-aligned tr td:last-child { text-align: right; }

RuleID

URL

Count

Timeframe

Matching Criteria

Action

1

/login

4

1 minute

Method: POSTStatus Code: 401,403

JavaScript Challenge

2

/login

10

5 minutes

Method: POSTStatus Code: 401,403

Challenge

3

/login

20

1 hour

Method: POSTStatus Code: 401,403

Block for 1 day

With this type of tiering, any genuine users that are just having a hard time remembering their login details whilst also being extremely fast typers will not be fully blocked. Instead, they will first be given out automated JavaScript challenge followed by a traditional CAPTCHA if they hit the next limit. This is a much more user-friendly approach while still securing your login endpoints.

Time-based Firewall

Our IP Firewall is a powerful feature to block problematic IP addresses from accessing your app. Particularly this is related to repeated abuse, or based on IP Reputation or Threat Intelligence feeds that are integrated at the origin level.

While the the IP firewall is powerful, maintaining and managing a list of IP addresses which are currently being blocked can be cumbersome. It becomes more complicated if you want to allow blocked IP addresses to “age out” if bad behavior stops after a period of time. This often requires authoring and managing a script and multiple API calls to Cloudflare.

The new Rate Limiting Origin Headers feature makes this all so much easier. You can now configure your origin to respond with a Header to trigger a Rate-Limit. To make this happen, we need to generate a Header at the Origin, which is then added to the response to Cloudflare. As we are matching on a static header, we can set a severity level based on the content of the Header. For example, if it was a repeat offender, you could respond with High as the Header value, which could Block for a longer period.

Create a Rate Limiting rule based on the following criteria:

RuleID

URL

Count

Timeframe

Matching Criteria

Action

1

*

1

1 second

Method: _ALL_Header: X-CF-Block = low

Block for 5 minutes

2

*

1

1 second

Method: _ALL_Header: X-CF-Block = medium

Block for 15 minutes

3

*

1

1 second

Method: _ALL_Header: X-CF-Block = high

Block for 60 minutes

Once that Rate-Limit has been created, Cloudflare’s Rate-Limiting will then kick-in immediately when that Header is received.

Enumeration Attacks

Enumeration attacks are proving to be increasingly popular and pesky to mitigate. With enumeration attacks, attackers identify an expensive operation in your app and hammer at it to tie up resources and slow or crash your app. For example, an app that offers the ability to look up a user profile requires a database lookup to validate whether the user exists. In a enumeration attack, attackers will send a random set of characters to that endpoint in quick succession, causing the database to ground to a halt.

Rate Limiting to the rescue!

One of our customers was hit with a huge enumeration attack on their platform earlier this year, where the aggressors were trying to do exactly what we described above, in an attempt to overload their database platform. Their Rate Limiting configuration blocked over 100,000,000 bad requests during the 6 hour attack.

When a query is sent to the app, and the user is not found, the app serves a 404 (page not found) . A very basic approach is to set a rate limit for 404s. If a user crosses a threshold of 404’s in a period of time, set the app to challenge the user to prove themselves to be a real person.

RuleID

URL

Count

Timeframe

Matching Criteria

Action

1

*

10

1 minute

Method: GETStatus Code: 404

Challenge

To catch repeat offenders, you can tier the tier Rate Limits:

RuleID

URL

Count

Timeframe

Matching Criteria

Action

1

/public/profile*

10

1 minute

Method: GETStatus Code: 404

JavaScript Challenge

2

/public/profile*

25

1 minute

Method: GETStatus Code: 200

Challenge

3

/public/profile*

50

10 minutes

Method: GETStatus Code: 200, 404

Block for 4 hours

With this type of tiered defense in place, it means that you can “caution” an offender with a JavaScript challenge or Challenge (Google Captcha), and then “block” them if they continue.

Content Scraping

Increasingly, content owners are wrestling with content scraping - malicious bots copying copyrighted images or assets and redistributing or reusing them. For example, we work with an eCommerce store that uses copyrighted images and their images are appearing elsewhere on the web without their consent. Rate Limiting can help!

In their app, each page displays 4 copyrighted images, 1 which is actual size, and 3 which are thumbnails. By looking at logs and user patterns, they determined that most users, at a stretch, would never view more than 10-15 products in a minute, which would equate to 40-60 loads from the images store.

They chose to tier their Rate Limiting rules to prevent end users from getting unnecessarily blocked when they were browsing heavily. To block malicious attempts at content scraping can be quite simple, however it does require some forward planning. Placing the rate limit on the right URL is key to insure you are placing the rule on exactly what you are trying to protect and not the broader content. Here’s an example set of rate limits this customer set to protect their images:

RuleID

URL

Count

Timeframe

Matching Criteria

Action

1

/img/thumbs/*

10

1 minute

Method: GETStatus Code: 404

Challenge

2

/img/thumbs/*

25

1 minute

Method: GETStatus Code: 200

Challenge

3

/img/*

75

1 minute

Method: GETStatus Code: 200

Block for 4 hours

4

/img/*

5

1 minute

Method: GETStatus Code: 403, 404

Challenge

As we can see here, rules 1 and 2 are counting based on the number of requests to each endpoint. Rule 3 is counting based on all hits to the image store, and if it gets above 75 requests, the user will be blocked for 4 hours. Finally, to avoid any enumeration or bots guessing image names and numbers, we are counting on 404 and 403s and challenging if we see an unusual spikes.

We want to ensure you have the rules you need to secure your app. To do that, we are increasing the number of available rules for Pro and Business, for no additional charge.

• Pro plans increase from 3 to 10 rules

• Business plans increase from 3 to 15 rules

As always, Cloudflare only charges for good traffic - requests that are allowed through Rate Limiting, not blocked. For more information click here.

The Rate-Limiting feature can be enabled within the Firewall tab on the Dashboard, or by visiting: cloudflare.com/a/firewall