After initially providing our customers control over the HTTP-layer DDoS protection settings earlier this year, we’re now excited to extend the control our customers have to the packet layer. Using these new controls, Cloudflare Enterprise customers using the Magic Transit and Spectrum services can now tune and tweak their L3/4 DDoS protection settings directly from the Cloudflare dashboard or via the Cloudflare API.

The new functionality provides customers control over two main DDoS rulesets:

1. Network-layer DDoS Protection ruleset — This ruleset includes rules to detect and mitigate DDoS attacks on layer 3/4 of the OSI model such as UDP floods, SYN-ACK reflection attacks, SYN Floods, and DNS floods. This ruleset is available for Spectrum and Magic Transit customers on the Enterprise plan.

2. Advanced TCP Protection ruleset — This ruleset includes rules to detect and mitigate sophisticated out-of-state TCP attacks such as spoofed ACK Floods, Randomized SYN Floods, and distributed SYN-ACK Reflection attacks. This ruleset is available for Magic Transit customers only.

To learn more, review our DDoS Managed Ruleset developer documentation. We’ve put together a few guides that we hope will be helpful for you:

1. Onboarding & getting started with Cloudflare DDoS protection

2. Handling false negatives

3. Handling false positives

4. Best practices when using VPNs, VoIP, and other third-party services

5. How to simulate a DDoS attack

A Distributed Denial of Service (DDoS) attack is a type of cyberattack that aims to disrupt the victim’s Internet services. There are many types of DDoS attacks, and they can be generated by attackers at different layers of the Internet. One example is the HTTP flood. It aims to disrupt HTTP application servers such as those that power mobile apps and websites. Another example is the UDP flood. While this type of attack can be used to disrupt HTTP servers, it can also be used in an attempt to disrupt non-HTTP applications. These include TCP-based and UDP-based applications, networking services such as VoIP services, gaming servers, cryptocurrency, and more.

To defend organizations against DDoS attacks, we built and operate software-defined systems that run autonomously. They automatically detect and mitigate DDoS attacks across our entire network. You can read more about our autonomous DDoS protection systems and how they work in our deep-dive technical blog post.

The level of protection that we offer is unmetered and unlimited — It is not bounded by the size of the attack, the number of the attacks, or the duration of the attacks. This is especially important these days because as we’ve recently seen, attacks are getting larger and more frequent. Consequently, in Q3, network-layer attacks increased by 44% compared to the previous quarter. Furthermore, just recently, our systems automatically detected and mitigated a DDoS attack that peaked just below 2 Tbps — the largest we’ve seen to date.

Mirai botnet launched an almost 2 Tbps DDoS attack

Read more about recent DDoS trends.

You can think of our autonomous DDoS protection systems as groups (rulesets) of intelligent rules. There are rulesets of HTTP DDoS Protection rules, Network-layer DDoS Protection rules and Advanced TCP Protection rules. In this blog post, we will cover the latter two rulesets. We’ve already covered the former in the blog post How to customize your HTTP DDoS protection settings.

Cloudflare L3/4 DDoS Managed Rules

In the Network-layer DDoS Protection rulesets, each rule has a unique set of conditional fingerprints, dynamic field masking, activation thresholds, and mitigation actions. These rules are managed (by Cloudflare), meaning that the specifics of each rule is curated in-house by our DDoS experts. Before deploying a new rule, it is first rigorously tested and optimized for mitigation accuracy and efficiency across our entire global network.

In the Advanced TCP Protection ruleset, we use a novel TCP state classification engine to identify the state of TCP flows. The engine powering this ruleset is flowtrackd — you can read more about it in our announcement blog post. One of the unique features of this system is that it is able to operate using only the ingress (inbound) packet flows. The system sees only the ingress traffic and is able to drop, challenge, or allow packets based on their legitimacy. For example, a flood of ACK packets that don’t correspond to open TCP connections will be dropped.

Initially, traffic is routed through the Internet via BGP Anycast to the nearest Cloudflare edge data center. Once the traffic reaches our data center, our DDoS systems sample it asynchronously allowing for out-of-path analysis of traffic without introducing latency penalties. The Advanced TCP Protection ruleset needs to view the entire packet flow and so it sits inline for Magic Transit customers only. It, too, does not introduce any latency penalties.

The analysis for the Advanced TCP Protection ruleset is straightforward and efficient. The system qualifies TCP flows and tracks their state. In this way, packets that don’t correspond to a legitimate connection and its state are dropped or challenged. The mitigation is activated only above certain thresholds that customers can define.

The analysis for the Network-layer DDoS Protection ruleset is done using data streaming algorithms. Packet samples are compared to the conditional fingerprints and multiple real-time signatures are created based on the dynamic masking. Each time another packet matches one of the signatures, a counter is increased. When the activation threshold is reached for a given signature, a mitigation rule is compiled and pushed inline. The mitigation rule includes the real-time signature and the mitigation action, e.g., drop.

As a simple example, one fingerprint could include the following fields: source IP, source port, destination IP, and the TCP sequence number. A packet flood attack with a fixed sequence number would match the fingerprint and the counter would increase for every packet match until the activation threshold is exceeded. Then a mitigation action would be applied.

However, in the case of a spoofed attack where the source IP addresses and ports are randomized, we would end up with multiple signatures for each combination of source IP and port. Assuming a sufficiently randomized/distributed attack, the activation thresholds would not be met and mitigation would not occur. For this reason, we use dynamic masking, i.e. ignoring fields that may not be a strong indicator of the signature. By masking (ignoring) the source IP and port, we would be able to match all the attack packets based on the unique TCP sequence number regardless of how randomized/distributed the attack is.

For now, we’ve only exposed a handful of the Network-layer DDoS protection rules that we’ve identified as the ones most prone to customizations. We will be exposing more and more rules on a regular basis. This shouldn’t affect any of your traffic.

Overriding the sensitivity level and mitigation action

For the Network-layer DDoS Protection ruleset, for each of the available rules, you can override the sensitivity level (activation threshold), customize the mitigation action, and apply expression filters to exclude/include traffic from the DDoS protection system based on various packet fields. You can create multiple overrides to customize the protection for your network and your various applications.

Configuring expression fields for the DDoS Managed Rules to match on

In the past, you’d have to go through our support channels to customize the rules. In some cases, this may have taken longer to resolve than desired. With today’s announcement, you can tailor and fine-tune the settings of our autonomous edge system by yourself to quickly improve the accuracy of the protection for your specific network needs.

For the Advanced TCP Protection ruleset, for now, we’ve only exposed the ability to enable or disable it as a whole in the dashboard. To enable or disable the ruleset per IP prefix, you must use the API. At this time, when initially onboarding to Cloudflare, the Cloudflare team must first create a policy for you. After onboarding, if you need to change the sensitivity thresholds, use Monitor mode, or add filter expressions you must contact Cloudflare Support. In upcoming releases, this too will be available via the dashboard and API without requiring help from our Support team.

If you previously contacted Cloudflare Support to apply customizations, your customizations have been preserved, and you can visit the dashboard to view the settings of the Network-layer DDoS Protection ruleset and change them if you need. If you require any changes to your Advanced TCP Protection customizations, please reach out to Cloudflare Support.

If so far you didn't have the need to customize this protection, there is no action required on your end. However, if you would like to view and customize your DDoS protection settings, follow this dashboard guide or review the API documentation to programmatically configure the DDoS protection settings.

At Cloudflare, everything we do is guided by our mission to help build a better Internet. The DDoS team’s vision is derived from this mission: our goal is to make the impact of DDoS attacks a thing of the past. Our first step was to build the autonomous systems that detect and mitigate attacks independently. Done. The second step was to expose the control plane over these systems to our customers (announced today). Done. The next step will be to fully automate the configuration with an auto-pilot feature — training the systems to learn your specific traffic patterns to automatically optimize your DDoS protection settings. You can expect many more improvements, automations, and new capabilities to keep your Internet properties safe, available, and performant.

Not using Cloudflare yet? Start now.

Earlier this summer, Cloudflare’s autonomous edge DDoS protection systems automatically detected and mitigated a 17.2 million request-per-second (rps) DDoS attack, an attack almost three times larger than any previous one that we're aware of. For perspective on how large this attack was: Cloudflare serves over 25 million HTTP requests per second on average. This refers to the average rate of legitimate traffic in 2021 Q2. So peaking at 17.2 million rps, this attack reached 68% of our Q2 average rps rate of legitimate HTTP traffic.

Comparison graph of Cloudflare’s average request per second rate versus the DDoS attack

This attack, along with the additional attacks provided in the next sections, were automatically detected and mitigated by our autonomous edge DDoS protection systems. The system is powered by our very own denial of service daemon (dosd). Dosd is a home-grown software-defined daemon. A unique dosd instance runs in every server in each one of our data centers around the world. Each dosd instance independently analyzes traffic samples out-of-path. Analyzing traffic out-of-path allows us to scan asynchronously for DDoS attacks without causing latency and impacting performance. DDoS findings are also shared between the various dosd instances within a data center, as a form of proactive threat intelligence sharing.

Once an attack is detected, our systems generate a mitigation rule with a real-time signature that matches the attack patterns. The rule is propagated to the most optimal location in the tech stack. As an example, a volumetric HTTP DDoS attack may be blocked at L4 inside the Linux iptables firewall instead of at L7 inside the L7 reverse proxy which runs in the user space. Mitigating lower in the stack, e.g. dropping the packets at L4 instead of responding with a 403 error page in L7, is more cost-efficient. It reduces our edge CPU consumption and intra-data center bandwidth utilization — thus helping us mitigate large attacks at scale without impacting performance.

This autonomous approach, along with our network’s global scale and reliability, allow us to mitigate attacks that reach 68% of our average per-second-rate, and higher, without requiring any manual mitigation by Cloudflare personnel, nor causing any performance degradation.

This attack was launched by a powerful botnet, targeting a Cloudflare customer in the financial industry. Within seconds, the botnet bombarded the Cloudflare edge with over 330 million attack requests.

Graph of 17.2M rps attack

The attack traffic originated from more than 20,000 bots in 125 countries around the world. Based on the bots’ source IP addresses, almost 15% of the attack originated from Indonesia and another 17% from India and Brazil combined. Indicating that there may be many malware infected devices in those countries.

Distribution of the attack sources by top countries

This 17.2 million rps attack is the largest HTTP DDoS attack that Cloudflare has ever seen to date and almost three times the size of any other reported HTTP DDoS attack. This specific botnet, however, has been seen at least twice over the past few weeks. Just last week it also targeted a different Cloudflare customer, a hosting provider, with an HTTP DDoS attack that peaked just below 8 million rps.

Graph of 8M rps attack

Two weeks before, a Mirai-variant botnet launched over a dozen UDP and TCP based DDoS attacks that peaked multiple times above 1 Tbps, with a max peak of approximately 1.2 Tbps. And while the first HTTP attacks targeted Cloudflare customers on the WAF/CDN service, the 1+ Tbps network-layer attacks targeted Cloudflare customers on the Magic Transit and Spectrum services. One of these targets was a major APAC-based Internet services, telecommunications and hosting provider. The other was a gaming company. In all cases, the attacks were automatically detected and mitigated without human intervention.

Graph of Mirai botnet attack peaking at 1.2 Tbps

The Mirai botnet started with roughly 30K bots and slowly shrinked to approximately 28K. However, despite losing bots from its fleet, the botnet was still able to generate impressive volumes of attack traffic for short periods. In some cases, each burst lasted only a few seconds.

These attacks join the increase in Mirai-based DDoS attacks that we’ve observed on our network over the past weeks. In July alone, L3/4 Mirai attacks increased by 88% and L7 attacks by 9%. Additionally, based on the current August per-day average of the Mirai attacks, we can expect L7 Mirai DDoS attacks and other similar botnet attacks to increase by 185% and L3/4 attacks by 71% by the end of the month.

Graph of change in Mirai based DDoS attacks by month

Mirai, which means ‘future’ in Japanese, is a codename for malware that was first discovered in 2016 by MalwareMustDie, a non-profit security research workgroup. The malware spreads by infecting Linux-operated devices such as security cameras and routers. It then self-propagates by searching for open Telnet ports 23 and 2323. Once found, it then attempts to gain access to vulnerable devices by brute forcing known credentials such as factory default usernames and passwords. Later variants of Mirai also took advantage of zero-day exploits in routers and other devices. Once infected, the devices will monitor a Command & Control (C2) server for instructions on which target to attack.

Diagram of Botnet operator controlling the botnet to attack websites

While the majority of attacks are small and short, we continue to see these types of volumetric attacks emerging more often. It’s important to note that these volumetric short burst attacks can be especially dangerous for legacy DDoS protection systems or organizations without active, always-on cloud-based protection.

Furthermore, while the short duration may say something about the botnet’s capability to deliver sustained levels of traffic over time, it can be challenging or impossible for humans to react to it in time. In such cases, the attack is over before a security engineer even has time to analyze the traffic or activate their stand-by DDoS protection system. These types of attacks highlight the need for automated, always-on protection.

1. Onboard to Cloudflare to protect your Internet properties.

2. DDoS is enabled out of the box, and you can also customize the protection settings.

3. Follow our preventive best practices, to ensure that both your Cloudflare settings and your origin server settings are optimized. As an example, make sure that you allow only traffic from Cloudflare’s IP range. Ideally, ask your upstream Internet Service Provider (ISP) to apply an access control list (ACL), otherwise, attackers may target your servers’ IP addresses directly and bypass your protection.

1. Change the default username and password of any device that is connected to the Internet such as smart cameras and routers. This will reduce the risk that malware such as Mirai can gain access to your router and IoT devices.

2. Protect your home against malware with Cloudflare for Families. Cloudflare for Families is a free service that automatically blocks traffic from your home to malicious websites and malware communication.

Today, I’m excited to talk about our autonomous DDoS (Distributed Denial of Service) protection system. This system has been deployed globally to all of our 200+ data centers and actively protects all our customers against DDoS attacks across layers 3 to 7 (in the OSI model) without requiring any human intervention. As part of our unmetered DDoS protection commitment, we won’t charge a customer more just because they got hit by a DDoS.

To protect our customers quickly and with precision against DDoS attacks, we built an autonomous edge detection and mitigation system that can make decisions on its own without seeking a centralized consensus. It is completely software-defined and runs on our edge on commodity servers. It’s powered by our denial of service daemon (dosd) which originally went live in mid-2019 for protection against L3/4 DDoS attacks. Since then, we’ve been investing in enhancing and improving its capabilities to stay ahead of attackers and to disrupt the economics of attacks. The latest set of improvements have expanded our edge mitigation component to protect against L7 attacks in addition to L3/4.

This system runs on every single server in all our edge data centers. It constantly analyzes packets and HTTP requests, scanning for DDoS attacks. Upon detection, it immediately pushes a mitigation rule with a real-time generated signature to the most optimal location in the Linux stack where the most cost-efficient mitigation can be applied.

A conceptual diagram of Cloudflare DDoS mitigation systems

Our new edge detection capabilities complement our existing global threat detection mechanism, Gatebot, which resides in our network’s core. Detecting attacks at the network core with Gatebot is great for larger, distributed volumetric attacks that require coordination across the entire Cloudflare edge, but smaller, localized attacks require a different approach. Detecting network-layer and HTTP attacks at the edge means we can sample at a higher rate, detect both small and large attacks, and immediately generate a mitigation rule. Over the past few months, 98.6% of all L3/4 DDoS attacks were detected by dosd. Similarly, since deploying the expanded version of dosd, it has been mitigating 81% of all L7 attacks.

In previous blogs, we’ve already covered Gatebot and flowtrackd. So in this blog, we’ll be focusing on the expanded dosd capabilities.

Ten years ago, Linux networking was slow. Today, we’re dropping packets at wire speed thanks to Linux — specifically, with iptables and the eXpress Data Path (XDP).

A packet destined for a Cloudflare-protected customer makes its way to the closest Cloudflare data center through BGP Anycast. Once it arrives, it is passed from the router to a server using equal-cost multi-path routing groups (ECMP) algorithm via network switches. When it arrives at a server, the packet is sent into a group of eXpress Data Path (XDP) programs. The first group of XDP programs, L4Drop, applies mitigation rules from previously detected attacks and transmits packet samples to dosd for further analysis.

If a packet is not dropped as malicious, it’s then passed to Unimog, our proprietary L4 load balancer. Using server health and performance metrics, Unimog decides whether it should keep the packet in the same server or pass it on to another server in the data center better able to handle it. After Unimog, it is passed through the iptables firewall and then, if targeting an L7 application, e.g., a service protected by the Cloudflare WAF, to our HTTP reverse proxy. The reverse-proxy runs in userspace and HTTP requests go through our Web Application Firewall, application Firewall rules, and additional customer configurations. If the packet is instead destined for a TCP/UDP application (Spectrum) or an IP destination that is routed rather than proxied (Magic Transit), it would pass through those systems rather than our HTTP proxy.

Life of a packet

In addition to L4Drop, our HTTP proxy also transmits samples and metadata of HTTP requests to dosd. This edge sampling happens at a rate that’s 10 times greater than core sampling, as signals can now be analyzed (and acted upon) locally rather than shipped to a core data center. Similarly, packets are sampled by dosd at a rate that is 81 times faster than gatebot.

Together, dosd, gatebot, and flowtrackd analyze samples they receive and apply mitigation rules when DDoS attacks are detected. They push mitigation rules into the web proxy to mitigate HTTP attacks. Attack requests are handled with a block, rate limit, or challenge action, depending on the system’s decision. However, if the attack is highly volumetric, the mitigation rule is pushed down the stack into the iptables firewall and L7 attacks are dropped at L4 using IP Jails, for a more cost-efficient mitigation. Similarly, L3/4 attacks are mitigated in the iptables firewall using extended Berkeley Packet Filter (eBPF) programs inside L4Drop. Leveraging these components allows us to automatically mitigate DDoS attacks at scale.

Our expanded autonomous system, described above, along with our existing threat mitigation components, was developed to protect our customers against DDoS attacks that have become very easy and cheap to launch. These attacks are used by malicious actors that aim to take down a website, mobile app, game, or any Internet-connected property. These expanded protections were a necessary step as during the past year the number of attacks has increased, as we’ve documented in our DDoS trends reports. Additionally, the attacks are getting bigger and more sophisticated, such as the attack that imitated acoustic beats. Just as important are small attacks that could take down a small web property; we want to block the large and small.

In many cases, attackers can launch DDoS attacks for free using publicly available tools, or for a small fee by hiring a DDoS-as-a-service botnet such as Moobot in the dark web. According to the Dark Web Price Index for 2020, the price of a DDoS attack starts at $10 for a one-hour attack at a rate of 10-50k requests per second. Attacks are far cheaper to launch than the damage they cause. By causing an outage or even just by degrading the service, attackers can take a substantial toll on their victim. As an example, taking down an ecommerce website means that users cannot log in and make purchases. Even increased latency can cause users to abandon their shopping carts and pop over to the competition. A minute of downtime can easily translate to the loss of tens of thousands of dollars.

The frequency, sophistication, and size of DDoS attacks require a new approach — one that is fast, accurate, and precise. And this is why we developed the expanded protections described in this post.

Cloudflare’s mission is to help build a better Internet — one that is secure, faster, and more reliable for all. The DDoS team’s vision is derived from this mission: our goal is to make the impact of DDoS attacks a thing of the past. In the ’90s and 2000s, spam emails became a serious problem. Today, email services filter them out for us, and our objective is to do the same for DDoS attacks.

For more information about Cloudflare’s DDoS protection, reach out to us or have a go with a hands-on evaluation of Cloudflare’s free plan.

By the way, if you are interested in working in the DDoS Protection engineering team, we're currently hiring in our London and Austin offices. Submit your application here:

 Austin: https://boards.greenhouse.io/cloudflare/jobs/2291040?gh_jid=2291040 London: https://boards.greenhouse.io/cloudflare/jobs/2291038?gh_jid=2291038