Today, I’m excited to talk about our autonomous DDoS (Distributed Denial of Service) protection system. This system has been deployed globally to all of our 200+ data centers and actively protects all our customers against DDoS attacks across layers 3 to 7 (in the OSI model) without requiring any human intervention. As part of our unmetered DDoS protection commitment, we won’t charge a customer more just because they got hit by a DDoS.
Autonomous protection at the edge
To protect our customers quickly and with precision against DDoS attacks, we built an autonomous edge detection and mitigation system that can make decisions on its own without seeking a centralized consensus. It is completely software-defined and runs on our edge on commodity servers. It’s powered by our denial of service daemon (dosd) which originally went live in mid-2019 for protection against L3/4 DDoS attacks. Since then, we’ve been investing in enhancing and improving its capabilities to stay ahead of attackers and to disrupt the economics of attacks. The latest set of improvements have expanded our edge mitigation component to protect against L7 attacks in addition to L3/4.
This system runs on every single server in all our edge data centers. It constantly analyzes packets and HTTP requests, scanning for DDoS attacks. Upon detection, it immediately pushes a mitigation rule with a real-time generated signature to the most optimal location in the Linux stack where the most cost-efficient mitigation can be applied.
A conceptual diagram of Cloudflare DDoS mitigation systems
Our new edge detection capabilities complement our existing global threat detection mechanism, Gatebot, which resides in our network’s core. Detecting attacks at the network core with Gatebot is great for larger, distributed volumetric attacks that require coordination across the entire Cloudflare edge, but smaller, localized attacks require a different approach. Detecting network-layer and HTTP attacks at the edge means we can sample at a higher rate, detect both small and large attacks, and immediately generate a mitigation rule. Over the past few months, 98.6% of all L3/4 DDoS attacks were detected by dosd. Similarly, since deploying the expanded version of dosd, it has been mitigating 81% of all L7 attacks.
In previous blogs, we’ve already covered Gatebot and flowtrackd. So in this blog, we’ll be focusing on the expanded dosd capabilities.
Harnessing Linux networking to drop packets and requests at wire speed
Ten years ago, Linux networking was slow. Today, we’re dropping packets at wire speed thanks to Linux — specifically, with iptables and the eXpress Data Path (XDP).
The life of a packet
A packet destined for a Cloudflare-protected customer makes its way to the closest Cloudflare data center through BGP Anycast. Once it arrives, it is passed from the router to a server using equal-cost multi-path routing groups (ECMP) algorithm via network switches. When it arrives at a server, the packet is sent into a group of eXpress Data Path (XDP) programs. The first group of XDP programs, L4Drop, applies mitigation rules from previously detected attacks and transmits packet samples to dosd for further analysis.
If a packet is not dropped as malicious, it’s then passed to Unimog, our proprietary L4 load balancer. Using server health and performance metrics, Unimog decides whether it should keep the packet in the same server or pass it on to another server in the data center better able to handle it. After Unimog, it is passed through the iptables firewall and then, if targeting an L7 application, e.g., a service protected by the Cloudflare WAF, to our HTTP reverse proxy. The reverse-proxy runs in userspace and HTTP requests go through our Web Application Firewall, application Firewall rules, and additional customer configurations. If the packet is instead destined for a TCP/UDP application (Spectrum) or an IP destination that is routed rather than proxied (Magic Transit), it would pass through those systems rather than our HTTP proxy.
Life of a packet
In addition to L4Drop, our HTTP proxy also transmits samples and metadata of HTTP requests to dosd. This edge sampling happens at a rate that’s 10 times greater than core sampling, as signals can now be analyzed (and acted upon) locally rather than shipped to a core data center. Similarly, packets are sampled by dosd at a rate that is 81 times faster than gatebot.
Together, dosd, gatebot, and flowtrackd analyze samples they receive and apply mitigation rules when DDoS attacks are detected. They push mitigation rules into the web proxy to mitigate HTTP attacks. Attack requests are handled with a block, rate limit, or challenge action, depending on the system’s decision. However, if the attack is highly volumetric, the mitigation rule is pushed down the stack into the iptables firewall and L7 attacks are dropped at L4 using IP Jails, for a more cost-efficient mitigation. Similarly, L3/4 attacks are mitigated in the iptables firewall using extended Berkeley Packet Filter (eBPF) programs inside L4Drop. Leveraging these components allows us to automatically mitigate DDoS attacks at scale.
Disrupting the attack economics
Our expanded autonomous system, described above, along with our existing threat mitigation components, was developed to protect our customers against DDoS attacks that have become very easy and cheap to launch. These attacks are used by malicious actors that aim to take down a website, mobile app, game, or any Internet-connected property. These expanded protections were a necessary step as during the past year the number of attacks has increased, as we’ve documented in our DDoS trends reports. Additionally, the attacks are getting bigger and more sophisticated, such as the attack that imitated acoustic beats. Just as important are small attacks that could take down a small web property; we want to block the large and small.
In many cases, attackers can launch DDoS attacks for free using publicly available tools, or for a small fee by hiring a DDoS-as-a-service botnet such as Moobot in the dark web. According to the Dark Web Price Index for 2020, the price of a DDoS attack starts at $10 for a one-hour attack at a rate of 10-50k requests per second. Attacks are far cheaper to launch than the damage they cause. By causing an outage or even just by degrading the service, attackers can take a substantial toll on their victim. As an example, taking down an ecommerce website means that users cannot log in and make purchases. Even increased latency can cause users to abandon their shopping carts and pop over to the competition. A minute of downtime can easily translate to the loss of tens of thousands of dollars.
The frequency, sophistication, and size of DDoS attacks require a new approach — one that is fast, accurate, and precise. And this is why we developed the expanded protections described in this post.
Helping build a better Internet
Cloudflare’s mission is to help build a better Internet — one that is secure, faster, and more reliable for all. The DDoS team’s vision is derived from this mission: our goal is to make the impact of DDoS attacks a thing of the past. In the ’90s and 2000s, spam emails became a serious problem. Today, email services filter them out for us, and our objective is to do the same for DDoS attacks.
For more information about Cloudflare’s DDoS protection, reach out to us or have a go with a hands-on evaluation of Cloudflare’s free plan.
By the way, if you are interested in working in the DDoS Protection engineering team, we're currently hiring in our London and Austin offices. Submit your application here:
Austin: https://boards.greenhouse.io/cloudflare/jobs/2291040?gh_jid=2291040 London: https://boards.greenhouse.io/cloudflare/jobs/2291038?gh_jid=2291038