Today, I’m excited to talk about our autonomous DDoS (Distributed Denial of Service) protection system. This system has been deployed globally to all of our 200+ data centers and actively protects all our customers against DDoS attacks across layers 3 to 7 (in the OSI model) without requiring any human intervention. As part of our unmetered DDoS protection commitment, we won’t charge a customer more just because they got hit by a DDoS.

To protect our customers quickly and with precision against DDoS attacks, we built an autonomous edge detection and mitigation system that can make decisions on its own without seeking a centralized consensus. It is completely software-defined and runs on our edge on commodity servers. It’s powered by our denial of service daemon (dosd) which originally went live in mid-2019 for protection against L3/4 DDoS attacks. Since then, we’ve been investing in enhancing and improving its capabilities to stay ahead of attackers and to disrupt the economics of attacks. The latest set of improvements have expanded our edge mitigation component to protect against L7 attacks in addition to L3/4.

This system runs on every single server in all our edge data centers. It constantly analyzes packets and HTTP requests, scanning for DDoS attacks. Upon detection, it immediately pushes a mitigation rule with a real-time generated signature to the most optimal location in the Linux stack where the most cost-efficient mitigation can be applied.

A conceptual diagram of Cloudflare DDoS mitigation systems

Our new edge detection capabilities complement our existing global threat detection mechanism, Gatebot, which resides in our network’s core. Detecting attacks at the network core with Gatebot is great for larger, distributed volumetric attacks that require coordination across the entire Cloudflare edge, but smaller, localized attacks require a different approach. Detecting network-layer and HTTP attacks at the edge means we can sample at a higher rate, detect both small and large attacks, and immediately generate a mitigation rule. Over the past few months, 98.6% of all L3/4 DDoS attacks were detected by dosd. Similarly, since deploying the expanded version of dosd, it has been mitigating 81% of all L7 attacks.

In previous blogs, we’ve already covered Gatebot and flowtrackd. So in this blog, we’ll be focusing on the expanded dosd capabilities.

Ten years ago, Linux networking was slow. Today, we’re dropping packets at wire speed thanks to Linux — specifically, with iptables and the eXpress Data Path (XDP).

A packet destined for a Cloudflare-protected customer makes its way to the closest Cloudflare data center through BGP Anycast. Once it arrives, it is passed from the router to a server using equal-cost multi-path routing groups (ECMP) algorithm via network switches. When it arrives at a server, the packet is sent into a group of eXpress Data Path (XDP) programs. The first group of XDP programs, L4Drop, applies mitigation rules from previously detected attacks and transmits packet samples to dosd for further analysis.

If a packet is not dropped as malicious, it’s then passed to Unimog, our proprietary L4 load balancer. Using server health and performance metrics, Unimog decides whether it should keep the packet in the same server or pass it on to another server in the data center better able to handle it. After Unimog, it is passed through the iptables firewall and then, if targeting an L7 application, e.g., a service protected by the Cloudflare WAF, to our HTTP reverse proxy. The reverse-proxy runs in userspace and HTTP requests go through our Web Application Firewall, application Firewall rules, and additional customer configurations. If the packet is instead destined for a TCP/UDP application (Spectrum) or an IP destination that is routed rather than proxied (Magic Transit), it would pass through those systems rather than our HTTP proxy.

Life of a packet

In addition to L4Drop, our HTTP proxy also transmits samples and metadata of HTTP requests to dosd. This edge sampling happens at a rate that’s 10 times greater than core sampling, as signals can now be analyzed (and acted upon) locally rather than shipped to a core data center. Similarly, packets are sampled by dosd at a rate that is 81 times faster than gatebot.

Together, dosd, gatebot, and flowtrackd analyze samples they receive and apply mitigation rules when DDoS attacks are detected. They push mitigation rules into the web proxy to mitigate HTTP attacks. Attack requests are handled with a block, rate limit, or challenge action, depending on the system’s decision. However, if the attack is highly volumetric, the mitigation rule is pushed down the stack into the iptables firewall and L7 attacks are dropped at L4 using IP Jails, for a more cost-efficient mitigation. Similarly, L3/4 attacks are mitigated in the iptables firewall using extended Berkeley Packet Filter (eBPF) programs inside L4Drop. Leveraging these components allows us to automatically mitigate DDoS attacks at scale.

Our expanded autonomous system, described above, along with our existing threat mitigation components, was developed to protect our customers against DDoS attacks that have become very easy and cheap to launch. These attacks are used by malicious actors that aim to take down a website, mobile app, game, or any Internet-connected property. These expanded protections were a necessary step as during the past year the number of attacks has increased, as we’ve documented in our DDoS trends reports. Additionally, the attacks are getting bigger and more sophisticated, such as the attack that imitated acoustic beats. Just as important are small attacks that could take down a small web property; we want to block the large and small.

In many cases, attackers can launch DDoS attacks for free using publicly available tools, or for a small fee by hiring a DDoS-as-a-service botnet such as Moobot in the dark web. According to the Dark Web Price Index for 2020, the price of a DDoS attack starts at $10 for a one-hour attack at a rate of 10-50k requests per second. Attacks are far cheaper to launch than the damage they cause. By causing an outage or even just by degrading the service, attackers can take a substantial toll on their victim. As an example, taking down an ecommerce website means that users cannot log in and make purchases. Even increased latency can cause users to abandon their shopping carts and pop over to the competition. A minute of downtime can easily translate to the loss of tens of thousands of dollars.

The frequency, sophistication, and size of DDoS attacks require a new approach — one that is fast, accurate, and precise. And this is why we developed the expanded protections described in this post.

Cloudflare’s mission is to help build a better Internet — one that is secure, faster, and more reliable for all. The DDoS team’s vision is derived from this mission: our goal is to make the impact of DDoS attacks a thing of the past. In the ’90s and 2000s, spam emails became a serious problem. Today, email services filter them out for us, and our objective is to do the same for DDoS attacks.

For more information about Cloudflare’s DDoS protection, reach out to us or have a go with a hands-on evaluation of Cloudflare’s free plan.

By the way, if you are interested in working in the DDoS Protection engineering team, we're currently hiring in our London and Austin offices. Submit your application here:

 Austin: https://boards.greenhouse.io/cloudflare/jobs/2291040?gh_jid=2291040 London: https://boards.greenhouse.io/cloudflare/jobs/2291038?gh_jid=2291038

Moobot Targets 654 Gbps towards a Magic Transit Customer

Over those ten days, our systems automatically detected and mitigated over 5,000 DDoS attacks against this one customer, mainly UDP floods, SYN floods, ACK floods, and GRE floods. The largest DDoS attack was a UDP flood and lasted a mere 2 minutes. This attack targeted only one IP address but hit multiple ports. The attack originated from 18,705 unique IP addresses, each believed to be a Moobot-infected IoT device.

Attack Distribution by Country - From 100 countries

The attack was observed in Cloudflare’s data centers in 100 countries around the world. Approximately 89% of the attack traffic originated from just 10 countries with the US leading at 41%, followed by South Korea and Japan in second place (12% each), and India in third (10%). What this likely means is that the malware has infected at least 18,705 devices in 100 countries around the world.

Attack Distribution by Country - Top 10

‘Moobot’ sounds like a cute name, but there’s nothing cute about it. According to Netlab 360, Moobot is the codename of a self-propagating Mirai-based malware first discovered in 2019. It infects IoT (Internet of Things) devices using remotely exploitable vulnerabilities or weak default passwords. IoT is a term used to describe smart devices such as security hubs and cameras, smart TVs, smart speakers, smart lights, sensors, and even refrigerators that are connected to the Internet.

Once a device is infected by Moobot, control of the device is transferred to the operator of the command and control (C2) server, who can issue commands remotely such as attacking a target and locating additional vulnerable IoT devices to infect (self-propagation).

Moobot is a Mirai-based botnet, and has similar capabilities (modules) as Mirai:

1. Self-propagation - The self-propagation module is in charge of the botnet’s growth. After an IoT device is infected, it randomly scans the Internet for open telnet ports and reports back to the C2 server. Once the C2 server gains knowledge of open telnet ports around the world, it tries to leverage known vulnerabilities or brute force its way into the IoT devices with common or default credentials.

Self-propagation

2. Synchronized attacks - The C2 server orchestrates a coordinated flood of packets or HTTP requests with the goal of creating a denial of service event for the target's website or service.

Synchronized attacks

The botnet operator may use multiple C2 servers in various locations around the world in order to reduce the risk of exposure. Infected devices may be assigned to different C2 servers varying by region and module; one server for self-propagation and another for launching attacks. Thus if a C2 server is compromised and taken down by law enforcement authorities, only parts of the botnet are deactivated.

This is the second large scale attack in the past few months that we observed on Cloudflare's network. The previous one peaked at 754M packets per second and attempted to take down our routers with a high packet rate. Despite the high packet rate, the 754Mpps attack peaked at a mere 253 Gbps.

As opposed to the high packet rate attack, this attack was a high bit rate attack, peaking at 654 Gbps. Due to the high bit rates of this attack, it seems as though the attacker tried (and failed) to cause a denial of service event by saturating our Internet link capacity. So let’s explore why this attack was not successful.

Cloudflare’s global network capacity is over 42 Tbps and growing. Our network spans more than 200 cities in over 100 countries, including 17 cities in mainland China. It interconnects with over 8,800 networks globally, including major ISPs, cloud services, and enterprises. This level of interconnectivity along with the use of Anycast ensures that our network can easily absorb even the largest attacks.

The Cloudflare Network

After traffic arrives at an edge data center, it is then load-balanced efficiently using our own Layer 4 load-balancer that we built, Unimog, which uses our appliances' health and other metrics to load-balance traffic intelligently within a data center to avoid overwhelming any single server.

Besides the use of Anycast for inter-data center load balancing and Unimog for intra-data center load balancing, we also utilize various forms of traffic engineering in order to deal with sudden changes in traffic loads across our network. We utilize both automatic and manual traffic engineering methods that can be employed by our 24/7/365 Site Reliability Engineering (SRE) team.

These combined factors significantly reduce the likelihood of a denial of service event due to link saturation or appliances being overwhelmed -- and as seen in this attack, no link saturation occurred.

Detecting & Mitigating DDoS attacks

Once traffic arrives at our edge, it encounters our three software-defined DDoS protection systems:

1. Gatebot - Cloudflare’s centralized DDoS protection systems for detecting and mitigating globally distributed volumetric DDoS attacks. Gatebot runs in our network’s core data center. It receives samples from every one of our edge data centers, analyzes them, and automatically sends mitigation instructions when attacks are detected. Gatebot is also synchronized to each of our customers’ web servers to identify its health and triggers mitigation accordingly.

2. dosd (denial of service daemon) - Cloudflare’s decentralized DDoS protection systems. dosd runs autonomously in each server in every Cloudflare data center around the world, analyzing traffic and applying local mitigation rules when needed. Besides being able to detect and mitigate attacks at super-fast speeds, dosd significantly improves our network resilience by delegating the detection and mitigation capabilities to the edge.

3. flowtrackd (flow tracking daemon) - Cloudflare’s TCP state tracking machine for detecting and mitigating the most randomized and sophisticated TCP-based DDoS attacks in unidirectional routing topologies (such as the case for Magic Transit). flowtrackd is able to identify the state of a TCP connection and then drops, challenges, or rate-limits packets that don’t belong to a legitimate connection.

Cloudflare DDoS Protection Lifecycle

The three DDoS protection systems collect traffic samples in order to detect DDoS attacks. The types of traffic data that they sample include:

1. Packet fields such as the source IP, source port, destination IP, destination port, protocol, TCP flags, sequence number, options, and packet rate.

2. HTTP request metadata such as HTTP headers, user agent, query-string, path, host, HTTP method, HTTP version, TLS cipher version, and request rate.

3. HTTP response metrics such as error codes returned by customers’ origin servers and their rates.

Our systems then crunch these sample data points together to form a real-time view of our network’s security posture and our customer’s origin server health. They look for attack patterns and traffic anomalies. When found, a mitigation rule with a dynamically crafted attack signature is generated in real-time. Rules are propagated to the most optimal place for cost-effective mitigation. For example, an L7 HTTP flood might be dropped at L4 to reduce the CPU consumption.

Rules that are generated by dosd and flowtrackd are propagated within a single data center for rapid mitigation. Gatebot’s rules are propagated to all of the edge data centers which then take priority over dosd’s rules for an even and optimal mitigation. Even if the attack is detected in a subset of edge data centers, Gatebot propagates the mitigation instructions to all of Cloudflare’s edge data centers -- effectively sharing the threat intelligence across our network as a form of proactive protection.

In the case of this attack, in each edge data center, dosd generated rules to mitigate the attack promptly. Then as Gatebot received and analyzed samples from the edge, it determined that this was a globally distributed attack. Gatebot propagated unified mitigation instructions to the edge, which prepared each and every one of our 200+ data centers to tackle the attack as the attack traffic may shift to a different data center due to Anycast or traffic engineering.

DDoS attacks obviously pose the risk of an outage and service disruption. But there is another risk to consider -- the cost of mitigation. During these ten days, more than 65 Terabytes of attack traffic were generated by the botnet. However, as part of Cloudflare’s unmetered DDoS protection guarantee, Cloudflare mitigated and absorbed the attack traffic without billing the customer. The customer doesn't need to submit a retroactive credit request. Attack traffic is automatically excluded from our billing system. We eliminated the financial risk.

Magic Transit is Cloudflare’s L3 DDoS Scrubbing service for protecting network infrastructure. As part of our ongoing investment in Magic Transit and our DDoS protection capabilities, we’re excited to talk about a new piece of software helping to protect Magic Transit customers: flowtrackd. flowtrackd is a software-defined DDoS protection system that significantly improves our ability to automatically detect and mitigate even the most complex TCP-based DDoS attacks. If you are a Magic Transit customer, this feature will be enabled by default at no additional cost on July 30, 2020.

In the first quarter of 2020, one out of every two L3/4 DDoS attacks Cloudflare mitigated was an ACK Flood, and over 66% of all L3/4 attacks were TCP based. Most types of DDoS attacks can be mitigated by finding unique characteristics that are present in all attack packets and using that to distinguish ‘good’ packets from the ‘bad’ ones. This is called "stateless" mitigation, because any packet that has these unique characteristics can simply be dropped without remembering any information (or "state") about the other packets that came before it. However, when attack packets have no unique characteristics, then "stateful" mitigation is required, because whether a certain packet is good or bad depends on the other packets that have come before it.

The most sophisticated types of TCP flood require stateful mitigation, where every TCP connection must be tracked in order to know whether any particular TCP packet is part of an active connection. That kind of mitigation is called "flow tracking", and it is typically implemented in Linux by the iptables conntrack module. However, DDoS protection with conntrack is not as simple as flipping the iptable switch, especially at the scale and complexity that Cloudflare operates in. If you're interested to learn more, in this blog we talk about the technical challenges of implementing iptables conntrack.

Complex TCP DDoS attacks pose a threat as they can be harder to detect and mitigate. They therefore have the potential to cause service degradation, outages and increased false positives with inaccurate mitigation rules. So how does Cloudflare block patternless DDoS attacks without affecting legitimate traffic?

Using Cloudflare's traditional products, HTTP applications can be protected by the WAF service, and TCP/UDP applications can be protected by Spectrum. These services are "reverse proxies", meaning that traffic passes through Cloudflare in both directions. In this bidirectional topology, we see the entire TCP flow (i.e., segments sent by both the client and the server) and can therefore track the state of the underlying TCP connection. This way, we know if a TCP packet belongs to an existing flow or if it is an “out of state” TCP packet. Out of state TCP packets look just like regular TCP packets, but they don’t belong to any real connection between a client and a server. These packets are most likely part of an attack and are therefore dropped.

Reverse Proxy: What Cloudflare Sees

While not trivial, tracking TCP flows can be done when we serve as a proxy between the client and server, allowing us to absorb and mitigate out of state TCP floods. However it becomes much more challenging when we only see half of the connection: the ingress flow. This visibility into ingress but not egress flows is the default deployment method for Cloudflare’s Magic Transit service, so we had our work cut out for us in identifying out of state packets.

With Magic Transit, Cloudflare receives inbound internet traffic on behalf of the customer, scrubs DDoS attacks, and routes the clean traffic to the customer’s data center over a tunnel. The data center then responds directly to the eyeball client using a technique known as Direct Server Return (DSR).

Magic Transit: Asymmetric L3 Routing

Using DSR, when a TCP handshake is initiated by an eyeball client, it sends a SYN packet that gets routed via Cloudflare to the origin data center. The origin then responds with a SYN-ACK directly to the client, bypassing Cloudflare. Finally, the client responds with an ACK that once again routes to the origin via Cloudflare and the connection is then considered established.

L3 Routing: What Cloudflare Sees

In a unidirectional flow we don’t see the SYN+ACK sent from the origin to the eyeball client, and therefore can't utilize our existing flow tracking capabilities to identify out of state packets.

To overcome the challenges of unidirectional flows, we recently completed the development and rollout of a new system, codenamed flowtrackd (“flow tracking daemon”). flowtrackd is a state machine that hooks into the network interface. Using only the ingress traffic that routes through Cloudflare, flowtrackd determines whether to forward or drop each received TCP packet based on the state of its related connection.

The state machine that determines the state of the flows was developed in-house and complements Gatebot and dosd. Together Gatebot, dosd, and flowtrackd provide a comprehensive multi layer DDoS protection.

And it works! Less than a day after releasing flowtrackd to an early access customer, flowtrackd automatically detected and mitigated an ACK flood that peaked at 6 million packets per second. No downtime, service disruption, or false positives were reported.

flowtrackd Mitigates 6M pps Flood

As opposed to legacy scrubbing center providers with limited network infrastructures, Cloudflare provides DDoS Protection from every one of our data centers in over 200 locations around the world. We write our own software-defined DDoS protection systems. Notice I say systems, because as opposed to vendors that use a dedicated third party appliance, we’re able to write and spin up whatever software we need, deploy it in the optimal location in our tech stack and are therefore not dependent on other vendors or be limited to the capabilities of one appliance.

flowtrackd joins the Cloudflare DDoS protection family which includes our veteran Gatebot and the younger and energetic dosd. flowtrackd will be available from every one of our data centers, with a total mitigation capacity of over 37 Tbps, protecting our Magic Transit customers against the most complex TCP DDoS attacks.

New to Magic Transit? Replace your legacy provider with Magic Transit and pay nothing until your current contract expires. Offer expires September 1, 2020. Click here for details.

On June 21, Cloudflare automatically mitigated a highly volumetric DDoS attack that peaked at 754 million packets per second. The attack was part of an organized four day campaign starting on June 18 and ending on June 21: attack traffic was sent from over 316,000 IP addresses towards a single Cloudflare IP address that was mostly used for websites on our Free plan. No downtime or service degradation was reported during the attack, and no charges accrued to customers due to our unmetered mitigation guarantee.

The attack was detected and handled automatically by Gatebot, our global DDoS detection and mitigation system without any manual intervention by our teams. Notably, because our automated systems were able to mitigate the attack without issue, no alerts or pages were sent to our on-call teams and no humans were involved at all.

Attack Snapshot - Peaking at 754 Mpps. The two different colors in the graph represent two separate systems dropping packets. 

During those four days, the attack utilized a combination of three attack vectors over the TCP protocol: SYN floods, ACK floods and SYN-ACK floods. The attack campaign sustained for multiple hours at rates exceeding 400-600 million packets per second and peaked multiple times above 700 million packets per second, with a top peak of 754 million packets per second. Despite the high and sustained packet rates, our edge continued serving our customers during the attack without impacting performance at all

Attacks with high bits per second rates aim to saturate the Internet link by sending more bandwidth per second than the link can handle. Mitigating a bit-intensive flood is similar to a dam blocking gushing water in a canal with limited capacity, allowing just a portion through.

Bit Intensive DDoS Attacks as a Gushing River Blocked By Gatebot

In such cases, the Internet service provider may block or throttle the traffic above the allowance resulting in denial of service for legitimate users that are trying to connect to the website but are blocked by the service provider. In other cases, the link is simply saturated and everything behind that connection is offline.

Swarm of Mosquitoes as a Packet Intensive DDoS Attack

However in this DDoS campaign, the attack peaked at a mere 250 Gbps (I say, mere, but ¼ Tbps is enough to knock pretty much anything offline if it isn’t behind some DDoS mitigation service) so it does not seem as the attacker intended to saturate our Internet links, perhaps because they know that our global capacity exceeds 37 Tbps. Instead, it appears the attacker attempted (and failed) to overwhelm our routers and data center appliances with high packet rates reaching 754 million packets per second. As opposed to water rushing towards a dam, flood of packets can be thought of as a swarm of millions of mosquitoes that you need to zap one by one.

Zapping Mosquitoes with Gatebot

Depending on the ‘weakest link’ in a data center, a packet intensive DDoS attack may impact the routers, switches, web servers, firewalls, DDoS mitigation devices or any other appliance that is used in-line. Typically, a high packet rate may cause the memory buffer to overflow and thus voiding the router’s ability to process additional packets. This is because there’s a small fixed CPU cost of handing each packet and so if you can send a lot of small packets you can block an Internet connection not by filling it but by causing the hardware that handles the connection to be overwhelmed with processing.

Another form of DDoS attack is one with a high HTTP request per second rate. An HTTP request intensive DDoS attack aims to overwhelm a web server’s resources with more HTTP requests per second than the server can handle. The goal of a DDoS attack with a high request per second rate is to max out the CPU and memory utilization of the server in order to crash it or prevent it from being able to respond to legitimate requests. Request intensive DDoS attacks allow the attacker to generate much less bandwidth, as opposed to bit intensive attacks, and still cause a denial of service.

So how did we handle 754 million packets per second? First, Cloudflare’s network utilizes BGP Anycast to spread attack traffic globally across our fleet of data centers. Second, we built our own DDoS protection systems, Gatebot and dosd, which drop packets inside the Linux kernel for maximum efficiency in order to handle massive floods of packets. And third, we built our own L4 load-balancer, Unimog, which uses our appliances' health and other various metrics to load-balance traffic intelligently within a data center.

In 2017, we published a blog introducing Gatebot, one of our two DDoS protection systems. The blog was titled Meet Gatebot - a bot that allows us to sleep, and that’s exactly what happened during this attack. The attack surface was spread out globally by our Anycast, then Gatebot detected and mitigated the attack automatically without human intervention. And traffic inside each datacenter was load-balanced intelligently to avoid overwhelming any one machine. And as promised in the blog title, the attack peak did in fact occur while our London team was asleep.

So how does Gatebot work? Gatebot asynchronously samples traffic from every one of our data centers in over 200 locations around the world. It also monitors our customers’ origin server health. It then analyzes the samples to identify patterns and traffic anomalies that can indicate attacks. Once an attack is detected, Gatebot sends mitigation instructions to the edge data centers.

To complement Gatebot, last year we released a new system codenamed dosd (denial of service daemon) which runs in every one of our data centers around the world in over 200 cities. Similarly to Gatebot, dosd detects and mitigates attacks autonomously but in the scope of a single server or data center. You can read more about dosd in our recent blog.

While in recent months we’ve observed a decrease in the size and duration of DDoS attacks, highly volumetric and globally distributed DDoS attacks such as this one still persist. Regardless of the size, type or sophistication of the attack, Cloudflare offers unmetered DDoS protection to all customers and plan levels—including the Free plans.

Cloudflare is protected from attacks by the Gatebot DDoS mitigation pipeline. Gatebot performs hundreds of mitigations a day, shielding our infrastructure and our customers from L3/L4 and L7 attacks. Here is a chart of a count of daily Gatebot actions this year:

In the past, we have blogged about our systems:

• Meet Gatebot, a bot that allows us to sleep

Today, things didn't go as planned.

Cloudflare’s network is large, handles many different types of traffic and mitigates different types of known and not-yet-seen attacks. The Gatebot pipeline manages this complexity in three separate stages:

• attack detection - collects live traffic measurements across the globe and detects attacks

• reactive automation - chooses appropriate mitigations

• mitigations - executes mitigation logic on the edge

The benign-sounding "reactive automation" part is actually the most complicated stage in the pipeline. We expected that from the start, which is why we implemented this stage using a custom Functional Reactive Programming (FRP) framework. If you want to know more about it, see the talk and the presentation.

Our mitigation logic often combines multiple inputs from different internal systems, to come up with the best, most appropriate mitigation. One of the most important inputs is the metadata about our IP address allocations: we mitigate attacks hitting HTTP and DNS IP ranges differently. Our FRP framework allows us to express this in clear and readable code. For example, this is part of the code responsible for performing DNS attack mitigation:

def action_gk_dns(...):

    [...]

    if port != 53:
        return None

    if whitelisted_ip.get(ip):
        return None

    if ip not in ANYCAST_IPS:
        return None
        [...] 

It's the last check in this code that we tried to improve today.

Clearly, the code above is a huge oversimplification of all that goes into attack mitigation, but making an early decision about whether the attacked IP serves DNS traffic or not is important. It's that check that went wrong today. If the IP does serve DNS traffic then attack mitigation is handled differently from IPs that never serve DNS.

Gatebot was created in early 2015. Three years may not sound like much time, but since then we've grown dramatically and added layers of services to our software stack. Many of the internal integration points that we rely on today didn't exist then.

One of them is what we call the Provision API. When Gatebot sees an IP address, it needs to be able to figure out whether or not it’s one of Cloudflare’s addresses. Provision API is a simple RESTful API used to provide this kind of information.

This is a relatively new API, and prior to its existence, Gatebot had to figure out which IP addresses were Cloudflare addresses by reading a list of networks from a hard-coded file. In the code snippet above, the ANYCAST_IPS variable is populated using this file.

Today, in an effort to reclaim some technical debt, we deployed new code that introduced Gatebot to Provision API.

What we did not account for, and what Provision API didn’t know about, was that 1.1.1.0/24 and 1.0.0.0/24 are special IP ranges. Frankly speaking, almost every IP range is "special" for one reason or another, since our IP configuration is rather complex. But our recursive DNS resolver ranges are even more special: they are relatively new, and we're using them in a very unique way. Our hardcoded list of Cloudflare addresses contained a manual exception specifically for these ranges.

As you might be able to guess by now, we didn't implement this manual exception while we were doing the integration work. Remember, the whole idea of the fix was to remove the hardcoded gotchas!

The effect was that, after pushing the new code release, our systems interpreted the resolver traffic as an attack. The automatic systems deployed DNS mitigations for our DNS resolver IP ranges for 17 minutes, between 17:58 and 18:13 May 31st UTC. This caused 1.1.1.1 DNS resolver to be globally inaccessible.

While Gatebot, the DDoS mitigation system, has great power, we failed to test the changes thoroughly. We are using today’s incident to improve our internal systems.

Our team is incredibly proud of 1.1.1.1 and Gatebot, but today we fell short. We want to apologize to all of our customers. We will use today’s incident to improve. The next time we mitigate 1.1.1.1 traffic, we will make sure there is a legitimate attack hitting us.

Around the fall of 2016, we started seeing DDoS attacks that looked a little different than usual. One attack we saw around that time had traffic coming from 52,467 unique IP addresses. The clients weren’t servers or desktop computers; when we tried to connect to the clients over port 80, we got the login pages to CCTV cameras.

Obviously it’s important to lock down IoT devices so that they can’t be co-opted into evil botnet armies, but when we talk to some IoT developers, we hear a few concerning security patterns. We’ll dive into two problematic areas and their solutions: software updates and TLS.

With PCs, the end user is ultimately responsible for securing their devices. People understand that they need to update their computers and phones. Just 4 months after Apple released iOS 10, it was installed on 76% of active devices.

People just don’t know that they are supposed to update IoT things like they are supposed to update their computers because they’ve never had to update things in the past. My parents are never going to install a software update for their thermometer.

And the problem gets worse over time. The longer a device stays on an older software version, the less likely it will be compatible with the newer version. At some point, an update may not be possible anymore. This is a very real concern as the shelf life of a connected thing can be 10 years in the case of a kitchen appliance - have you ever bought a refrigerator?

This is if the device can be patched at all. First, devices that are low battery are programmed not to receive updates because it’s too draining on the battery. Second, IoT devices are too lightweight to run a full operating system, they run just a compiled binary on firmware which means there’s a limit to the code that can later be pushed to it. Some devices cannot receive specific patches.

The other thing we hear about updates from IoT developers is that often they are afraid to push a new update because it could mean breaking hundreds of thousands of devices at once.

All this may not seem like a big deal - ok, so a toaster can get hacked, so what - but two very real things are at stake. First, every device that’s an easy target makes it easier to make other applications a target. Second, once someone is sitting on a device, they are in your network, which can put at stake any traffic sent over the wire.

The security model that worked for PC doesn’t work for IoT — the end user can’t be responsible, and patching isn’t reliable. We need something else. What’s the solution?

Traffic to an IoT device passes through many different networks: the transit provider from the application server, the content delivery network used to deliver device traffic, the ISP to the building where the device sits.

It is at those network layers that protection can be added. As IoT device traffic moves through these networks, packets can be filtered to only let in good traffic. Even if a device is running vulnerable code, filters added in the network level can keep hackers out.

TLS is used in two ways in IoT devices: First, TLS is used to encrypt data in transit. This is used for data privacy and to make it harder to reverse engineer the communications used by the device. Second, devices store client TLS certificates that are used to authenticate the devices to the application - makes it one step harder to fake a device.

There are three problems developers run into when they want to implement TLS in IoT. The first is that while IoT traffic needs to be quick and lightweight, TLS adds an additional two round trips to the start of every session. The second is that certificates can be large files, and device memory is limited in IoT. And the third is that some of the protocols that are being developed for IoT are plaintext by default.

IoT devices run on low power chips. An IoT device may only have 256 or 512 KB of RAM and often need to conserve battery. They send and receive lots of small information constantly. Imagine an internet connected wind sensor - it measures wind speed and every 30 seconds, sends the new wind speed to the application server. It’s just a few bytes of data it needs to get over the wire and it wants to be able to do so without as much overhead as possible to conserve RAM and battery life.

Here’s an HTTP POST to do that:

But let’s say the same device is going to use TLS. Here’s what the same POST looks like with the TLS handshake — this is with TLS 1.2:

Depending on distance between the device and the application server and the latency of the server, this can be hundreds of milliseconds added. The solution is likely the newest version of TLS, TLS 1.3.

TLS 1.3 eliminates a complete round trip in the TLS handshake, which makes TLS much lighter and faster. It cuts the number of round trips in the handshake by half by predicting what key agreement protocol and algorithm the server will decide to use and sends those guessed parameters and the key share directly in the client hello. And if the server likes that, it sends back its own key share for the same algorithm, and the whole handshake is done.

If the same IoT device talks to the same server again, there’s actually no round trip at all. The parameters chosen in the initial handshake are sent alongside application data in the first packet.

Why isn’t every IoT device using 1.3 today? TLS 1.3 is still actively being developed in the IETF standards track and while Chrome as of version 56 in January and Firefox as of version 52 in March support 1.3, not everything does. The biggest problem today are middleboxes that are used by ISP’s and enterprises that panic when they see a 1.3 handshake and close the connection. This also happened when the world was upgrading to TLS 1.2 and middleboxes only understood TLS 1.1, so it’s just a matter of time.

In a TLS handshake, the server can use a server-side TLS certificate to authenticate itself to the client, and the client can use a client-side certificate to authenticate itself to the server. Devices often store certificates to authenticate themselves to the application server. However, device memory is often limited in IoT, and certificates can be large. What can we do?

Most certificates today use the RSA algorithm, which has been around since the 70’s. The certificates are large because the keys in RSA to be secure need to be large - either 1,024 to 2,048 bytes, however, a newer algorithm using elliptic curve cryptography has been in wide use since the early 2000’s that can solve this problem. With elliptic curve cryptography we can use smaller keys with the same level of security as a larger RSA key and save space on the device.

IoT devices need to be lightweight so two emerging protocols are replacing HTTP as the dominant transfer protocol for some IoT devices: MQTT and CoAP.

MQTT is a pub/sub protocol that has been around almost 20 years. In MQTT, a proxy server acts as a broker. An IoT device or web app publishes a message to the broker, and the broker distributes those messages to all the other IoT devices that need to receive that message.

When MQTT was written almost 20 years ago, it was written without security by intention. It was written for oil and gas companies and they were just sending sensor data and no one thought it needed to be encrypted.

CoAP was standardized just three years ago. It has all the same methods as HTTP, but it’s over UDP so it’s really light.

The problem is, if you want to add TLS (DTLS really because CoAP is over UDP), it no longer is light anymore.

It will be quite interesting to see how update mechanisms and TLS implementations change as the number of deployed IoT devices continues to grow. If this type of thing interests you, come join us.

We don't use separate scrubbing boxes or specialized hardware - every one of our edge servers can perform advanced traffic filtering if the need arises. This allows us to scale up our DDoS capacity as we grow. Each of the new servers we add to our datacenters increases our maximum theoretical DDoS “scrubbing” power. It also scales down nicely - in smaller datacenters we don't have to overinvest in expensive dedicated hardware.

During normal operations our attitude to attacks is rather pragmatic. Since the inbound traffic is distributed across hundreds of servers we can survive periodic spikes and small attacks without doing anything. Vanilla Linux is remarkably resilient against unexpected network events. This is especially true since kernel 4.4 when the performance of SYN cookies was greatly improved.

But at some point, malicious traffic volume can become so large that we must take the load off the networking stack. We have to minimize the amount of CPU spent on dealing with attack packets. Cloudflare operates a multi-tenant service and we must always have enough processing power to serve valid traffic. We can't afford to starve our HTTP proxy (nginx) or custom DNS server (named RRDNS, written in Go) of CPU. When the attack size crosses a predefined threshold (which varies greatly depending on specific attack type), we must intervene.

During large attacks we deploy mitigations to reduce the CPU consumed by malicious traffic. We have multiple layers of defense, each tuned to specific attack vector.

First, there is “scattering”. Since we control DNS resolution we are able to move the domains we serve between IP addresses (we call this "scattering"). This is an effective technique as long as the attacks don’t follow the updated DNS resolutions. This often happens for L3 attacks where the attacker has hardcoded the IP address of the target.

Next, there is a wide range of mitigation techniques that leverage iptables, the firewall built in to the Linux kernel. But we don't use it like a conventional firewall, with a static set of rules. We continuously add, tweak and remove rules, based on specific attack characteristics. Over the years we have mastered the most effective iptables extensions:

• xt_bpf

• ipsets

• hashlimits

• connlimit

To make the most of iptables, we built a system to manage the iptables configuration across our entire fleet, allowing us to rapidly deploy rules everywhere. This fits our architecture nicely: due to Anycast, an attack against a single IP will be delivered to multiple locations. Running iptables rules for that IP on all servers makes sense.

Using stock iptables gives us plenty of confidence. When possible we prefer to use off-the-shelf tools to deal with attacks.

Sometimes though, even this is not sufficient. Iptables is fast in the general case, but has its limits. During very large attacks, exceeding 1M packets per second per server, we shift the attack traffic from kernel iptables to a kernel bypass user space program (which we call floodgate). We use a partial kernel bypass solution using Solarflare EF_VI interface. With this on each server we can process more than 5M attack packets per second while consuming only a single CPU core. With floodgate we have comfortable amount of CPU left for our applications, even during the largest network events.

Finally, there are a number of tweaks we can make on at the HTTP layer. For specific attacks we disable HTTP Keep-Alives forcing attackers to re-establish TCP sessions for each request. This sacrifices a bit of performance for valid traffic as well, but is a surprisingly powerful tool throttling many attacks. For other attack patterns we turn the “I’m under attack” mode on, forcing the attack to hit our JavaScript challenge page.

Early on these mitigations were applied manually by our tireless System Reliability Engineers (SRE's). Unfortunately, it turns out that humans under stress... well, make mistakes. We learned it the hard way - one of the most famous incidents happened in March 2013 when a simple typo brought our whole network down.

Humans are also not great at applying precise rules. As our systems grew and mitigations became more complex, having many specific toggles, our SREs got overwhelmed by the details. It was challenging to present all the specific information about the attack to the operator. We often applied overly-broad mitigations, which were unnecessarily affecting legitimate traffic. All that changed with the introduction of Gatebot.

To aid our SREs we developed a fully automatic mitigation system. We call it Gatebot[1].

The main goal of Gatebot was to automate as much of the mitigation workflow as possible. That means: to observe the network and note the anomalies, understand the targets of attacks and their metadata (such as the type of customer involved), and perform appropriate mitigation action.

Nowadays we have multiple Gatebot instances - we call them “mitigation pipelines”. Each pipeline has three parts:

1. “attack detection” or “signal” - A dedicated system detects anomalies in network traffic. This is usually done by sampling a small fraction of the network packets hitting our network, and analyzing them using streaming algorithms. With this we have a real-time view of the current status of the network. This part of the stack is written in Golang, and even though it only examines the sampled packets, it's pretty CPU intensive. It might comfort you to know that at this very moment two big Xeon servers burn all of their combined 48 Skylake CPU cores toiling away counting packets and performing sophisticated analytics looking for attacks.

2. “reactive automation” or “business logic”. For each anomaly (attack) we see who the target is, can we mitigate it, and with what parameters. Depending on the specific pipeline, the business logic may be anything from a trivial procedure to a multi-step process requiring a number of database lookups and potentially confirmation from a human operator. This code is not performance critical and is written in Python. To make it more accessible and readable by others in company, we developed a simple functional, reactive programming engine. It helps us to keep the code clean and understandable, even as we add more steps, more pipelines and more complex logic. To give you a flavor of the complexity: imagine how the system should behave if a customer upgraded a plan during an attack.

3. “mitigation”. The previous step feeds specific mitigation instructions into the centralized mitigation management systems. The mitigations are deployed across the world to our servers, applications, customer settings and, in some cases, to the network hardware.

Gatebot operates constantly, without breaks for lunch. For the iptables mitigations pipelines alone, Gatebot got engaged between 30 and 1500 times a day. Here is a chart of mitigations per day over last 6 months:

Gatebot is much faster and much more precise than even our most experienced SREs. Without Gatebot we wouldn’t be able to operate our service with the appropriate level of confidence. Furthermore, Gatebot has proved to be remarkably adaptable - we started by automating handling of Layer 3 attacks, but soon we proved that the general model works well for automating other things. Today we have more than 10 separate Gatebot instances doing everything from mitigating Layer 7 attacks to informing our Customer Support team of misbehaving customer origin servers.

Since Gatebot’s inception we learned greatly from the "detection / logic / mitigation" workflow. We reused this model in our Automatic Network System which is used to relieve network congestion[2].

Gatebot allows us to protect our users no matter of the plan. Whether you are a on a Free, Pro, Business or Enterprise plan, Gatebot is working for you. This is why we can afford to provide the same level of DDoS protection for all our customers[3].

Dealing with attacks sound interesting? Join our world famous DDoS team in London, Austin, San Francisco and our elite office in Warsaw, Poland.

1. Fun fact: all our components in this area are called “gate-something”, like: gatekeeper, gatesetter, floodgate, gatewatcher, gateman... Who said that naming things must be hard? ↩︎

2. Some of us have argued that this system should be called Netbot. ↩︎

3. Note: there are caveats. Ask your Success Engineer for specifics! ↩︎