Say Cheese: a snapshot of the massive DDoS attacks coming from IoT cameras

Over the last few weeks we've seen DDoS attacks hitting our systems that show that attackers have switched to new, large methods of bringing down web applications. They appear to come from an IoT botnet (like Mirai and relations) which were responsible for the large attacks against Brian Krebs.

Our automatic DDoS mitigation systems have been handling these attacks, but we thought it would be interesting to publish some of the details of what we are seeing. In this article we'll share data on two attacks, which are perfect examples of the new trends in DDoS.

CC BY 2.0 image by E Magnuson

In the past we've written extensively about volumetric DDoS attacks and how to mitigate them. The attacks are distinguished by their heavy use of L7 (i.e. HTTP) attacks as opposed to the more familiar SYN floods, ACK floods, and NTP and DNS reflection attacks.

Many DDoS mitigation systems are tuned to handle volumetric L3/4 attacks; in this instance attackers have switched to L7 attacks in an attempt to knock web applications offline.

Seeing the move towards L7 DDoS attacks we put in place a new system that recognizes and blocks these attacks as they happen. The L7 mitigator recognizes attacks against a single host and distributes a fingerprint that protects all 4 million Cloudflare customers. We'll write more about it in the future.

Often when DDoS attacks are reported the size of the attack is reported in Gbps (or even Tbps), but there are many ways to measure the size of an attack.

For L7 HTTP-based attacks it also makes sense to measure requests per second. That's because, unlike volumetric L3/4 attacks, HTTP-based attacks eat up resources by making actual HTTP requests to the attacked server.

Recently we were hit by a couple of unusually large L7 attacks, crossing 1 million HTTP requests per second (1 Mrps). Here is one of them:

This attack continued for 15 minutes. Multiple recent attacks had >1 Mrps and lasted for minutes.

This particular attack peaked at 1.75 Mrps. It was composed of short HTTP requests (around 121 bytes per request), without anything unusual in the HTTP headers. The requests had a fixed Cookie header. We counted 52,467 unique IP addresses taking part in this attack.

Due to the Anycast nature of the Cloudflare network, the malicious traffic was spread across multiple Cloudflare cities and with 100 cities we are able to get a good picture of where the bots are located.

Here are the top affected datacenters:

This attack went largely to our Hong Kong and Prague datacenters. This is another common characteristic; most of the recent attacks looked similar.

Since the attack looks concentrated, we wondered if only a small number of AS numbers (networks) were the source of the attack. Unfortunately no, the IP addresses participating in the flood are evenly distributed. Out of 10,000 random requests we analyzed, we saw source IP addresses from over 300 AS numbers. These are the biggest sources:

   48 AS24086 ; Vietnam
  101 AS4134  ; China
  128 AS7552  ; Vietnam
  329 AS45899 ; Vietnam 
 2366 AS15895 ; Ukraine

These attacks are a new trend, so it's not fair to blame the AS operators for not cleaning up devices participating in them. Having said that, the Ukrainian ISP and Vietnamese AS45899 seem to stand out. We'll get back to those in a moment.

Although requests per second is a common metric for measuring these attacks, it's not the only one. We can also measure the bandwidth used in the attack.

By this count the attack mentioned above was pretty small (since we've got used to DDoS attacks being reported in 100s of Gbps). It peaked at roughly 2Gbps. But recall that these L7 attacks end up hitting a web server and are not simply volumetric: they use server resources.

However, we saw another attack that was unusual in that it was an L7 with similar bandwidth consumption to traditional L3/4 volumetric attacks. First, here's the requests per second graph:

This attack generated "only" 220k requests per second at peak. However, it generated significant inbound bandwidth:

This attack topped out at 360Gbps per second of inbound HTTP traffic. It's pretty unusual for an HTTP attack to generate a substantial amount of network traffic. This attack was special, and was composed of HTTP requests like this:

GET /en HTTP/1.1
User-Agent: <some string>
Cookie: <some cookie>
Host: example.com
Connection: close
Content-Length: 800000

a[]=&b[]=&a[]=&b[]=&a[]=&b[]=&a[]=&b[]=&a[]=&b[]=&a[]=&b[]=...

It's the long payload sent after the request headers that allowed the attackers to generate substantial traffic. Since this attack we've seen similar events with varying parameters in the request body. Sometimes these attacks came as GET requests, sometimes as POST.

Additionally, this particular attack lasted roughly one hour, with 128,833 unique IP addresses. The datacenter distribution was different, with most of the attack concentrated on Frankfurt:

As the attack was composed of a very large number of bots, we expected the AS distribution to be fairly even. Indeed, in the 10,000 request sample we recorded a whopping 737 unique AS numbers. Here are the top sources:

  286 AS45899 ; Vietnam
  314 AS7552  ; Vietnam
  316 AS3462  ; Taiwan
  323 AS18403 ; Vietnam
 1510 AS15895 ; Ukraine

Once again, the Ukrainian ISP and couple of Vietnamese networks are the top hitters.

We wondered why AS15895 was so special. First, we investigated our traffic charts. Here is the inbound traffic we received from them over last 30 days:

The first significant attack was clearly seen as a spike on September 29 and reached 30Gbps. A very similar chart is visible for AS45899:

We can see some smaller attacks attempted around September 26. A couple of days later the attackers turned the throttle up hitting 7.5Gbps non-stop from this ASN. Other AS numbers we investigated reveal a similar story.

While it's not possible for us to investigate all the attacking devices, it is fair to say that these attacks came from Internet-of-Things (IoT) category of devices.

There are multiple hints confirming this theory.

First, all of the attacking devices have port 23 (telnet) open (closing connection immediately) or closed. Never filtered. This is a strong hint that the malware disabled the telnet port just after it installed itself.

Most of the hosts from the Vietnamese networks look like connected CCTV cameras. Multiple have open port 80 with presenting "NETSurveillance WEB" page.

The Ukrainian devices are a bit different though. Most have port 80 closed, making it harder to identify.

We had noticed one device with port 443 open serving a valid TLS cert issued by Western Digital, handling domain device-xxxx.wd2go.com suggesting it was a hard drive (Network Attached Storage to be precise).

We plan to continue our investigation and collaborate with external researchers to find a permanent solution to this rising threat.

Although the most recent attacks have mostly involved Internet-connected cameras, there's no reason to think that they are likely the only source of future DDoS attacks. As more and more devices (fridges, fitness trackers, sleep monitors, ...) are added to the Internet they'll likely be unwilling participants in future attacks.

CC BY 2.0 image by CODE_n

We're hiring multiple roles, including our DDoS mitigation team. Help us save the Internet from DDoS attacks.

Update:

Originally this article attributed the Mirai botnet for the shown attacks. We now believe that, for technical reasons, the large-bandwidth attack might not have come from a botnet running the leaked Mirai code.