A key component of effective corporate network security is establishing end to end visibility across all traffic that flows through the network. Every network engineer needs a complete overview of their network traffic to confirm their security policies work, to identify new vulnerabilities, and to analyze any shifts in traffic behavior. Often, it’s difficult to build out effective network monitoring as teams struggle with problems like configuring and tuning data collection, managing storage costs, and analyzing traffic across multiple visibility tools.

Today, we’re excited to announce that a free version of Cloudflare’s network flow monitoring product, Magic Network Monitoring, is available to all Enterprise Customers. Every Enterprise Customer can configure Magic Network Monitoring and immediately improve their network visibility in as little as 30 minutes via our self-serve onboarding process.

Enterprise Customers can visit the Magic Network Monitoring product page, click “Talk to an expert”, and fill out the form. You’ll receive access within 24 hours of submitting the request. Over the next month, the free version of Magic Network Monitoring will be rolled out to all Enterprise Customers. The product will automatically be available by default without the need to submit a form.

Cloudflare customers can send their network flow data (either NetFlow or sFlow) from their routers to Cloudflare’s network edge.

Magic Network Monitoring will pick up this data, parse it, and instantly provide insights and analytics on your network traffic. These analytics include traffic volume overtime in bytes and packets, top protocols, sources, destinations, ports, and TCP flags.

Let’s review a recent example of how Magic Network Monitoring improved Cloudflare’s own network security and traffic visibility during the Thanksgiving 2023 security incident. Our security team needed a lightweight method to identify malicious packet characteristics in our core data center traffic. We monitored for any network traffic sourced from or destined to a list of ASNs associated with the bad actor. Our security team setup Magic Network Monitoring and established visibility into our first core data center within 24 hours of the project kick-off. Today, Cloudflare continues to use Magic Network Monitoring to monitor for traffic related to bad actors and to provide real time traffic analytics on more than 1 Tbps of core data center traffic.

Magic Network Monitoring - Traffic Analytics

Magic Network Monitoring also improves visibility on any network traffic that doesn’t go through Cloudflare. Imagine that you’re a network engineer at ACME Corporation, and it’s your job to manage and troubleshoot IoT devices in a factory that are connected to the factory’s internal network. The traffic generated by these IoT devices doesn’t go through Cloudflare because it is destined to other devices and endpoints on the internal network. Nonetheless, you still need to establish network visibility into device traffic over time to monitor and troubleshoot the system.

To solve the problem, you configure a router or other network device to securely send encrypted traffic flow summaries to Cloudflare via an IPSec tunnel. Magic Network Monitoring parses the data, and instantly provides you with insights and analytics on your network traffic. Now, when an IoT device goes down, or a connection between IoT devices is unexpectedly blocked, you can analyze historical network traffic data in Magic Network Monitoring to speed up the troubleshooting process.

As cloud networking becomes increasingly prevalent, it is essential for enterprises to invest in visibility across their cloud environments. Let’s say you’re responsible for monitoring and troubleshooting your corporation's cloud network operations which are spread across multiple public cloud providers. You need to improve visibility into your cloud network traffic to analyze and troubleshoot any unexpected traffic patterns like configuration drift that leads to an exposed network port.

To improve traffic visibility across different cloud environments, you can export cloud traffic flow logs from any virtual device that supports NetFlow or sFlow to Cloudflare. In the future, we are building support for native cloud VPC flow logs in conjunction with Magic Cloud Networking. Cloudflare will parse this traffic flow data and provide alerts plus analytics across all your cloud environments in a single pane of glass on the Cloudflare dashboard.

If you’re an existing Enterprise customer, and you want to improve your corporate network security, you can get started right away. Visit the Magic Network Monitoring product page, click “Talk to an expert”, and fill out the form. You’ll receive access within 24 hours of submitting the request. You can begin the self-serve onboarding tutorial, and start monitoring your first batch of network traffic in less than 30 minutes.

Over the next month, the free version of Magic Network Monitoring will be rolled out to all Enterprise Customers. The product will be automatically available by default without the need to submit a form.

If you’re interested in becoming an Enterprise Customer, and have more questions about Magic Network Monitoring, you can talk with an expert. If you’re a free customer, and you’re interested in testing a limited beta of Magic Network Monitoring, you can fill out this form to request access.

It’s hard to imagine life without our smartphones. Whereas computers were mostly fixed and often shared, smartphones meant that every individual on the planet became a permanent, mobile node on the Internet — with some 6.5B smartphones on the planet today.

While that represents an explosion of devices on the Internet, it will be dwarfed by the next stage of the Internet’s evolution: connecting devices to give them intelligence. Already, Internet of Things (IoT) devices represent somewhere in the order of double the number of smartphones connected to the Internet today — and unlike smartphones, this number is expected to continue to grow tremendously, since they aren’t bound to the number of humans that can carry them.

But the exponential growth in devices has brought with it an explosion in risk. We’ve been defending against DDoS attacks from Internet of Things (IoT) driven botnets like Mirai and Meris for years now. They keep growing, because securing IoT devices still remains challenging, and manufacturers are often not incentivized to secure them. This has driven NIST (the U.S. National Institute of Standards and Technology) to actively define requirements to address the (lack of) IoT device security, and the EU isn’t far behind.

It’s also the type of problem that Cloudflare solves best.

Today, we’re excited to announce our Internet of Things platform: with the goal to provide a single pane-of-glass view over your IoT devices, provision connectivity for new devices, and critically, secure every device from the moment it powers on.

It’s common to immediately think of lightbulbs or simple motion sensors when you read “IoT”, but that’s because we often don’t consider many of the devices we interact with on a daily basis as an IoT device.

Think about:

• Almost every payment terminal

• Any modern car with an infotainment or GPS system

• Millions of industrial devices that power — and are critical to — logistics services, industrial processes, and manufacturing businesses

You especially may not realize that nearly every one of these devices has a SIM card, and connects over a cellular network.

Cellular connectivity has become increasingly ubiquitous, and if the device can connect independently of Wi-Fi network configuration (and work out of the box), you’ve immediately avoided a whole class of operational support challenges. If you’ve just read our earlier announcement about the Zero Trust SIM, you’re probably already seeing where we’re headed.

Hundreds of thousands of IoT devices already securely connect to our network today using mutual TLS and our API Shield product. Major device manufacturers use Workers and our Developer Platform to offload authentication, compute and most importantly, reduce the compute needed on the device itself. Cloudflare Pub/Sub, our programmable, MQTT-based messaging service, is yet another building block.

But we realized there were still a few missing pieces: device management, analytics and anomaly detection. There are a lot of “IoT SIM” providers out there, but the clear majority are focused on shipping SIM cards at scale (great!) and less so on the security side (not so great) or the developer side (also not great). Customers have been telling us that they wanted a way to easily secure their IoT devices, just as they secure their employees with our Zero Trust platform.

Cloudflare’s IoT Platform will build in support for provisioning cellular connectivity at scale: we’ll support ordering, provisioning and managing cellular connectivity for your devices. Every packet that leaves each IoT device can be inspected, approved or rejected by policies you create before it reaches the Internet, your cloud infrastructure, or your other devices.

Emerging standards like IoT SAFE will also allow us to use the SIM card as a root-of-trust, storing device secrets (and API keys) securely on the device, whilst raising the bar to compromise.

This also doesn’t mean we’re leaving the world of mutual TLS behind: we understand that not every device makes sense to connect over solely over a cellular network, be it due to per-device costs, lack of coverage, or the need to support an existing deployment that can’t just be re-deployed.

Unlike humans, who need to be able to access a potentially unbounded number of destinations (websites), the endpoints that an IoT device needs to speak to are typically far more bounded. But in practice, there are often few controls in place (or available) to ensure that a device only speaks to your API backend, your storage bucket, and/or your telemetry endpoint.

Our Zero Trust platform, however, has a solution for this: Cloudflare Gateway. You can create DNS, network or HTTP policies, and allow or deny traffic based not only on the source or destination, but on richer identity- and location- based controls. It seemed obvious that we could bring these same capabilities to IoT devices, and allow developers to better restrict and control what endpoints their devices talk to (so they don’t become part of a botnet).

At the same time, we also identified ways to extend Gateway to be aware of IoT device specifics. For example, imagine you’ve provisioned 5,000 IoT devices, all connected over cellular directly into Cloudflare’s network. You can then choose to lock these devices to a specific geography if there’s no need for them to “travel”; ensure they can only speak to your API backend and/or metrics provider; and even ensure that if the SIM is lifted from the device it no longer functions by locking it to the IMEI (the serial of the modem).

Building these controls at the network layer raises the bar on IoT device security and reduces the risk that your fleet of devices becomes the tool of a bad actor.

We’ve talked a lot about security, but what about compute and storage? A device can be extremely secure if it doesn’t have to do anything or communicate anywhere, but clearly that’s not practical.

Simultaneously, doing non-trivial amounts of compute “on-device” has a number of major challenges:

• It requires a more powerful (and thus, more expensive) device. Moderately powerful (e.g. ARMv8-based) devices with a few gigabytes of RAM might be getting cheaper, but they’re always going to be more expensive than a lower-powered device, and that adds up quickly at IoT-scale.

• You can’t guarantee (or expect) that your device fleet is homogenous: the devices you deployed three years ago can easily be several times slower than what you’re deploying today. Do you leave those devices behind?

• The more business logic you have on the device, the greater the operational and deployment risk. Change management becomes critical, and the risk of “bricking” — rendering a device non-functional in a way that you can’t fix it remotely — is never zero. It becomes harder to iterate and add new features when you’re deploying to a device on the other side of the world.

• Security continues to be a concern: if your device needs to talk to external APIs, you have to ensure you have explicitly scoped the credentials they use to avoid them being pulled from the device and used in a way you don’t expect.

We’ve heard other platforms talk about “edge compute”, but in practice they either mean “run the compute on the device” or “in a small handful of cloud regions” (introducing latency) — neither of which fully addresses the problems highlighted above.

Instead, by enabling secure access to Cloudflare Workers for compute, Analytics Engine for device telemetry, D1 as a SQL database, and Pub/Sub for massively scalable messaging — IoT developers can both keep the compute off the device, but still keep it close to the device thanks to our global network (275+ cities and counting).

On top of that, developers can use modern tooling like Wrangler to both iterate more rapidly and deploy software more safely, avoiding the risk of bricking or otherwise breaking part of your IoT fleet.

You can register your interest in our IoT Platform today: we’ll be reaching out over the coming weeks to better understand the problems teams are facing and working to get our closed beta into the hands of customers in the coming months. We’re especially interested in teams who are in the throes of figuring out how to deploy a new set of IoT devices and/or expand an existing fleet, no matter the use-case.

In the meantime, you can start building on API Shield and Pub/Sub (MQTT) if you need to start securing IoT devices today.

One of the underlying questions that drives Platform Week is “how do we enable developers to build full stack applications on Cloudflare?”. With Workers as a serverless environment for easily deploying distributed-by-default applications, KV and Durable Objects for caching and coordination, and R2 as our zero-egress cost object store, we’ve continued to discuss what else we need to build to help developers both build new apps and/or bring existing ones over to Cloudflare’s Developer Platform.

With that in mind, we’re excited to announce the private beta of Cloudflare Pub/Sub, a programmable message bus built on the ubiquitous and industry-standard MQTT protocol supported by tens of millions of existing devices today.

In a nutshell, Pub/Sub allows you to:

• Publish event, telemetry or sensor data from any MQTT capable client (and in the future, other client-facing protocols)

• Write code that can filter, aggregate and/or modify messages as they’re published to the broker using Cloudflare Workers, and before they’re distributed to subscribers, without the need to ferry messages to a single “cloud region”

• Push events from applications in other clouds, or from on-prem, with Pub/Sub acting as a programmable event router or a hook into persistent data storage (such as R2 or KV)

• Move logic out of the client, where it can be hard (or risky!) to push updates, or where running code on devices raises the materials cost (CPU, memory), while still keeping latency as low as possible (your code runs in every location).

And there’s likely a long list of things we haven’t even predicted yet. We’ve seen developers build incredible things on top of Cloudflare Workers, and we’re excited to see what they build with the power of a programmable message bus like Pub/Sub, too.

If you haven’t heard of MQTT before, you might be surprised to know that it’s one of the most pervasive “messaging protocols” deployed today. There are tens of millions (at least!) of devices that speak MQTT today, from connected payment terminals through to autonomous vehicles, cell phones, and even video games. Sensor readings, telemetry, financial transactions and/or mobile notifications & messages are all common use-cases for MQTT, and the flexibility of the protocol allows developers to make trade-offs around reliability, topic hierarchy and persistence specific to their use-case.

We chose MQTT as the foundation for Cloudflare Pub/Sub as we believe in building on top of open, accessible standards, as we did when we chose the Service Worker API as the foundation for Workers, and with our recently announced participation in the Winter Community Group around server-side runtime APIs. We also wanted to enable existing clients an easy path to benefit from Cloudflare’s scale and programmability, and ensure that developers have a rich ecosystem of client libraries in languages they’re familiar with today.

Beyond that, however, we also think MQTT meets the needs of a modern “publish-subscribe” messaging service. It has flexible delivery guarantees, TLS for transport encryption (no bespoke crypto!), a scalable topic creation and subscription model, extensible per-message metadata, and importantly, it provides a well-defined specification with clear error messages.

With that in mind, we expect to support many more “on-ramps” to Pub/Sub: a lot of the best parts of MQTT can be abstracted away from clients who might want to talk to us over HTTP or WebSockets.

Given the ability to write code that acts on every message published to a Pub/Sub Broker, what does it look like in practice?

Here’s a simple-but-illustrative example of handling Pub/Sub messages directly in a Worker. We have clients (in this case, payment terminals) reporting back transaction data, and we want to capture the number of transactions processed in each region, so we can track transaction volumes over time.

Specifically, we:

1. Filter on a specific topic prefix for messages we care about

2. Parse the message for a specific key:value pair as a metric

3. Write that metric directly into Workers Analytics Engine, our new serverless time-series analytics service, so we can directly query it with GraphQL.

This saves us having to stand up and maintain an external metrics service, configure another cloud service, or think about how it will scale: we can do it all directly on Cloudflare.

async function pubsub(
  messages: Array<PubSubMessage>,
  env: any,
  ctx: ExecutionContext
): Promise<Array<PubSubMessage>> {
  
  for (let msg of messages) {
    // Extract a value from the payload and write it to Analytics Engine
    // In this example, a transactionsProcessed counter that our clients are sending
    // back to us.
    if (msg.topic.startsWith(“/transactions/”)) {
      // This is non-blocking, and doesn’t hold up our message
      // processing.
      env.TELEMETRY.writeDataPoint({
        // We label this metric so that we can query against these labels
        labels: [`${msg.broker}.${msg.namespace}`, msg.payload.region, msg.payload.merchantId],
        metrics: [msg.payload.transactionsProcessed ?? 0]
      });
    }
  }

  // Return our messages back to the Broker
  return messages;
}

const worker = {
  async fetch(req: Request, env: any, ctx: ExecutionContext) {
    // Critical: you must validate the incoming request is from your Broker
    // In the future, Workers will be able to do this on your behalf for Workers
    // in the same account as your Pub/Sub Broker.
    if (await isValidBrokerRequest(req)) {

      // Parse the incoming PubSub messages
      let incomingMessages: Array<PubSubMessage> = await req.json();
      
      // Pass the message to our pubsub handler, and capture the returned
      // messages
      let outgoingMessages = await pubsub(incomingMessages, env, ctx);

      // Re-serialize the messages and return a HTTP 200 so our Broker
      // knows we’ve successfully handled them
      return new Response(JSON.stringify(outgoingMessages), { status: 200 });
    }

    return new Response("not a valid Broker request", { status: 403 });
  },
};

export default worker;

We can then query these metrics directly using a familiar language: SQL. Our query takes the metrics we’ve written and gives us a breakdown of transactions processed by our payment devices, grouped by merchant (and again, all on Cloudflare):

SELECT
  label_2 as region,
  label_3 as merchantId,
  sum(metric_1) as total_transactions
FROM TELEMETRY
WHERE
  metric_1 > 0
  AND timestamp >= now() - 604800
GROUP BY
  region,
  merchantId
ORDER BY
  total_transactions DESC
LIMIT 10

You could replace or augment the calls to Analytics Engine with any number of examples:

• Asynchronously writing messages (using ctx.waitUntil) on specific topics to our R2 object storage without blocking message delivery

• Rewriting messages on-the-fly with data populated from KV, before the message is pushed to subscribers

• Aggregate messages based on their payload and HTTP POST them to legacy infrastructure hosted outside of Cloudflare

Pub/Sub gives you a way to get data into Cloudflare’s network, filter, aggregate and/or mutate it, and push it back out to subscribers — whether there’s 10, 1,000 or 10,000 of them listening on that topic.

As we often like to say: we’re just getting started. The private beta for Pub/Sub is just the beginning of our journey, and we have a long list of capabilities we’re already working on.

Critically, one of our priorities is to cover as much of the MQTT v5.0 specification as we can, so that customers can migrate existing deployments and have it “just work”. Useful capabilities like shared subscriptions that allow you to load-balance messages across many subscribers; wildcard subscriptions (both single- and multi-tier) for aggregation use cases, stronger delivery guarantees (QoS), and support for additional authentication modes (specifically, Mutual TLS) are just a few of the things we’re working on.

Beyond that, we’re focused on making sure Pub/Sub’s developer experience is the best it  can be, and during the beta we’ll be:

• Supporting a new set of “pubsub” sub-commands in Wrangler, our developer CLI, so that getting started is as low-friction as possible

• Building ‘native’ bindings (similar to how Workers KV operates) that allow you to publish messages and subscribe to topics directly from Worker code, regardless of whether the message originates from (or is destined for) a client beyond Cloudflare

• Exploring more ways to publish & subscribe from non-MQTT based clients, including HTTP requests and WebSockets, so that integrating existing code is even easier.

Our developer documentation will cover these capabilities as we land them.

We’re also aware that pricing is a huge part of developer experience, and are committed to ensuring that there is an accessible and flexible free tier. We want to enable developers to experiment, prototype and solve problems we haven’t thought of yet. We’ll be sharing more on pricing during the course of the beta.

If you want to start using Pub/Sub, sign up for the private beta: we plan to start enabling access within the next month. We’re looking forward to collecting feedback from developers and seeing what folks start to build.

In the meantime, review the brand-new Pub/Sub developer documentation to understand how Pub/Sub works under the hood, the MQTT protocol, and how it integrates with Cloudflare Workers.

There is probably an insecure device with an exploitable vulnerability sitting in your house. And your office. And probably even your child’s school. Cameras, printers, speakers, access control readers, thermostats, even heart monitors... all of these devices are, or can be, Internet of Things (IoT) devices. These IoT devices are seamlessly integrated into our modern lives to improve efficiency and control of our environments — yet they are notoriously insecure. This is due to the constrained nature of device hardware and their limited computational capacity, which often lead to minimize access controls, hard-coded passwords, and an inability to patch remotely.

The reality of this threat can play out dramatically. Take, for example, the 2016 Mirai botnet attack, in which hackers exploited millions of IoT devices to become a large-scale botnet network capable of launching DDoS attacks that took down major portions of the Internet, including Twitter, the Guardian, and CNN. These types of attacks are hardly an infrequent occurrence. Cloudflare experienced this reality firsthand in March 2021, when one of our potential vendors for physical security cameras, Verkada, was compromised. The incident allowed a hacker to access Verkada's internal support tools to manage the cameras remotely, enabling them to attempt to move laterally to other devices in the network. Thankfully, our use of a Zero Trust model at Cloudflare prevented the hackers from moving laterally; the incident affected the cameras and nothing else.

But even if we're unaffected, we view security incidents as a challenge to take our security products one step further. So we asked ourselves: can we use our own products to secure IoT devices themselves — even on the same network as production systems - thus ensuring any compromise of these notoriously insecure devices is isolated? Using Cloudflare One, the answer is yes.

As aforementioned, Cloudflare was not impacted by an IoT compromise because we use a Zero Trust model. If the environmental considerations are suitable, such as in Cloudflare's corporate offices, the enterprise network can be ring-fenced with a Zero Trust model. Lateral movement is prevented because hitting any other part of the network requires authentication for access. Given that the IoT device itself is not isolated, careful attention must be paid to put all other network points of entry behind zero trust access.

However, not all environments can be so diligently controlled. Take, for example, data centers. The presence of other vendors, complexity of old production networks, and prevalence of machine-to-machine connections means we can't provide the same zero trust guarantees upon installing an IoT device (such as an access reader or a camera) as we can in our corporate offices. As the complexity of the environment grows, successfully deploying a Zero Trust model to prevent successful lateral movement from IoT devices only becomes more difficult.

Many enterprises deploy their IoT devices on a completely separate network from production, utilizing an out-of-band network or VLAN. While VLANs create isolation at Layer 2, they require access lists at their upstream routed interface to restrict Layer 3 traffic. And many network administrators want additional configurations for more stringent guarantees, such as access lists for both ingress and egress traffic and logging for both successful and denied connections. The management of these access lists can quickly grow in complexity: each new network is another landscape to protect, patch, and detect.

If not properly configured, the security guarantees of VLANs can be easily undermined. Take a network with two separate VLANs; if the access lists are not consistently applied to the two respective switches, a hacked device from one VLAN could easily pivot to a device under the other VLAN. A network administrator could use private VLANs on a per switch basis but, again, this adds additional complexity.

Consistent configurations and architectural choices, from the access list rules to the type of gear used at each site, are required to successfully deploy VLANs to prevent lateral movement. As the number of sites increases and an environment's footprint expands, this only becomes more challenging.

We use our own products to isolate devices without deploying on a separate network. This provides security guarantees without requiring additional hardware. It's a severless, infrastructure-less solution to isolating a network.

How is this done? The hardware device (for our proof of concept, a Verkada camera) is connected to a Power over Ethernet switch, which is configured to tunnel its traffic via Anycast GRE to the Cloudflare Global network. There, we can configure rules from the Cloudflare dash to write egress rules, ensuring that outbound traffic only goes where it is supposed to. Thus, lateral movement is prevented.

This architecture allows a network administrator to control traffic from Layer 3 and up from a single dashboard by applying policies to ingress and egress traffic instantly. This architecture provides a simple set-it-and-forget-it solution that can adapt to a changing environment: protection is vendor agnostic, uses common standards, and log collection is automatic. Compared to other methods of lateral movement protection, Cloudflare provides superior ease, adaptability, and security guarantees necessary to securely manage any environment with IoT devices.

In Q4 2021, we hired a trusted pen test partner, to test this setup by replicating a rogue device attempting to pivot to production networks from behind the Cloudflare One architecture. They were unable to move laterally — as in, this architecture works.

Given that this architecture has been tested, we will begin to formalize the rollout of Cloudflare One internally at our data center sites as part of a Physical Security initiative to keep our most trusted assets secure.

For more information on using Cloudflare One at your enterprise, please reach out to our Sales team.

GL.iNet GL-AR750S in black, same form-factor as the prior white GL.iNet GL-AR750. Credit card for comparison.

Back in April, I wrote about how it was possible to modify a router to encrypt DNS queries over TLS using Cloudflare's 1.1.1.1 DNS Resolver. For this, I used the GL.iNet GL-AR750 because it was pre-installed with OpenWRT (LEDE). The folks at GL.iNet read that blog post and decided to bake DNS-Over-TLS support into their new router using the 1.1.1.1 resolver, they sent me one to take a look at before it's available for pre-release. Their new router can also be configured to force DNS traffic to be encrypted before leaving your local network, which is particularly useful for any IoT or mobile device with hard-coded DNS settings that would ordinarily ignore your routers DNS settings and send DNS queries in plain-text.

In my previous blog post I discussed how DNS was often the weakest link in the chain when it came to browsing privacy; whilst HTTP traffic is increasingly encrypted, this is seldom the case for DNS traffic. This makes it relatively trivial for an intermediary to work out what site you're sending traffic to. In that post, I went through the technical steps required to modify a router using OpenWRT to support DNS Privacy using the DNS-Over-TLS protocol.

GL.iNet were in contact since I wrote the original blog post and very supportive of encrypting DNS queries at the router level. Last week whilst working in Cloudflare's San Francisco office, they reached out to me over Twitter to let me know they were soon to launch a new product with a new web UI containing a "DNS over TLS from Cloudflare" feature and offered to send me the new router before it was even available for pre-order.

On arrival back to our London office, I found a package from Hong Kong waiting for me. Aside from the difference in colour, the AR750S itself is identical in form-factor to the AR750 and was packaged up very similarly. They both have capacity for external storage, an OpenVPN client and can be powered over USB; amongst many other useful functionalities. Alongside the S suffixing the model number, I did notice the new model had some upgraded specs, but I won't dwell on that here.

Below you can see the white AR750 and the new black AR750S router together for comparison. Both have a WAN ethernet port, 2 LAN ethernet ports, a USB port for external storage (plus a micro SD port) and a micro USB power port.

The UI is where the real changes come. In the More Settings tab, there's an option to configure DNS with some nice options.

One notable option is the DNS over TLS from Cloudflare toggle. This option uses the TLS security protocol for encrypting DNS queries, helping increase privacy and prevent eavesdropping.

Another option, Override DNS Settings for All Clients, forcibly overrides the DNS configuration on all clients so that queries are encrypted to the WAN. Unencrypted DNS traffic is intercepted by the router, and by forcing traffic to use it's own local resolver, it is able to transparently rewrite traffic to be encrypted before leaving the router and heading out into the public internet to the upstream resolver - 1.1.1.1.

This option is particularly useful when dealing with embedded systems or IoT devices which don't have configurable DNS options; Smart TVs, TV boxes, your toaster, etc. As this router can proxy traffic over to other Wi-Fi networks (and is portable), this is particularly useful when connecting out to an ordinarily insecure Wi-Fi network; the router can sit in the middle and transparently upgrade unencrypted DNS queries. This is even useful when dealing with phones and tablets where you can't install a DNS-Over-TLS client.

These options both come disabled by default, but can easily be toggled in the UI. As before, you can configure other DNS resolvers by toggling "Manual DNS Server Settings" and entering in any other DNS servers.

There are a number of other cool features I've noticed in this router; for example, the More Settings > Advanced option takes you into a standard LuCi UI that ordinarily comes bundled with LEDE routers. Like previous routers, you can easily SSH into the device and install various program and perform customisations.

For example; after installing TCPDump on the router, I am able to run tcpdump -n -i wlan-sta 'port 853' to see encrypted DNS traffic leaving the router. When I run a DNS query over an unencrypted resolver (using dig A junade.com on my local computer), I can see the outgoing DNS traffic upgraded to encrypted queries on 1.1.1.1 and 1.0.0.1.

If you're interested in learning how to configure 1.1.1.1 on other routers, your computer or your phone - check out the project landing page at https://1.1.1.1/. If you're a developer and want to learn about how you can integrate 1.1.1.1 into your project with either DNS-Over-TLS or DNS-Over-HTTPS, checkout the 1.1.1.1 Developer Documentation.

The core functionality of Spectrum is its ability to block large DDoS attacks. Spectrum benefits from Cloudflare’s existing DDoS mitigation (which this week blocked a 900 Gbps flood). Spectrum’s DDoS protection has already been battle tested. Just soon as we opened up Spectrum for beta, Spectrum received its first SYN flood.

One of Spectrum's earliest deployments was in front of Hypixel’s infrastructure. Hypixel runs the largest minecraft server, and because gamers can be - uh, passionate - they were one of the earliest targets of the terabit-per-second Mirai botnet. “Hypixel was one of the first subjects of the Mirai botnet DDoS attacks and frequently receives large attacks. Before Spectrum, we had to rely on unstable services and techniques that increased latency, worsening user's experience. Now, we're able to be continually protected without added latency, which makes it the best option for any latency and uptime sensitive service such as online gaming,” Bruce Blair, the CTO at Hypixel, told us.

Another early team we talked to about Spectrum was the security team at Montecito Bank & Trust. As a financial institution, they have a highly technical and active security team; they were also one of the first customers to use Cloudflare’s DNSSEC when it was brand new. Paul Abramson, Montecito Bank & Trust’s Director of Technology told us, “We were looking for a security solution to protect additional services like email and SSH so that if we are subject to attack, our operations can continue to run reliably and securely.”

Security and encryption go hand in hand. With Spectrum, you can terminate TLS at Cloudflare’s edge. The main benefit of TLS termination at the edge is that is speeds up performance (there’s less distance to travel for the three round trips of the TLS handshake).

We think the most interesting outcome is that just by adding support for TLS in the client, Cloudflare can now add encryption to legacy protocols and services that don’t traditionally support encrypted transit.

Spectrum integrates with Cloudflare’s IP Firewall so that you can choose which connections should be forwarded to your servers and which should be blocked at Cloudflare’s edge.

This can be managed via API too, so you can write scripts that allow and deny access on the fly.

curl -X POST "https://api.cloudflare.com/client/v4/user/firewall/access_rules/rules" \
     -H "X-Auth-Email: hello@example.com" \
     -H "X-Auth-Key: 0000000000000000000" \
     -H "Content-Type: application/json" \
     --data '{"mode":"block","configuration":{"target":"ip","value":"192.0.2.1"}}'

Many TCP load balancers and proxies can be cumbersome to set up, but Spectrum takes a few clicks. Tito Esterline on our team recorded a demo you can watch below. My suggestion is to play it with audio so you can hear the play by play.

If you want to get started, get in touch with our team. Today Spectrum is available for applications on the Enterprise plan.

Why just Enterprise? While HTTP can use the Host header to identify services, TCP relies on each service having a unique IP address in order to identify it. Since IPv4 addresses are endangered, it’s quite expensive for us to delegate an IP per application and we needed to limit use. We’re actively thinking about ways to bring Spectrum to everyone. One idea is to offer IPv6-only Spectrum to non-Enterprise customers. Another idea is let anyone use Spectrum but pay for the IPv4 address. We’re not sure yet, but if you prefer one to the other, feel free to comment and let us know.

Oh and P.S. If you want to read about how Spectrum works, Marek wrote a great blog post about the Linux behavior that let us build it.

Around the fall of 2016, we started seeing DDoS attacks that looked a little different than usual. One attack we saw around that time had traffic coming from 52,467 unique IP addresses. The clients weren’t servers or desktop computers; when we tried to connect to the clients over port 80, we got the login pages to CCTV cameras.

Obviously it’s important to lock down IoT devices so that they can’t be co-opted into evil botnet armies, but when we talk to some IoT developers, we hear a few concerning security patterns. We’ll dive into two problematic areas and their solutions: software updates and TLS.

With PCs, the end user is ultimately responsible for securing their devices. People understand that they need to update their computers and phones. Just 4 months after Apple released iOS 10, it was installed on 76% of active devices.

People just don’t know that they are supposed to update IoT things like they are supposed to update their computers because they’ve never had to update things in the past. My parents are never going to install a software update for their thermometer.

And the problem gets worse over time. The longer a device stays on an older software version, the less likely it will be compatible with the newer version. At some point, an update may not be possible anymore. This is a very real concern as the shelf life of a connected thing can be 10 years in the case of a kitchen appliance - have you ever bought a refrigerator?

This is if the device can be patched at all. First, devices that are low battery are programmed not to receive updates because it’s too draining on the battery. Second, IoT devices are too lightweight to run a full operating system, they run just a compiled binary on firmware which means there’s a limit to the code that can later be pushed to it. Some devices cannot receive specific patches.

The other thing we hear about updates from IoT developers is that often they are afraid to push a new update because it could mean breaking hundreds of thousands of devices at once.

All this may not seem like a big deal - ok, so a toaster can get hacked, so what - but two very real things are at stake. First, every device that’s an easy target makes it easier to make other applications a target. Second, once someone is sitting on a device, they are in your network, which can put at stake any traffic sent over the wire.

The security model that worked for PC doesn’t work for IoT — the end user can’t be responsible, and patching isn’t reliable. We need something else. What’s the solution?

Traffic to an IoT device passes through many different networks: the transit provider from the application server, the content delivery network used to deliver device traffic, the ISP to the building where the device sits.

It is at those network layers that protection can be added. As IoT device traffic moves through these networks, packets can be filtered to only let in good traffic. Even if a device is running vulnerable code, filters added in the network level can keep hackers out.

TLS is used in two ways in IoT devices: First, TLS is used to encrypt data in transit. This is used for data privacy and to make it harder to reverse engineer the communications used by the device. Second, devices store client TLS certificates that are used to authenticate the devices to the application - makes it one step harder to fake a device.

There are three problems developers run into when they want to implement TLS in IoT. The first is that while IoT traffic needs to be quick and lightweight, TLS adds an additional two round trips to the start of every session. The second is that certificates can be large files, and device memory is limited in IoT. And the third is that some of the protocols that are being developed for IoT are plaintext by default.

IoT devices run on low power chips. An IoT device may only have 256 or 512 KB of RAM and often need to conserve battery. They send and receive lots of small information constantly. Imagine an internet connected wind sensor - it measures wind speed and every 30 seconds, sends the new wind speed to the application server. It’s just a few bytes of data it needs to get over the wire and it wants to be able to do so without as much overhead as possible to conserve RAM and battery life.

Here’s an HTTP POST to do that:

But let’s say the same device is going to use TLS. Here’s what the same POST looks like with the TLS handshake — this is with TLS 1.2:

Depending on distance between the device and the application server and the latency of the server, this can be hundreds of milliseconds added. The solution is likely the newest version of TLS, TLS 1.3.

TLS 1.3 eliminates a complete round trip in the TLS handshake, which makes TLS much lighter and faster. It cuts the number of round trips in the handshake by half by predicting what key agreement protocol and algorithm the server will decide to use and sends those guessed parameters and the key share directly in the client hello. And if the server likes that, it sends back its own key share for the same algorithm, and the whole handshake is done.

If the same IoT device talks to the same server again, there’s actually no round trip at all. The parameters chosen in the initial handshake are sent alongside application data in the first packet.

Why isn’t every IoT device using 1.3 today? TLS 1.3 is still actively being developed in the IETF standards track and while Chrome as of version 56 in January and Firefox as of version 52 in March support 1.3, not everything does. The biggest problem today are middleboxes that are used by ISP’s and enterprises that panic when they see a 1.3 handshake and close the connection. This also happened when the world was upgrading to TLS 1.2 and middleboxes only understood TLS 1.1, so it’s just a matter of time.

In a TLS handshake, the server can use a server-side TLS certificate to authenticate itself to the client, and the client can use a client-side certificate to authenticate itself to the server. Devices often store certificates to authenticate themselves to the application server. However, device memory is often limited in IoT, and certificates can be large. What can we do?

Most certificates today use the RSA algorithm, which has been around since the 70’s. The certificates are large because the keys in RSA to be secure need to be large - either 1,024 to 2,048 bytes, however, a newer algorithm using elliptic curve cryptography has been in wide use since the early 2000’s that can solve this problem. With elliptic curve cryptography we can use smaller keys with the same level of security as a larger RSA key and save space on the device.

IoT devices need to be lightweight so two emerging protocols are replacing HTTP as the dominant transfer protocol for some IoT devices: MQTT and CoAP.

MQTT is a pub/sub protocol that has been around almost 20 years. In MQTT, a proxy server acts as a broker. An IoT device or web app publishes a message to the broker, and the broker distributes those messages to all the other IoT devices that need to receive that message.

When MQTT was written almost 20 years ago, it was written without security by intention. It was written for oil and gas companies and they were just sending sensor data and no one thought it needed to be encrypted.

CoAP was standardized just three years ago. It has all the same methods as HTTP, but it’s over UDP so it’s really light.

The problem is, if you want to add TLS (DTLS really because CoAP is over UDP), it no longer is light anymore.

It will be quite interesting to see how update mechanisms and TLS implementations change as the number of deployed IoT devices continues to grow. If this type of thing interests you, come join us.

It is fitting, therefore, that it is under Estonia’s Presidency that the European Commission announced a number of initiatives last month aimed at stepping up the European Union’s cybersecurity capacity and response to cyber attacks, while laying the foundations for increased cyber awareness and better cyber hygiene overall.

This EU’s Cybersecurity Strategy is a welcome initiative, as we already know that the overall cyber threat level is rising. At Cloudflare, we deal with a new type of DDoS attack every 3 minutes, and it has been that way for the last 6 months. This year alone, we've seen a DDoS attack that peaked at 300 Mpps and another at 480 Gbps. Furthermore, as DDoS mitigation companies like Cloudflare have become adept at handling 'traditional' DDoS attacks, the attackers have also adapted and increasingly try out new techniques.

In its Communication announcing the Cybersecurity Strategy, the European Commission sets out a multi-pronged approach to ensuring that Europe is better placed to face the rise in cybercrime, increasingly sophisticated cyber tools leveraged for malicious purposes, and attacks on critical infrastructure. The proposals range from educational initiatives to encourage increased cyber awareness and skills, to investment in research projects and public-private partnerships where technology in cybersecurity and industrial capabilities are developed, to encouraging the use of cyber secure tools in eGovernment operations.

Cybersecurity is a common societal challenge which should involve multiple layers of stakeholders, including industry, Government and individuals. The cybersecurity industry can, however, play a key role in helping the fight against cybercrime and attacks by providing training and educational information to better inform policy makers, politicians and law enforcement on what is happening on the ground, and highlight emerging technologies and best practices. Companies such as Cloudflare are on the front line, reacting and adapting to dynamic and evolving threat landscapes, such as that recently seen with infected IoT devices. We are, in a sense, in a somewhat privileged position, and we want to do and share what we can to help raise the bar.

Cloudflare has been actively participating in a number of European initiatives which feature in the Commission’s Cybersecurity Strategy. Earlier this year, we joined Europol's Advisory Group on Internet Security to share our knowledge on matters related to internet security and emerging threats, along with other industry peers. We are also participating in the IoT Security Group set up by the European Union Agency for Network and Information Security. We shared our well-known views and strong support for encryption during discussions held by the European Commission on cross-border access to electronic evidence, and we are now participating in some work related to software vulnerability disclosures in Europe, led by the Brussels think-tank CEPS.

Next year, the EU Network and Information Security Directive will usher in a new era of security awareness and protection in the EU. This new legal framework will ensure that security is an essential consideration for an even broader range of actors than before - such as companies in the banking, transport, energy and digital infrastructure sectors - and it asks that businesses take a risk-based approach in their cyber security activities and preparations. While most of the ideas are not new to a security-conscious company like Cloudflare, we are now in the process of preparing for this new framework.

There are numerous strands to the Commission’s Cybersecurity strategy and it will be important that all stakeholders work quickly and cohesively to make the words a reality. However, with all these initiatives in play, Europe will certainly be in a better position to address the latest cybersecurity challenges, while helping ensure that the internet remains open, secure and resilient.

Internet-of-Things technologies inherit many attack vectors that appear in other internet connected devices, however the low-powered hardware-centric nature of embedded systems presents them with unique security threats. Engineers building Internet-of-Things devices must take additional precautions to ensure they do not implement security anti-patterns when addressing new problems, this blog post will investigate four such anti-patterns that have been used by real Internet-of-Things devices.

Atmel ATMEGA8 Microcontroller Wikimedia Commons - CC BY-SA 3.0

Every time your IoT-enabled alarm clock sounds, you may want it to tell your coffee machine to brew some coffee. In order to do this, your coffee machine may subscribe to messages published by your alarm clock. One such way of doing this is to implement the Publish/Subscribe Pattern within the API of the IoT devices, for this example let's assume our alarm clock and coffee machine communicate through HTTP.

In order to subscribe to messages from the alarm clock, the coffee machine sends a HTTP POST request to https://alarmclock/publish with the following body:

{  
   "on":"alarmSound",
   "to":"https:\/\/coffeemachine\/subscribe"
}

This JSON request instructs the alarm clock on every “alarmSound” event to send a HTTP request to the coffee machine. Whilst this may seem a simple and effective way of implementing the Pub/Sub pattern in HTTP, this poses a significant security risk.

By not being able to validate if the receiver of the subscribed message wants the message or not, there is effectively a DDOS vulnerability. An attacker with the ability to set subscriptions on the alarm clock can effectively send HTTP messages to any device or internet property they want. If this is done across enough devices, a DDOS vulnerability is created.

Toast popping out of a toaster or a car driving across a road traffic sensor could be the trigger of a future large scale DDOS against a web property.

For further reading, the limitation of Pub/Sub patterns in IoT within the context of the MQTT protocol are discussed in: Limitations of the Pub/Sub pattern for cloud based IoT and their implications - Happ, D. and Wolisz, A. (2016).

As the embedded systems which power IoT devices have gained additional computational resources, more and more IoT developers have chosen to implement TLS to secure communications. Additionally cryptography libraries such as wolfSSL and mbedTLS, better suited to the performance requirements for embedded systems, have removed many of the engineering and performance complexities of adding TLS support to embedded systems.

One anti-pattern in implementing TLS on IoT devices is running devices themselves as web servers and using self-signed server-side certificates to encrypt such requests. Aside from the performance overhead of running web servers on embedded systems, they also pose a severe security risk by failing to maintain trust relationships.

In place of this; IoT devices can communicate through brokers; such brokers can act as web servers with a signed TLS certificates installed server-side which can be authenticated with devices using the X.509 standard. This can be done effectively by simply using the HTTP protocol. Like other kinds of message brokers, they will receive messages, perform validation, transformation and routing before sending them on by recipients.

Protocols like CoAP are specifically designed for brokerless Machine-to-Machine communication in low power environments, in its current form CoAP uses DTLS (RFC 4347) which supports pre-shared keys, raw public keys and X.509 certificate authentication.

Far from being in a datacenter, IoT devices can be left in insecure environments - as such extra precautions need to be taken to protect the memory on such a device.

Take the example of an IoT traffic counting device, installed in a locked cabinet at a roadside. A malicious adversary can break into the cabinet and steal the device with physical force. With the device in their possession they can extract software from the embedded system chip on the device to obtain the software running on it. By reverse engineering this software they can learn secrets in the memory of the onboard microcontroller.

When IoT devices can contain sensitive data in memory, locking them away may not always be sufficient. One solution to this problem is to encrypt the bootloaders on such devices - this provides a degree of cryptographic assurance that the secrets on the device will remain secret.

Documentation from Atmel AVR231: AES Bootloader

There exists dedicated integrated Controller chips that allow for Hardware-based Key Storage at low cost; these chips can be used to prevent cloning and tampering of Embedded Systems. Examples include Atmel’s CryptoAuthentication chips and Microchip’s PIC24F GB2 microcontroller.

Other precautions should of course be taken, ensuring that keys are unique to a device and that revocation systems are in place, should keys be disclosed for whatever reason.

It is not merely enough to leave your secrets unencrypted on an embedded system in the hope no one will attempt to reverse engineer it. Taking appropriate security steps for the secrets your IoT device will store is vital.

Instead of communicating over an abstract protocol like HTTP, developers may choose to use shortcuts by simply directly connecting to a database server to push data. Aside from being making permissions control harder, this opens up performance difficulties.

Lock contention is one such issue, when running UPDATE queries on rows other devices will end up waiting for locks on the database to be released before other queries can be done. Additionally, polling databases for changes can lead to IoT brokers becoming easily overloaded.

This Anti-Pattern can be mitigated by using a message broker service, exposed by a HTTP API. REST APIs are easier to cache, easier to scale and allow you to vary your database implementation independently of your API.

No matter how closed you think the usage of your IoT API is, take pride in it and seek to make it resilient to the forces of change later on. Abstracting your database away from your devices allows you to introduce a greater degree of security and also prevents other architectural headaches later on.

As the technology that powers the internet matures, so will the attacks it faces. IoT devices are set to see a unique set of security vulnerabilities, different to the set seen by other internet connected devices. As more IoT devices start to be unveiled, new security challenges will also unfold.

In a traditional TLS handshake, the client authenticates the server, and the server doesn’t know too much about the client. However, starting now, Cloudflare is offering enterprise customers TLS with client authentication, meaning that the server additionally authenticates that the client connecting to it is authorized to connect.

TLS Client Authentication is useful in cases where a server is keeping track of hundreds of thousands or millions of clients, as in IoT, or in a mobile app with millions of installs exchanging secure information. For example, an IoT company can issue a unique client certificate per device, and then limit connections to their IoT infrastructure to only their devices by blocking connections where the client doesn’t present a certificate signed by the company’s certificate authority.

Or in the case of a mobile banking app, where the bank wants to ensure customers’ secure financial data doesn’t get stolen by bots spoofing their mobile app, they can issue a unique certificate to every app install and in the TLS handshake validate requests are coming from their mobile app. Client authentication is also useful for VPNs, enterprise networks or staging sites, where corporations and developers need to lock down connections to only laptops and phones owned by their employees and teammates.

You may be thinking - don’t we have API keys for that? But client certificates offer a layer of security that API keys cannot provide. If an API key gets compromised mid-connection, it can be reused to fire its own valid, trusted requests to the backend infrastructure. However, the private key of the client certificate is used to create a digital signature in every TLS connection, and so even if the certificate is sniffed mid-connection, new requests can’t be instantiated with it.

In a handshake with TLS Client Authentication, the server expects the client to present a certificate, and sends the client a client certificate request with the server hello. Then in the key exchange in the next trip to the server, the client also sends its client certificate. The client certificate is then used to sign the TLS handshake and the digital signature is sent to the server for verification. You can see the whole handshake here:

TLS Client Authentication can be CPU intensive to implement - it’s an additional cryptographic operation on every request. And if there’s a flood of invalid traffic, each request in that traffic flood kicks off a verification step. Companies can move the TLS client authentication to Cloudflare’s edge to offload the expensive verification.

If we are performing TLS Client Authentication for a company, the company sends us the root certificate(s) we should validate the client certificates against. Then the company can set TLS Client Authentication to one of two modes: enforce mode returns a 403 and optional custom JSON or HTML when the client certificate is invalid, and report mode forwards all requests to the origin, even if the certificate is invalid. Cloudflare will send a header including the status of the certificate (none, valid, invalid) and the certificate Subject Key Identifier (SKI) to the origin. For companies that use the client certificate for identification, Cloudflare can also forward any field of the client certificate as a custom header.

To use TLS client authentication, you must first set up PKI (Public Key Infrastructure) infrastructure to issue client certificates. If you are interested in running TLS client authentication but don’t have PKI infrastructure set up to issue client certificates, we have open sourced our PKI for you to use. Here is great documentation by our friends at CoreOS on how to use cfssl to issue client certificates. If you prefer not to run your own CA and rely on an established certificate authority, we have partnered with a few certificate authorities who can provide the client certificates for you.

If you are an enterprise customer and would like to get started using TLS client authentication with Cloudflare, reach out to your account team and we’ll help you get setup. If you are not yet an enterprise customer but are interested in trying out TLS client authentication, get in touch.

Within the next year, we’ll be adding TLS client authentication support for all Cloudflare plans. After all, using encryption to make the web more trusted is what we’re about. Stay tuned.

UPDATE - 1/22/19: This functionality has changed and is being incorporated into Cloudflare Access. A beta is currently underway. Apologies for any confusion.

In October, we wrote about a 1.75M rps DDoS attack we mitigated on our network, launched by 52,467 unique IP’s, mostly hacked CCTV cameras.

We continued to see more IoT devices in DDoS attacks, and so we started to put together a security solution to protect the devices from becoming part of the botnet in the first place. Today we’re announcing it: Cloudflare Orbit.

As we talked to IoT companies, over and over again we heard the same thing. In the consumer electronics space, IoT manufacturers were telling us that they were shipping patches to their devices, but their end users didn’t always download and install them. (Reserve your judgment, how many times have you pressed ignore when your phone asked you to update its operating system?) In the industrial control, medical and automotive spaces, where devices are used in life-critical functions, we heard a different story. Even if someone wanted to apply a patch, it just wasn’t that easy. For example, even if the manager of a nuclear power plant wants to update software on their thermostats, shutting down operations long enough to do that means the update has to be scheduled.

This is if the device can be patched at all - most devices are patchable, but up to a point. When Jeep was shown to be vulnerable to remote code execution, Chrysler had to recall 1.4 million vehicles.

This model where the end user is responsible for the security of the overall device is a relic of the PC age, where users knew and understood that their computers could have vulnerabilities, and as their software vendors released patches ––on so called “Patch Tuesdays,” for example––users knew to go and download them.

PC security does not work for IoT. Consumers do not share that similar understanding that they need to update their toasters, lightbulbs and cars, because they’ve never needed to in the past. And since we will never write perfect code, we need a better way of securing devices without waiting for users to do it for us.

Thinking about this challenge, we were reminded of when the ShellShock vulnerability was discovered, Cloudflare was able to automatically keep the requests that would trigger the vulnerability from reaching our customers. With Orbit, Cloudflare can do the same thing, only for devices. For example, when Jeep was shown to be vulnerable, instead of recalling 1.4 million vehicles, Fiat Chrysler could have patched the bug in all the vehicles with just a simple rule in Cloudflare's firewall restricting access to the vulnerable DBUS service listening on port 6667 of every Jeep.

Orbit sits one layer before the device and provides a shield of security, so even if the device is running past its operating system’s expiration date, Cloudflare protects it from exploits. And while devices may be seldom patched, the Cloudflare security team is shipping code every day, adding new firewall rules to Cloudflare’s edge. Think of it like changing IoT to I*oT — devices can still access the Internet, but only after passing through Cloudflare where malicious requests can be filtered.

For the last year, Cloudflare has been working with a number of IoT vendors to develop Orbit. Already more than 120 million IoT devices are safer behind Cloudflare’s network. Lockitron is one of the IoT companies using Cloudflare. “Keeping our products and customers secure is our primary concern,” says Paul Gerhardt, co-founder of Lockitron. “Cloudflare provides an extra layer of security that allows us to keep our devices continually updated and ahead of any vulnerabilities.”

Instead of writing and shipping a patch, IoT companies can write logic on Cloudflare’s edge, and write their own firewall rules to run on Cloudflare, and it updates the Cloudflare Orbit layer immediately, for all of their devices, without their users ever being so much as nudged to install something.

Plus, with requests going through Cloudflare, Cloudflare can compress transmitted data and speed up traffic, meaning less time is spent waiting on open connections and more time left in battery.

A common challenge we heard from IoT device manufacturers was how to authenticate devices and know which connecting clients were authorized company devices, and which were bots only pretending to be. Starting today, Cloudflare now offers Enterprise domains TLS Client Authentication, a TLS handshake where the client authenticates the server’s certificate (as with any TLS handshake) and also the client has a certificate that the server authenticates.

Some IoT vendors already implement their own Client Authentication, but do so at the same origin servers that handle the rest of their IoT infrastructure. Not only is this computationally expensive, but any invalid traffic flood causes a burden on the whole server.

With Client Authentication on Cloudflare, Cloudflare’s edge handles the load of the TLS handshakes, validating the device client certificates and only sending the IoT infrastructure traffic from authorized devices.

“This approach of adding security to the network is extremely important for industrial manufacturers. Being able to patch vulnerabilities from the network rather than at the device level is a major shift in the way we secure IoT devices, and one that is completely necessary.” — Sam Cece, CEO of Swift Sensors, an industrial IoT company.

“Car controllers are IoT devices. Karamba Security hardens these devices and prevents cyberattacks with zero false positives to maintain driver and passenger safety. We view Cloudflare’s Orbit as a complementary solution that enables secure connectivity between the cars’ hardened controllers and the car company’s data center for trusted, over-the-air updates.” — Ami Dotan, CEO of Karamba Security, which is building secure platforms for smart automobiles.

“We are at the beginning of a new era in which a vast number of devices will be connecting to the Internet and security will play a critical role in the successful roll-out and adoption of IoT devices. Cloudflare’s Orbit adds another layer of defense that compliments other security measures such as strong hardware-based device security and helps ensure a safer Internet of Things." — Quinn Li, VP and global head of Qualcomm Ventures, the investment arm of Qualcomm Incorporated, the leading supplier of components for IoT devices.

"IoT devices create a distinct security challenge both because of the inability of most end users to update their software, as well as the cost that manufacturers bear if they release an update that bricks devices. This is even worse for legacy devices, many of which are effectively unpatchable. Cloudflare's Orbit provides a unique approach to help with these challenges, by deploying a defensive layer in the network where security updates can be safely made without end-user intervention or on-device changes." — Michael Freedman, professor of computer science at Princeton University and CTO of Timescale.

Orbit is available now to all IoT device companies. To learn more or get started, visit the site, or get in touch. We’re excited to hear from you.

Our automatic DDoS mitigation systems have been handling these attacks, but we thought it would be interesting to publish some of the details of what we are seeing. In this article we'll share data on two attacks, which are perfect examples of the new trends in DDoS.

CC BY 2.0 image by E Magnuson

In the past we've written extensively about volumetric DDoS attacks and how to mitigate them. The attacks are distinguished by their heavy use of L7 (i.e. HTTP) attacks as opposed to the more familiar SYN floods, ACK floods, and NTP and DNS reflection attacks.

Many DDoS mitigation systems are tuned to handle volumetric L3/4 attacks; in this instance attackers have switched to L7 attacks in an attempt to knock web applications offline.

Seeing the move towards L7 DDoS attacks we put in place a new system that recognizes and blocks these attacks as they happen. The L7 mitigator recognizes attacks against a single host and distributes a fingerprint that protects all 4 million Cloudflare customers. We'll write more about it in the future.

Often when DDoS attacks are reported the size of the attack is reported in Gbps (or even Tbps), but there are many ways to measure the size of an attack.

For L7 HTTP-based attacks it also makes sense to measure requests per second. That's because, unlike volumetric L3/4 attacks, HTTP-based attacks eat up resources by making actual HTTP requests to the attacked server.

Recently we were hit by a couple of unusually large L7 attacks, crossing 1 million HTTP requests per second (1 Mrps). Here is one of them:

This attack continued for 15 minutes. Multiple recent attacks had >1 Mrps and lasted for minutes.

This particular attack peaked at 1.75 Mrps. It was composed of short HTTP requests (around 121 bytes per request), without anything unusual in the HTTP headers. The requests had a fixed Cookie header. We counted 52,467 unique IP addresses taking part in this attack.

Due to the Anycast nature of the Cloudflare network, the malicious traffic was spread across multiple Cloudflare cities and with 100 cities we are able to get a good picture of where the bots are located.

Here are the top affected datacenters:

This attack went largely to our Hong Kong and Prague datacenters. This is another common characteristic; most of the recent attacks looked similar.

Since the attack looks concentrated, we wondered if only a small number of AS numbers (networks) were the source of the attack. Unfortunately no, the IP addresses participating in the flood are evenly distributed. Out of 10,000 random requests we analyzed, we saw source IP addresses from over 300 AS numbers. These are the biggest sources:

   48 AS24086 ; Vietnam
  101 AS4134  ; China
  128 AS7552  ; Vietnam
  329 AS45899 ; Vietnam 
 2366 AS15895 ; Ukraine

These attacks are a new trend, so it's not fair to blame the AS operators for not cleaning up devices participating in them. Having said that, the Ukrainian ISP and Vietnamese AS45899 seem to stand out. We'll get back to those in a moment.

Although requests per second is a common metric for measuring these attacks, it's not the only one. We can also measure the bandwidth used in the attack.

By this count the attack mentioned above was pretty small (since we've got used to DDoS attacks being reported in 100s of Gbps). It peaked at roughly 2Gbps. But recall that these L7 attacks end up hitting a web server and are not simply volumetric: they use server resources.

However, we saw another attack that was unusual in that it was an L7 with similar bandwidth consumption to traditional L3/4 volumetric attacks. First, here's the requests per second graph:

This attack generated "only" 220k requests per second at peak. However, it generated significant inbound bandwidth:

This attack topped out at 360Gbps per second of inbound HTTP traffic. It's pretty unusual for an HTTP attack to generate a substantial amount of network traffic. This attack was special, and was composed of HTTP requests like this:

GET /en HTTP/1.1
User-Agent: <some string>
Cookie: <some cookie>
Host: example.com
Connection: close
Content-Length: 800000

a[]=&b[]=&a[]=&b[]=&a[]=&b[]=&a[]=&b[]=&a[]=&b[]=&a[]=&b[]=...

It's the long payload sent after the request headers that allowed the attackers to generate substantial traffic. Since this attack we've seen similar events with varying parameters in the request body. Sometimes these attacks came as GET requests, sometimes as POST.

Additionally, this particular attack lasted roughly one hour, with 128,833 unique IP addresses. The datacenter distribution was different, with most of the attack concentrated on Frankfurt:

As the attack was composed of a very large number of bots, we expected the AS distribution to be fairly even. Indeed, in the 10,000 request sample we recorded a whopping 737 unique AS numbers. Here are the top sources:

  286 AS45899 ; Vietnam
  314 AS7552  ; Vietnam
  316 AS3462  ; Taiwan
  323 AS18403 ; Vietnam
 1510 AS15895 ; Ukraine

Once again, the Ukrainian ISP and couple of Vietnamese networks are the top hitters.

We wondered why AS15895 was so special. First, we investigated our traffic charts. Here is the inbound traffic we received from them over last 30 days:

The first significant attack was clearly seen as a spike on September 29 and reached 30Gbps. A very similar chart is visible for AS45899:

We can see some smaller attacks attempted around September 26. A couple of days later the attackers turned the throttle up hitting 7.5Gbps non-stop from this ASN. Other AS numbers we investigated reveal a similar story.

While it's not possible for us to investigate all the attacking devices, it is fair to say that these attacks came from Internet-of-Things (IoT) category of devices.

There are multiple hints confirming this theory.

First, all of the attacking devices have port 23 (telnet) open (closing connection immediately) or closed. Never filtered. This is a strong hint that the malware disabled the telnet port just after it installed itself.

Most of the hosts from the Vietnamese networks look like connected CCTV cameras. Multiple have open port 80 with presenting "NETSurveillance WEB" page.

The Ukrainian devices are a bit different though. Most have port 80 closed, making it harder to identify.

We had noticed one device with port 443 open serving a valid TLS cert issued by Western Digital, handling domain device-xxxx.wd2go.com suggesting it was a hard drive (Network Attached Storage to be precise).

We plan to continue our investigation and collaborate with external researchers to find a permanent solution to this rising threat.

Although the most recent attacks have mostly involved Internet-connected cameras, there's no reason to think that they are likely the only source of future DDoS attacks. As more and more devices (fridges, fitness trackers, sleep monitors, ...) are added to the Internet they'll likely be unwilling participants in future attacks.

CC BY 2.0 image by CODE_n

We're hiring multiple roles, including our DDoS mitigation team. Help us save the Internet from DDoS attacks.

Originally this article attributed the Mirai botnet for the shown attacks. We now believe that, for technical reasons, the large-bandwidth attack might not have come from a botnet running the leaked Mirai code.