iCloud Private Relay is a new Internet privacy service from Apple that allows users with iOS 15, iPadOS 15, or macOS Monterey on their devices and an iCloud+ subscription, to connect to the Internet and browse with Safari in a more secure and private way. Cloudflare is proud to work with Apple to operate portions of Private Relay infrastructure.

In this post, we’ll explain how website operators can ensure the best possible experience for end users using iCloud Private Relay. Additional material is available from Apple, including “Set up iCloud Private Relay on all your devices”, and “Prepare Your Network or Web Server for iCloud Private Relay” which covers network operator scenarios in detail.

The design of the iCloud Private Relay system ensures that no single party handling user data has complete information on both who the user is and what they are trying to access.

To do this, Private Relay uses modern encryption and transport mechanisms to relay traffic from user devices through Apple and partner infrastructure before sending traffic to the destination website.

Here’s a diagram depicting what connection metadata is available to who when not using Private Relay to browse the Internet:

Let’s look at what happens when we add Private Relay to the mix:

By adding two "relays" (labeled “Ingress Proxy” and “Egress Proxy” above), connection metadata is split:

• The user’s original IP address is visible to the access network (e.g. the coffee shop you’re sitting in, or your home ISP) and the first relay (operated by Apple), but the server or website name is encrypted and not visible to either.

  The first relay hands encrypted data to a second relay (e.g. Cloudflare), but is unable to see “inside” the traffic to Cloudflare.

• Cloudflare-operated relays know only that it is receiving traffic from a Private Relay user, but not specifically who or their client IP address. Cloudflare relays then forward traffic on to the destination server.

Splitting connections in this way prevents websites from seeing user IP addresses and minimizes how much information entities “on path” can collect on user behavior.

Much more extensive information on how Private Relay works is available from Apple, including in the whitepaper “iCloud Private Relay Overview” (pdf).

As mentioned above, Cloudflare functions as a second relay in the iCloud Private Relay system. We’re well suited to the task — Cloudflare operates one of the largest, fastest networks in the world. Our infrastructure makes sure traffic reaches every network in the world quickly and reliably, no matter where in the world a user is connecting from.

We’re also adept at building and working with modern encryption and transport protocols, including TLS 1.3 and QUIC. QUIC, and closely related MASQUE, are the technologies that enable Private Relay to efficiently move data between multiple relay hops without incurring performance penalties.

The same building blocks that power Cloudflare products were used to build support for Private Relay: our network, 1.1.1.1, Cloudflare Workers, and software like quiche, our open-source QUIC (and now MASQUE) protocol handling library, which now includes proxy support.

We’ve gone out of our way to ensure the use of iCloud Private Relay does not have any noticeable impact on your websites, APIs, and other content you serve on the web.

IP addresses are often used by website operators to "geolocate" users, with user locations being used to show content specific to certain locations (e.g. search results) or to otherwise customize user experiences. Private Relay is designed to preserve IP address to geolocation mapping accuracy, even while preventing tracking and fingerprinting.

Preserving the ability to derive rough user location ensures that users with Private Relay enabled are able to:

1. See place search and other locally relevant content when they interact with geography-specific content without precise location sharing enabled.

2. Consume content subject to licensing restrictions limiting which regions have access to it (e.g. live sports broadcasts and similar rights-restricted content).

One of the key “acceptance tests” we think about when thinking about geolocating users is the “local pizza test”: with location services disabled, are the results returned for the search term “pizza near me” geographically relevant? Because of the geography preserving and IP address management systems we operate, they are!

At a high-level, here’s how it works:

• Apple relays geolocate user IP addresses and translate them into a “geohash”. Geohashes are compact representations of latitude and longitude. The system includes protections to ensure geohashes cannot be spoofed by clients, and operates with reduced precision to ensure user privacy is maintained. Apple relays do not send user IP addresses onward.

• Cloudflare relays maintain a pool of IP addresses for exclusive use by Private Relay. These IP addresses have been registered with geolocation database providers to correspond to specific cities around the world. When a Private Relay user connects and presents the previously determined geohash, the closest matching IP address is selected.

• Servers see an IP address that corresponds to the original user IP address’s location, without obtaining information that may be used to identify the specific user.

In most parts of the world, Private Relay supports geolocation to the nearest city by default. If users prefer to be located at more coarse location granularity, the option to locate based on country and timezone is available in Private Relay settings.

If your website relies on geolocation of client IP addresses to power or modify user experiences, please ensure your geolocation database is kept up to date. Apple and Cloudflare work directly with every major IP to geolocation provider to ensure they have an accurate mapping of Private Relay egress IP addresses (which present to your server as the client IP address) to geography. These mappings may change from time to time. Using the most up-to-date version of your provider’s database will ensure the most accurate geolocation results for all users, including those using Private Relay.

In addition to making sure your geolocation databases are up-to-date, even greater location accuracy and precision can be obtained by ensuring your origin is reachable via IPv6. Private Relay egress nodes prefer IPv6 whenever AAAA DNS records are available, and use IPv6 egress IP addresses that are geolocated with greater precision than their IPv4 equivalents. This means you can geolocate users to more specific locations (without compromising user privacy) and deliver more relevant content to users as a result.

If you’re a website operator using Cloudflare to protect and accelerate your site, no action is needed from you. Our geolocation feeds used to enrich client requests with location metadata are kept up-to-date and include the information needed to geolocate users using iCloud Private Relay.

One of the more counterintuitive things about performance on the Internet is that adding intermediate network “hops” between a user and a server can often speed up overall network performance, rather than slow it down, if those intermediate hops are well-connected and tuned for speed.

The networks that power iCloud Private Relay are exceptionally well-connected to other networks around the world, and we spend considerable effort squeezing every last ounce of performance out of our systems every day. We even have automated systems, like Argo Smart Routing, that take data on how the Internet is performing and find the best paths across it to ensure consistent performance even in the face of Internet congestion and other “weather”.

Using Private Relay to reach websites instead of going directly to the origin server can result in significant, measured decreases in page load time for clients using Private Relay vs those that are not. That’s pretty neat: increased privacy does not come at the price of reduced page load and render performance when using Private Relay.

To ensure that iCloud Private Relay users have good experiences interacting with your website, you should ensure that any systems that rely on IP address as a signal or way of indexing users properly accommodate many users originating from one or a handful of addresses.

Private Relay’s concentration of users behind a given IP address is similar to commonly deployed enterprise web gateways or carrier grade network address translation (CG-NAT) systems.

As explained in Apple technical documentation, “Private Relay is designed to ensure only valid Apple devices and accounts in good standing are allowed to use the service. Websites that use IP addresses to enforce fraud prevention and anti-abuse measures can trust that connections through Private Relay have been validated at the account and device level by Apple.” Because of these advanced device and user authorization steps, you might consider allowlisting Private Relay IP addresses explicitly. Should you wish to do so, Private Relay’s egress IP addresses are available in CSV form here.

If you as a server operator are interested in managing traffic from users using systems like iCloud Private Relay or similar NAT infrastructure, consider constructing rules using user level identifiers like cookies, and other metadata present including geography.

For Cloudflare customers, our rate limiting and bot management capabilities are well suited to handle traffic from systems like Private Relay. Cloudflare automatically detects when IP addresses are likely to be used by multiple users, tuning our machine learning and other security heuristics accordingly. Additionally, our WAF includes functionality specifically designed to manage traffic originating from shared IP addresses.

As discussed above, IP addresses used by iCloud Private Relay are specific to the service. However, network and server operators (including Cloudflare customers) studying their traffic and logs may notice large amounts of user traffic arriving from Cloudflare’s network, AS13335. These traffic flows originating from AS13335 include forward proxied traffic from iCloud Private Relay, our enterprise web gateway products, and other products including WARP, our consumer VPN.

In the case of Cloudflare customers, traffic traversing our network to reach your Cloudflare proxied property is included in all usage and billing metrics as traffic from any Internet user would be.

CIOs and network administrators may have questions about how iCloud Private Relay interacts with their corporate networks, and how they might be able to use similar technologies to make their networks more secure. Apple's document, “Prepare Your Network or Web Server for iCloud Private Relay” covers network operator scenarios in detail.

Most enterprise networks will not have to do anything to support Private Relay traffic. If the end-to-end encrypted nature of the system creates compliance challenges, local networks can block the use of Private Relay for devices connected to them.

Corporate customers of Cloudflare One services can put in place the name resolution blocks needed to disable Private Relay through their DNS filtering dashboard. Cloudflare One, Cloudflare’s corporate network security suite, includes Gateway, built on the same network and codebase that powers iCloud Private Relay.

iCloud Private Relay is an exciting step forward in preserving user privacy on the Internet, without forcing compromises in performance.

If you’re an iCloud+ subscriber you can enable Private Relay in iCloud Settings on your iPhone, iPad, or Mac on iOS15, iPadOS15, or macOS Monterey.

Cloudflare One helps enterprises build modern enterprise networks, operate efficiently and securely, and throw out on-premise hardware. It’s been more than a year since we announced the product suite, and we wanted to check in on how things are going.

We’re celebrating Chief Information Officers this week. Regardless of the size of their organization, they’ve had a challenging year. Overnight, their teams became responsible for years of digital transformation to prepare their networks and users to support work-from-home and to adopt new technologies. They worked with partners across security, engineering, and people teams to keep their critical infrastructure running.

Today, we want to focus on the problems that CIOs have been able to solve with Cloudflare One in the last year. Customers are using Cloudflare One at a scale we couldn’t have imagined a year ago to solve interesting problems that we didn't know existed yet. We’ll walk through some specific use cases later in the post, but first, let’s recap why we built Cloudflare One, what problems it solves, and some of the new things we’re launching this week.

Cloudflare One allows companies to purchase, provision, and manage connectivity, security, and analytics tools needed to operate a corporate network from one vendor and one control plane.

Historically, CIOs purchased point solutions from dozens of hardware vendors. They assembled a patchwork of appliances and services to keep their organization connected and secure. The band-aids held together for a while, despite the cost and maintenance burden.

However, the growth of what needed to be connected broke this model. Office locations became more distributed and, more recently, remote work became widespread. Applications that only existed in the corporate data center moved to public cloud providers or SaaS models. As these shifts pushed the limits on what these band-aids could support, the attacks against networks and endpoints became more sophisticated.

We talked to customers who explained that these changes presented a hierarchy of problems: at its base layer, they need their users, offices, data centers and clouds connected to each other and to the Internet. Next, they needed to filter the traffic between these entities. Finally, they needed to log, diagnose, and analyze that traffic. Once those initial needs were met, the network security solution needed to be fast and reliable, and comply with local laws and regulations.

Cloudflare runs a global, programmable edge network. We use that network to improve the speed and security of some of the largest websites and services on the Internet. We built Cloudflare One to make that network available to corporate customers to solve their new challenges. Today, Cloudflare helps CIOs deliver connectivity, security, and visibility without sacrificing performance, no matter where a customer or their employees work.

Cloudflare One starts with connectivity. Your team can connect offices, data centers, devices and cloud properties to Cloudflare’s network. We’re flexible with how you want to send that traffic to us. Connect your offices and data centers to Cloudflare through SD-WAN partnerships or soon our Cloudflare for Offices infrastructure. New this week, you can start using IPsec Tunnels in addition to our existing GRE Tunnels.

Connect your internal resources and the rest of the Internet with a lightweight agent. Does your team rely on contractors and unmanaged devices? Connect them to internal tools in a fully agentless mode. We’ll also be announcing new improvements to Cloudflare Tunnel and our network interfacing provisioning to keep making it easier to connect your organization to our global network.

Once connected, Cloudflare’s network provides a comprehensive suite of security functions to protect your traffic. Customers can rely on our network for everything from IP-layer DDoS mitigation to blocking threats with remote browser isolation. Later this week, we’ll be sharing details of new network firewall features that help your team continue to rip out even more boxes.

Beyond securing your organization from threats on the Internet, Cloudflare One also provides your team with comprehensive Zero Trust control over who can access your internal resources and SaaS applications.

Now that traffic is connected and secured through Cloudflare, we can help make you faster. Cloudflare is building the fastest network in the world. You can read more about where we are the fastest today and how we’re working to be the fastest in any location. New this week, we’ll be sharing updates to our network performance and new features that intelligently accelerate packets in our network.

Just being faster is not enough. The network that powers your organization should also be reliable, even despite factors out of your control. Cloudflare’s network is peered with over 10,000 networks around the world. With one of the most interconnected networks, we can find lots of paths from point A to point B when disruptions elsewhere on the Internet occur.

Finally, we hear from more and more customers that they need a global network with localized compliance features. Cloudflare One makes compliance with local data protection regulations easy. Customers can choose where Cloudflare’s network applies security functions and how we store and export your logs. As part of CIO week, we’ll be previewing new features that give your team the ability to create metadata boundaries in our network.

All that said, we think the best way to understand how Cloudflare One works is to walk through the problems that our customers no longer have.

Overall network traffic growth through Cloudflare One has increased by nearly 400% over the last year, with advanced traffic controls and filtering applied at wire-speed to each of those bits.

Cloudflare’s composable traffic filtering stack lets customers pick and choose which security controls to apply to which traffic, allowing for flexibility and specificity in how traffic is managed. Some customers are using simple “4-tuple” rules to allow or deny traffic to their networks based on IP addresses and port numbers, others are writing their own network filters in eBPF (more on this later this week!) to perform custom logic on hundreds of gigabits per second of traffic at a time, and others are using pure Zero Trust architectures with identity-based policy enforcement and endpoint protection integration.

Over a recent (and typical) stretch of 24 hours, customers prevented over 9.3 trillion unwanted packets, requests, and other network “nouns” from reaching their networks with custom rules. These rules can all be managed centrally, impose no performance penalty, and can be enforced on traffic no matter where it is coming from or where it is going, whether that is offices, data centers, or cloud providers.

The same rules and filtering logic are applied to traffic wherever it enters our network. Because our entire edge network is one giant firewall, there is no backhaul required to a central device or network location for a firewall policy to be applied.

We think Cloudflare One’s architectural advantages make for a pretty killer firewall, and the growth in usage we’ve seen bears that out. But what really sets our network and its integrated security functionality apart is our ability to offer Zero Trust controls from the same network, allowing CIOs to think about securing applications and users instead of IP addresses and TCP ports.

Legacy private networks and VPN clients provided brittle connectivity without real security. In most deployments, a user in the private network could connect to any resource unless explicitly prohibited. Security teams had no identity-driven controls and lacked visibility into their network while IT teams struggled with help desk tickets.

Cloudflare Access replaces private network security with a Zero Trust model that also makes any internal application feel like the Internet’s fastest SaaS applications. Customers connect their internal resources to Cloudflare’s network without poking holes in their firewall. Once connected, administrators can build global rules and per-resource rules to control who can log in and how they can connect. Users launch applications with a single click while Cloudflare’s network enforces those rules and accelerates their traffic around the world.

In the past year, customers have protected over 192,000 applications with Zero Trust rules in Cloudflare. These applications range from mission-critical tools that power the business to administrative panels that hold the company’s most sensitive data, and the next version of the new marketing website. Since announcing Cloudflare One last year, we’ve also brought non-HTTP use cases to the browser with SSH and VNC clients rendered without any additional client software.

Regardless of what’s being protected, customers can layer rules starting from “only my team can log in” all the way to “only allow access to this group of users, connecting from a corporate device, with a physical hardkey, from these countries.” We also know that sometimes security needs a second opinion. Earlier this year, we introduced new features that prompt users to input why they are connecting to a resource and require a second admin to sign off on the request in real time.

We also believe that security should never require a compromise in performance. The applications that customers secure with our Zero Trust products benefit from the same routing acceleration that some of the Internet’s largest websites use. We also bring security decisions closer to the user to avoid slowing them down — Cloudflare’s network enforces Zero Trust rules in every one of our 250 data centers around the world, made even faster by running on our own serverless compute platform.

We launched Cloudflare One with the goal of making Zero Trust security accessible to organizations of any size. When we first released Cloudflare Access over three years ago, smaller teams had limited or no options to replace their VPN. They were turned away from vendors who only serviced the enterprise and had to stick to a legacy private network.

We’re excited that more than 10,000 organizations are now protecting their resources without the need to sign a contract with Cloudflare. We’ve also made these tools even more accessible to smaller organizations. Last year, we raised the number of free users that customers could add to their plan to 50 seats.

Zero Trust rules do not just apply to your internal applications. When your users connect to the rest of the Internet, attackers work to phish their passwords, get malware on their devices, and steal their data.

Cloudflare One provides customers with multiple layers of security filters and across multiple on-ramps  that keep your organization safe from data loss and threats. Since last year’s Cloudflare One announcement, over 5,500 organizations secure the traffic leaving their devices, offices, and data centers.

In the last year, the security they deploy has improved every month. Customers rely on the world’s fastest DNS resolver and the intelligence from Cloudflare’s visibility into the Internet to filter DNS traffic for security threats and content categories. Cloudflare filters their network traffic with identity-based policies, block file transfers, and inspect HTTP traffic for viruses. Organizations control which tenants of SaaS applications employees can use and Cloudflare’s network generates a comprehensive Shadow IT report.

When organizations don’t trust anything on the Internet, they can connect to Cloudflare’s isolated browser. Customers can isolate all destinations or just specific ones, without requiring users to use a special browser client or to suffer through legacy approaches to browser isolation like pixel pushing and DOM manipulation. Cloudflare’s network can also add data control directly in the browser — blocking copy-paste, printing, or even text input by user and destination.

All of this functionality is delivered from our entire global network, on bare metal hardware Cloudflare owns and operates in over 250 cities around the world. There are no public clouds in the mix here, and all our services run on every server in every location in the world. There is no location selection of sizing of hardware, physical or virtualized. Every server is capable of processing every customer’s packet.

This unique architecture allows us to build reliable products quickly and efficiently. Our network is now handling more than 1.69Tbps of peak forward proxy traffic per day, our largest customers do traffic measured in hundreds of gigabits per second delivered over single virtual interfaces.

Customers are able to get value both from the connectivity, security and visibility products we offer, but also through the network of our customers themselves. Most Cloudflare One customers have significant interactions with other customer networks connected to Cloudflare, many of them through direct physical connections available in 158 peering facilities around the world.

Tens of thousands of customers solved problems at scale with Cloudflare One in the last year. We also want to highlight a few organizations and their specific journeys migrating to this model since last year’s announcement.

Within the United States Department of Homeland Security, the Cybersecurity and Infrastructure Security Agency (CISA) works as “the nation’s risk advisor.” CISA partners with teams across the public and private sector to secure critical infrastructure across the federal government as well as State, Local, Tribal, and Territorial agencies and departments.

One risk that CISA has repeatedly flagged is the threat of malicious hostnames, phishing emails with malicious links, and untrustworthy upstream Domain Name System resolvers. Attackers can compromise devices and users by tricking those endpoints into sending a DNS query to a specific hostname. When users connect to the destination behind that resolved query, attackers can steal passwords, data, and put malware on the devices.

Earlier this year, CISA and the National Security Agency (NSA) recommended that teams deploy protective DNS resolvers to prevent those attacks from becoming incidents. Unlike standard DNS resolvers, protective DNS resolvers check the hostname being queried to determine if the destination is malicious. If the hostname poses a risk, the resolver blocks the connection by not answering the DNS query.

Earlier this year, CISA announced that they are not only recommending a protective DNS resolver — they are delivering one to their partner agencies. CISA selected Cloudflare and Accenture Federal Services to deliver a joint solution to help the government defend itself against cyberattacks.

Back in 2018, the developer operations team inside of one of the world’s largest telecom and network equipment companies lost patience with their legacy VPN. Developers in their organization relied on the VPN to connect to the tools they needed to do their jobs. The requirement slowed them down and created user headaches, eventually leading to IT help desk tickets.

The leadership team in that group decided to fix their VPN frustrations by getting rid of it. They signed up to use Cloudflare Access, initially with the personal credit of one of the administrators, to move their development tools to a seamless platform that made their internal applications just feel like SaaS applications for their users.

Over the next three years, more departments in the organization became jealous and asked to also deprecate the VPN usage in their group. As thousands of users across the organization moved to a Zero Trust model, their security team began to take advantage of the rules that could be created, and the logs generated without the need for any server-side code changes.

Last month, that security team began using Cloudflare One to build Zero Trust rules for the rest of the Internet. Their organization chose Cloudflare Gateway to replace their legacy DNS filtering solution with a faster, more manageable platform that keeps the 100,000+ team members safe from phishing attacks, malware, and ransomware in any location.

BlockFi’s mission is to bring financial empowerment to traditionally underserved markets. BlockFi’s interest accounts, cryptocurrency-backed loans, rewards cards and crypto trading platforms connect hundreds of thousands of users to new financial tools. As of June 30, 2021, BlockFi supports over 450,000 funded clients and manages more than $10 billion in assets.

Keeping their service available and secure presented new challenges as they grew. BlockFi started their Cloudflare One journey after experiencing a major DDoS attack on its sign-up API. The BlockFi team contacted Cloudflare, and we were able to help mitigate the DDoS and API attacks, getting their systems back up and running within a few hours. BlockFi was then able to block approximately 10 million malicious bots in the first day of the addition of Cloudflare’s Bot Management platform.

Once their public web infrastructure was up and running again, BlockFi started to evaluate how to improve the security of their internal users and applications. BlockFi relied on a private network that used IP addresses to block or allow users to connect, spending engineering time just maintaining IP lists. As users left the office, that model fell apart.

BlockFi solved that challenge by replacing their legacy network with Cloudflare One to bring identity-driven Zero Trust control to their internal resources. Team members connect from any location and authenticate with their single-sign on.

Their security team didn’t stop there. To protect their employees from phishing and malware attacks, BlockFi deployed Cloudflare One’s DNS filtering and Secure Web Gateway to stop attacks that targeted their entire workforce or specific employees.

Our last customer story involves a large VoIP and unified communications infrastructure company that recently came under ransom attack. They quickly (over the course of less than 24 hours) deployed Cloudflare Magic Transit in front of their entire Internet presence, including their corporate and production networks.

Given the nature of Internet telephony, they were very concerned about performance regressions and impact to call quality. Fortunately, deploying Cloudflare actually improved key network quality metrics like latency and jitter, surprising their network administrators.

Cloudflare’s network excels at powering and protecting performance critical workloads where milliseconds matter and reliability is paramount.

Over the course of this week, we’re going to share dozens of new announcements that solve new problems with Cloudflare One. We’re just getting started building the next-generation of the corporate network, so stay tuned to learn more this week.

We’re also grateful for every organization that trusted Cloudflare One to be your corporate network since last year’s launch. For teams who are ready to begin that journey, follow this link to get started today.

Specifically, Google Search will prioritize results based on how pages score on Core Web Vitals, a measurement methodology Cloudflare has worked closely with Google to establish, and we have implemented support for in our analytics tools.

Source: "Search Page Experience Graphic" by Google is licensed under CC BY 4.0

The Core Web Vitals metrics are Largest Contentful Paint (LCP, a loading measurement), First Input Delay (FID, a measure of interactivity), and Cumulative Layout Shift (CLS, a measure of visual stability). Each one is directly associated with user perceptible page experience milestones. All three can be improved using our performance products, and all three can be measured with our Cloudflare Browser Insights product, and soon, with our free privacy-aware Cloudflare Web Analytics.

SEO experts have always suspected faster pages lead to better search ranking. With the recent announcement from Google, we can say with confidence that Cloudflare helps you achieve the web performance trifecta: our product suite makes your site faster, gives you direct visibility into how it is performing (and use that data to iteratively improve), and directly drives improved search ranking and business results.

"Google providing more transparency about how Search ranking works is great for the open Web. The fact they are ranking using real metrics that are easy to measure with tools like Cloudflare's analytics suite makes Google's recent announcement all the more exciting. Cloudflare offers a full set of tools to make sites incredibly fast and measure ‘incredibly’ directly."

– Matt Weinberg, president of Happy Cog, a full-service digital agency.

Cloudflare offers a diverse, easy to deploy set of products to improve page experience for your visitors. We offer a rich, configurable set of tools to improve page speed, which this post is too small to contain. Unlike Fermat, who once famously described a math problem and then said “the margin is too small to contain the solution”, and then let folks spend three hundred plus years trying to figure out his enigma, I’m going to tell you how to solve web performance problems with Cloudflare. Here are the highlights:

The typical website is composed of a mix of static assets, like images and product descriptions, and dynamic content, like the contents of a shopping cart or a user’s profile page. Cloudflare caches customers’ static content at our edge, avoiding the need for a full roundtrip to origin servers each time content is requested. Because our edge network places content very close (in physical terms) to users, there is less distance to travel and page loads are consequently faster. Thanks, Einstein.

And Argo Smart Routing helps speed page loads that require dynamic content. It analyzes and optimizes routing decisions across the global Internet in real-time. Think Waze, the automobile route optimization app, but for Internet traffic.

Just as Waze can tell you which route to take when driving by monitoring which roads are congested or blocked, Smart Routing can route connections across the Internet efficiently by avoiding packet loss, congestion, and outages.

Using caching and Smart Routing directly improves page speed and experience scores like Web Vitals. With Google's recent announcement, this also means improved search ranking.

Caching and Smart Routing are designed to reduce and speed up round trips from your users to your origin servers, respectively. Cloudflare also offers features to optimize the content we do serve.

Cloudflare Image Resizing allows on-demand sizing, quality, and format adjustments to images, including the ability to convert images to modern file formats like WebP and AVIF.

Delivering images this way to your end-users helps you save bandwidth costs and improve performance, since Cloudflare allows you to optimize images already cached at the edge.

For WordPress operators, we recently launched Automatic Platform Optimization (APO). With APO, Cloudflare will serve your entire site from our edge network, ensuring that customers see improved performance when visiting your site. By default, Cloudflare only caches static content, but with APO we can also cache dynamic content (like HTML) so the entire site is served from cache. This removes round trips from the origin drastically improving TTFB and other site performance metrics. In addition to caching dynamic content, APO caches third party scripts to further reduce the need to make requests that leave Cloudflare's edge network.

Reducing load on customer origins and making sure we serve the right content to the right clients at the right time are great, but what if customers want to take things a step further and eliminate origin round trips entirely? What if there was no origin? Before we get into Schrödinger’s cat/server territory, we can make this concrete: Cloudflare offers tools to serve entire websites from our edge, without an origin server being involved at all. For more on Workers Sites, check out our introductory blog post and peruse our Built With Workers project gallery.

As big proponents of dogfooding, many of Cloudflare’s own web properties are deployed to Workers Sites, and we use Web Vitals to measure our customers’ experiences.

Using Workers Sites, our developers.cloudflare.com site, which gets hundreds of thousands of visits a day and is critical to developers building atop our platform, is able to attain incredible Web Vitals scores:

These scores are superb, showing the performance and ease of use of our edge, our static website delivery system, and our analytics toolchain.

As illustrated above, Cloudflare makes it easy to directly measure Web Vitals with Browser Insights. Enabling Browser Insights for websites proxied by Cloudflare takes one click in the Speed tab of the Cloudflare dashboard. And if you’re not proxying sites through Cloudflare, Web Vitals measurements will be supported in our upcoming, free, Cloudflare Web Analytics product that any site, using Cloudflare’s proxy or not, can use.

Web Vitals breaks down user experience into three components:

• Loading: How long did it take for content to become available?

• Interactivity: How responsive is the website when you interact with it?

• Visual stability: How much does the page move around while loading?

This image is reproduced from work created and shared by Google and used according to terms described in the Creative Commons 4.0 Attribution License.

It’s challenging to create a single metric that captures these high-level components. Thankfully, the folks at Google Chrome team have thought about this, and earlier this year introduced three “Core” Web Vitals metrics:  Largest Contentful Paint,  First Input Delay, and Cumulative Layout Shift.

Cloudflare Browser Insights measures all three metrics directly in your users’ browsers, all with one-click enablement from the Cloudflare dashboard.

Once enabled, Browser Insights works by inserting a JavaScript "beacon" into HTML pages. You can control where the beacon loads if you only want to measure specific pages or hostnames. If you’re using CSP version 3, we’ll even automatically detect the nonce (if present) and add it to the script.

To start using Browser Insights, just head over to the Speed tab in the dashboard.

An example Browser Insights report, showing what pages on blog.cloudflare.com need improvement.

Google’s announcement that Web Vitals measurements will be a key part of search ranking starting in May 2021 places even more emphasis on running fast, accessible websites.

Using Cloudflare’s performance tools, like our best-of-breed caching, Argo Smart Routing, content optimization, and Cloudflare Workers® products, directly improves page experience and Core Web Vitals measurements, and now, very directly, where your pages appear in Google Search results. And you don’t have to take our word for this — our analytics tools directly measure Web Vitals scores, instrumenting your real users’ experiences.

We’re excited to help our customers build fast websites, understand exactly how fast they are, and rank highly on Google search as a result. Render on!

Enterprises used to build networks that resembled a castle-and-moat. The walls and moat kept attackers out and data in. Team members entered over a drawbridge and tended to stay inside the walls. Trust folks on the inside of the castle to do the right thing, and deploy whatever you need in the relative tranquility of your secure network perimeter.

The Internet, SaaS, and “the cloud” threw a wrench in that plan. Today, more of the workloads in a modern enterprise run outside the castle than inside. So why are enterprises still spending money building more complicated and more ineffective moats?

Today, we’re excited to share Cloudflare One™, our vision to tackle the intractable job of corporate security and networking.

Cloudflare One combines networking products that enable employees to do their best work, no matter where they are, with consistent security controls deployed globally.

Starting today, you can begin replacing traffic backhauls to security appliances with Cloudflare WARP and Gateway to filter outbound Internet traffic. For your office networks, we plan to bring next-generation firewall capabilities to Magic Transit with Magic Firewall to let you get rid of your top-of-shelf firewall appliances.

With multiple on-ramps to the Internet through Cloudflare, and the elimination of backhauled traffic, we plan to make it simple and cost-effective to manage that routing compared to MPLS and SD-WAN models. Cloudflare Magic WAN will provide a control plane for how your traffic routes through our network.

You can use Cloudflare One today to replace the other function of your VPN: putting users on a private network for access control. Cloudflare Access delivers Zero Trust controls that can replace private network security models. Later this week, we’ll announce how you can extend Access to any application - including SaaS applications. We’ll also preview our browser isolation technology to keep the endpoints that connect to those applications safe from malware.

Finally, the products in Cloudflare One focus on giving your team the logs and tools to both understand and then remediate issues. As part of our Gateway filtering launch this week we’re including logs that provide visibility into the traffic leaving your organization. We’ll be sharing how those logs get smarter later this week with a new Intrusion Detection System that detects and stops intrusion attempts.

Many of those components are available today, some new features are arriving this week, and other pieces will be launching soon. All together, we’re excited to share this vision and for the future of the corporate network.

The demands placed on a corporate network have changed dramatically. IT has gone from a back-office function to mission critical. In parallel with networks becoming more integral, users spread out from offices to work from home. Applications left the datacenter and are now being run out of multiple clouds or are being delivered by vendors directly over the Internet.

Employees sitting inside of an office could connect over a private network to applications running in a datacenter nearby. When team members left the office, they could use a VPN to sneak back onto the network from outside the walls. Branch offices hopped on that same network over expensive MPLS links.

When applications left the data center and users left their offices, organizations responded by trying to force that scattered world into the same castle-and-moat model. Companies purchased more VPN licenses and replaced MPLS links with difficult SD-WAN deployments. Networks became more complex in an attempt to mimic an older model of networking when in reality the Internet had become the new corporate network.

Attackers looking to compromise corporate networks have a multitude of tools at their disposal, and may execute surgical malware strikes, throw a volumetric kitchen sink at your network, or any number of things in between. Traditionally, defense against each class of attack was provided by a separate, specialized piece of hardware running in a datacenter.

Security controls used to be relatively easy when every user and every application sat in the same place. When employees left offices and workloads left data centers, the same security controls struggled to follow. Companies deployed a patchwork of point solutions, attempting to rebuild their topside firewall appliances across hybrid and dynamic environments.

The move to a patchwork model sacrificed more than just defense-in-depth — companies lost visibility into what was happening in their networks and applications. We hear from customers that this capture and standardization of logs has become one of their biggest hurdles. They purchased expensive data ingestion, analysis, storage, and analytics tools.

Enterprises now rely on multiple point solutions that one of the biggest hurdles is the capture and standardization of logs. Increasing regulatory and compliance pressures place more emphasis on data retention and analysis. Splintered security solutions become a data management nightmare.

Without visibility into this new networking model, security teams had to guess at what could go wrong. Organizations who wanted to adopt an “assume breach” model struggled to determine what kind of breach could even occur, so they threw every possible solution at the problem.

We talk to enterprises who purchase new scanning and filtering services, delivered in virtual appliances, for problems they are unsure they have. These teams attempt to remediate every possible event manually, because they lack visibility, rather than targeting specific events and adapting the security model.

Over the last several years, we’ve been assembling the components of Cloudflare One. We launched individual products to target some of these problems one-at-a-time. We’re excited to share our vision for how they all fit together in Cloudflare One.

Cloudflare launched as a reverse proxy. Customers put their Internet-facing properties on our network and their audience connected to those specific destinations through our network. Cloudflare One represents years of launches that allow our network to process any type of traffic flowing in either the “reverse” or “forward” direction.

In 2019, we launched Cloudflare WARP — a mobile application that kept Internet-bound traffic private with an encrypted connection to our network while also making it faster and more reliable. We’re now packaging that same technology into an enterprise version launching this week to connect roaming employees to Cloudflare Gateway.

Your data centers and offices should have the same advantage. We launched Magic Transit last year to secure your networks from IP-layer attacks. Our initial focus with Magic Transit has been delivering best-in-class DDoS mitigation to on-prem networks. DDoS attacks are a persistent thorn in network operators’ sides, and Magic Transit effectively defuses their sting without forcing performance compromises. That rock-solid DDoS mitigation is the perfect platform on which to build higher level security functions that apply to the same traffic already flowing across our network.

Earlier this year, we expanded that model when we launched Cloudflare Network Interconnect (CNI) to allow our customers to interconnect branch offices and data centers directly with Cloudflare. As part of Cloudflare One, we’ll apply outbound filtering to that same connection.

Cloudflare One should not just help your team move to the Internet as a corporate network, it should be faster than the Internet. Our network is carrier-agnostic, exceptionally well-connected and peered, and delivers the same set of services globally. In each of these on-ramps, we’re adding smarter routing based on our Argo Smart Routing technology, which has been shown to reduce latency by 30% or more in the real-world. Security + Performance, because they’re better together.

When users connect to the Internet from branch offices and devices, they skip the firewall appliances that used to live in headquarters altogether. To keep pace, enterprises need a way to secure traffic that no longer lives entirely within their own network. Cloudflare One applies standard security controls to all traffic - regardless of how that connection starts or where in the network stack it lives.

Cloudflare Access starts by introducing identity into Cloudflare’s network. Teams apply filters based on identity and context to both inbound and outbound connections. Every login, request, and response proxies through Cloudflare’s network regardless of the location of the server or user. The scale of our network and its distribution can filter and log enterprise traffic without compromising performance.

Cloudflare Gateway keeps connections to the rest of the Internet safe. Gateway inspects traffic leaving devices and networks for threats and data loss events that hide inside of connections at the application layer. Launching soon, Gateway will bring that same level of control lower in the stack to the transport layer.

You should have the same level of control over how your networks send traffic. We’re excited to announce Magic Firewall, a next-generation firewall for all traffic leaving your offices and data centers. With Gateway and Magic Firewall, you can build a rule once and run it everywhere, or tailor rules to specific use cases in a single control plane.

We know some attacks can’t be filtered because they launch before filters can be built to stop them. Cloudflare Browser, our isolated browser technology gives your team a bulletproof pane of glass from threats that can evade known filters. Later this week, we’ll invite customers to sign up to join the beta to browse the Internet on Cloudflare’s edge without the risk of code leaping out of the browser to infect an endpoint.

Finally, the PKI infrastructure that secures your network should be modern and simpler to manage. We heard from customers who described certificate management as one of the core problems of moving to a better model of security. Cloudflare works with, not against, modern encryption standards like TLS 1.3. Cloudflare made it easy to add encryption to your sites on the Internet with one click. We’re going to bring that ease-of-management to the network functions you run on Cloudflare One.

Cloudflare’s network serves 18 million HTTP requests per second on average. We’ve built logging pipelines that make it possible for some of the largest Internet properties in the world to capture and analyze their logs at scale. Cloudflare One builds on that same capability.

Cloudflare Access and Gateway capture every request, inbound or outbound, without any server-side code changes or advanced client-side configuration. Your team can export those logs to the SIEM provider of your choice with our Cloudflare Logpush service - the same pipeline that exports HTTP request events at scale for public sites. Magic Transit expands that logging capability to entire networks and offices to ensure you never lose visibility from any location.

We’re going beyond just logging events. Available today for your websites, Cloudflare Web Analytics converts logs into insights. We plan to keep expanding that visibility into how your network operates, as well. Just as Cloudflare has replaced the “band-aid boxes” that performed disparate network functions and unified them into a cohesive, adaptable edge, we intend to do the same for the fragmented, hard to use, and expensive security analytics ecosystem. More to come on this soon.

Data and analytics should surface events that a team can remediate. Log systems that lead to one-click fixes can be powerful tools, but we want to make that remediation automatic.

Launching into a closed preview later this week, Cloudflare Intrusion Detection System (IDS) will proactively scan your network for anomalous events and recommend actions or, better yet, take actions for you to remediate problems. We plan to bring that same proactive scanning and remediation approach to Cloudflare Access and Cloudflare Gateway.

Over 25 million Internet properties rely on Cloudflare’s network to reach their audiences. More than 10% of all websites connect through our reverse proxy, including 16% of the Fortune 1000. Cloudflare accelerates traffic for huge chunks of the Internet by delivering services from datacenters around the world.

We deliver Cloudflare One from those same data centers. And critically, every datacenter we operate delivers the same set of services, whether that is Cloudflare Access, WARP, Magic Transit, or our WAF. As an example, when your employees connect through Cloudflare WARP to one of our data centers, there is a real chance they never have to leave our network or that data center to reach the site or data they need. As a result, their entire Internet experience becomes extraordinarily fast, no matter where they are in the world.

We expect that performance bonus to become even more meaningful as browsing moves to Cloudflare’s edge with Cloudflare Browser. The isolated browsers running in Cloudflare’s data centers can request content that sits just centimeters away. Even further, as more web properties rely on Cloudflare Workers to power their applications, entire workflows can stay inside of a data center within 100 ms of your employees.

While many of these features are available today, we’re going to be launching several new features over the next several days as part of Cloudflare’s Zero Trust week. Stay tuned for announcements each day this week that add new pieces to the Cloudflare One featureset.

Most customers do not have to take any action to protect themselves from the newly disclosed vulnerabilities. Some configuration changes are recommended if you are a Cloudflare customer running unpatched versions of Microsoft IIS and have request filtering enabled on your origin or have forced caching of HTTP response code 400 through the use of Cloudflare Workers.

We have not seen any attempted exploitation of the vulnerabilities described in this paper.

Maintaining the integrity of our content caching infrastructure and ensuring our customers are able to quickly and reliably serve the content they expect to their visitors is of paramount importance to us. In practice, Cloudflare ensures caches serve the content they should in two ways:

1. We build our caching infrastructure to behave in ways compliant with industry standards.

2. We actively add defenses to our caching logic to protect customers from common caching pitfalls. We see our job as solving customer problems whenever possible, even if they’re not directly related to using Cloudflare. Examples of this philosophy can be found in how we addressed previously discovered cache attack techniques.

A summary of the three attacks disclosed in the paper and how Cloudflare handles them:

HTTP Header Method Override (HMO):

• Impact: Some web frameworks support headers for overriding the HTTP method sent in the HTTP request. Ex: A GET request sent with X-HTTP-Method: POST will be treated by the origin as a POST request (this is not a standard but something many frameworks support). An attacker can use this behavior to potentially trick a CDN into caching poisoned content.

• Mitigation: We include the following method override headers as part of customer cache keys for requests which include the headers. This ensures that requests made with the headers present do not poison cache contents for requests without them. Note that Cloudflare does not interpret these headers as an actual method override (ie. the GET request in the above example stays a GET request in our eyes). Headers we consider as part of this cache key modification logic are:

  1. X-HTTP-Method-Override

  2. X-HTTP-Method

  3. X-Method-Override

Oversized HTTP Headers (HHO):

• Impact: The attacker sends large headers that a CDN passes through to origin, but are too large for the origin server to handle. If in this case the origin returns an error page that a shared cache deems cacheable it can result in denial of service for subsequent visitors.

• Mitigation: Cloudflare does not cache HTTP status code 400 responses by default, which is the common denial of service vector called out by the exploit authors. Some CDN vendors did cache 400 responses, which created the poisoning vector called out by the exploit authors. Cloudflare customers were never vulnerable if their origins emitted 400 errors in response to oversized headers.

  The one exception to this is Microsoft IIS in specific circumstances. Versions of Microsoft IIS that have not applied the security update for CVE-2019-0941 will return an HTTP 404 response if limits are configured and exceeded for individual request header sizes using the “headerLimits” configuration directive. Shared caches are permitted to cache these 404 responses. We recommend either upgrading IIS or removing headerLimits configuration directives on your origin.

HTTP Meta Characters:

• Impact: Essentially the same attack as oversized HTTP headers, except the attack uses meta characters like \r and \n to cause origins to return errors to shared caches.

• Mitigation: Same as oversized HTTP headers; Cloudflare does not cache 400 errors by default.

In addition to the behavior laid out above, Cloudflare’s caching logic respects origin Cache-Control headers, which allows customers extremely granular control over how our caches behave. We actively work with customers to ensure that they are following best practices for avoiding cache poisoning attacks and add defense in depth through smarter software whenever possible.

We look forward to continuing to work with the security community on issues like those discovered to make the Internet safer and more secure for everyone.

Since then, we’ve been working with industry peers through the Internet Engineering Task Force, including Google Chrome and Mozilla Firefox, to iterate on the HTTP/3 and QUIC standards documents. In parallel with the standards maturing, we’ve also worked on improving support on our network.

We are now happy to announce that QUIC and HTTP/3 support is available on the Cloudflare edge network. We’re excited to be joined in this announcement by Google Chrome and Mozilla Firefox, two of the leading browser vendors and partners in our effort to make the web faster and more reliable for all.

In the words of Ryan Hamilton, Staff Software Engineer at Google, “HTTP/3 should make the web better for everyone. The Chrome and Cloudflare teams have worked together closely to bring HTTP/3 and QUIC from nascent standards to widely adopted technologies for improving the web. Strong partnership between industry leaders is what makes Internet standards innovations possible, and we look forward to our continued work together.”

What does this mean for you, a Cloudflare customer who uses our services and edge network to make your web presence faster and more secure? Once HTTP/3 support is enabled for your domain in the Cloudflare dashboard, your customers can interact with your websites and APIs using HTTP/3. We’ve been steadily inviting customers on our HTTP/3 waiting list to turn on the feature (so keep an eye out for an email from us), and in the coming weeks we’ll make the feature available to everyone.

What does this announcement mean if you’re a user of the Internet interacting with sites and APIs through a browser and other clients? Starting today, you can use Chrome Canary to interact with Cloudflare and other servers over HTTP/3. For those of you looking for a command line client, curl also provides support for HTTP/3. Instructions for using Chrome and curl with HTTP/3 follow later in this post.

Standards innovation on the Internet has historically been difficult because of a chicken and egg problem: which needs to come first, server support (like Cloudflare, or other large sources of response data) or client support (like browsers, operating systems, etc)? Both sides of a connection need to support a new communications protocol for it to be any use at all.

Cloudflare has a long history of driving web standards forward, from HTTP/2 (the version of HTTP preceding HTTP/3), to TLS 1.3, to things like encrypted SNI. We’ve pushed standards forward by partnering with like-minded organizations who share in our desire to help build a better Internet. Our efforts to move HTTP/3 into the mainstream are no different.

Throughout the HTTP/3 standards development process, we’ve been working closely with industry partners to build and validate client HTTP/3 support compatible with our edge support. We’re thrilled to be joined by Google Chrome and curl, both of which can be used today to make requests to the Cloudflare edge over HTTP/3. Mozilla Firefox expects to ship support in a nightly release soon as well.

Bringing this all together: today is a good day for Internet users; widespread rollout of HTTP/3 will mean a faster web experience for all, and today’s support is a large step toward that.

More importantly, today is a good day for the Internet: Chrome, curl, and Cloudflare, and soon, Mozilla, rolling out experimental but functional, support for HTTP/3 in quick succession shows that the Internet standards creation process works. Coordinated by the Internet Engineering Task Force, industry partners, competitors, and other key stakeholders can come together to craft standards that benefit the entire Internet, not just the behemoths.

Eric Rescorla, CTO of Firefox, summed it up nicely: “Developing a new network protocol is hard, and getting it right requires everyone to work together. Over the past few years, we've been working with Cloudflare and other industry partners to test TLS 1.3 and now HTTP/3 and QUIC. Cloudflare's early server-side support for these protocols has helped us work the interoperability kinks out of our client-side Firefox implementation. We look forward to advancing the security and performance of the Internet together.”

Before we dive deeper into HTTP/3, let’s have a quick look at the evolution of HTTP over the years in order to better understand why HTTP/3 is needed.

It all started back in 1996 with the publication of the HTTP/1.0 specification which defined the basic HTTP textual wire format as we know it today (for the purposes of this post I’m pretending HTTP/0.9 never existed). In HTTP/1.0 a new TCP connection is created for each request/response exchange between clients and servers, meaning that all requests incur a latency penalty as the TCP and TLS handshakes are completed before each request.

Worse still, rather than sending all outstanding data as fast as possible once the connection is established, TCP enforces a warm-up period called “slow start”, which allows the TCP congestion control algorithm to determine the amount of data that can be in flight at any given moment before congestion on the network path occurs, and avoid flooding the network with packets it can’t handle. But because new connections have to go through the slow start process, they can’t use all of the network bandwidth available immediately.

The HTTP/1.1 revision of the HTTP specification tried to solve these problems a few years later by introducing the concept of “keep-alive” connections, that allow clients to reuse TCP connections, and thus amortize the cost of the initial connection establishment and slow start across multiple requests. But this was no silver bullet: while multiple requests could share the same connection, they still had to be serialized one after the other, so a client and server could only execute a single request/response exchange at any given time for each connection.

As the web evolved, browsers found themselves needing more and more concurrency when fetching and rendering web pages as the number of resources (CSS, JavaScript, images, …) required by each web site increased over the years. But since HTTP/1.1 only allowed clients to do one HTTP request/response exchange at a time, the only way to gain concurrency at the network layer was to use multiple TCP connections to the same origin in parallel, thus losing most of the benefits of keep-alive connections. While connections would still be reused to a certain (but lesser) extent, we were back at square one.

Finally, more than a decade later, came SPDY and then HTTP/2, which, among other things, introduced the concept of HTTP “streams”: an abstraction that allows HTTP implementations to concurrently multiplex different HTTP exchanges onto the same TCP connection, allowing browsers to more efficiently reuse TCP connections.

But, yet again, this was no silver bullet! HTTP/2 solves the original problem — inefficient use of a single TCP connection — since multiple requests/responses can now be transmitted over the same connection at the same time. However, all requests and responses are equally affected by packet loss (e.g. due to network congestion), even if the data that is lost only concerns a single request. This is because while the HTTP/2 layer can segregate different HTTP exchanges on separate streams, TCP has no knowledge of this abstraction, and all it sees is a stream of bytes with no particular meaning.

The role of TCP is to deliver the entire stream of bytes, in the correct order, from one endpoint to the other. When a TCP packet carrying some of those bytes is lost on the network path, it creates a gap in the stream and TCP needs to fill it by resending the affected packet when the loss is detected. While doing so, none of the successfully delivered bytes that follow the lost ones can be delivered to the application, even if they were not themselves lost and belong to a completely independent HTTP request. So they end up getting unnecessarily delayed as TCP cannot know whether the application would be able to process them without the missing bits. This problem is known as “head-of-line blocking”.

This is where HTTP/3 comes into play: instead of using TCP as the transport layer for the session, it uses QUIC, a new Internet transport protocol, which, among other things, introduces streams as first-class citizens at the transport layer. QUIC streams share the same QUIC connection, so no additional handshakes and slow starts are required to create new ones, but QUIC streams are delivered independently such that in most cases packet loss affecting one stream doesn't affect others. This is possible because QUIC packets are encapsulated on top of UDP datagrams.

Using UDP allows much more flexibility compared to TCP, and enables QUIC implementations to live fully in user-space — updates to the protocol’s implementations are not tied to operating systems updates as is the case with TCP. With QUIC, HTTP-level streams can be simply mapped on top of QUIC streams to get all the benefits of HTTP/2 without the head-of-line blocking.

QUIC also combines the typical 3-way TCP handshake with TLS 1.3's handshake. Combining these steps means that encryption and authentication are provided by default, and also enables faster connection establishment. In other words, even when a new QUIC connection is required for the initial request in an HTTP session, the latency incurred before data starts flowing is lower than that of TCP with TLS.

But why not just use HTTP/2 on top of QUIC, instead of creating a whole new HTTP revision? After all, HTTP/2 also offers the stream multiplexing feature. As it turns out, it’s somewhat more complicated than that.

While it’s true that some of the HTTP/2 features can be mapped on top of QUIC very easily, that’s not true for all of them. One in particular, HTTP/2’s header compression scheme called HPACK, heavily depends on the order in which different HTTP requests and responses are delivered to the endpoints. QUIC enforces delivery order of bytes within single streams, but does not guarantee ordering among different streams.

This behavior required the creation of a new HTTP header compression scheme, called QPACK, which fixes the problem but requires changes to the HTTP mapping. In addition, some of the features offered by HTTP/2 (like per-stream flow control) are already offered by QUIC itself, so they were dropped from HTTP/3 in order to remove unnecessary complexity from the protocol.

QUIC and HTTP/3 are very exciting standards, promising to address many of the shortcomings of previous standards and ushering in a new era of performance on the web. So how do we go from exciting standards documents to working implementation?

Cloudflare's QUIC and HTTP/3 support is powered by quiche, our own open-source implementation written in Rust.

You can find it on GitHub at github.com/cloudflare/quiche.

We announced quiche a few months ago and since then have added support for the HTTP/3 protocol, on top of the existing QUIC support. We have designed quiche in such a way that it can now be used to implement HTTP/3 clients and servers or just plain QUIC ones.

As mentioned above, we have started on-boarding customers that signed up for the waiting list. If you are on the waiting list and have received an email from us communicating that you can now enable the feature for your websites, you can simply go to the Cloudflare dashboard and flip the switch from the "Network" tab manually:

We expect to make the HTTP/3 feature available to all customers in the near future.

Once enabled, you can experiment with HTTP/3 in a number of ways:

In order to use the Chrome browser to connect to your website over HTTP/3, you first need to download and install the latest Canary build. Then all you need to do to enable HTTP/3 support is starting Chrome Canary with the “--enable-quic” and “--quic-version=h3-23” command-line arguments.

Once Chrome is started with the required arguments, you can just type your domain in the address bar, and see it loaded over HTTP/3 (you can use the Network tab in Chrome’s Developer Tools to check what protocol version was used). Note that due to how HTTP/3 is negotiated between the browser and the server, HTTP/3 might not be used for the first few connections to the domain, so you should try to reload the page a few times.

If this seems too complicated, don’t worry, as the HTTP/3 support in Chrome will become more stable as time goes on, enabling HTTP/3 will become easier.

This is what the Network tab in the Developer Tools shows when browsing this very blog over HTTP/3:

Note that due to the experimental nature of the HTTP/3 support in Chrome, the protocol is actually identified as “http2+quic/99” in Developer Tools, but don’t let that fool you, it is indeed HTTP/3.

The curl command-line tool also supports HTTP/3 as an experimental feature. You’ll need to download the latest version from git and follow the instructions on how to enable HTTP/3 support.

If you're running macOS, we've also made it easy to install an HTTP/3 equipped version of curl via Homebrew:

 % brew install --HEAD -s https://raw.githubusercontent.com/cloudflare/homebrew-cloudflare/master/curl.rb

In order to perform an HTTP/3 request all you need is to add the “--http3” command-line flag to a normal curl command:

 % ./curl -I https://blog.cloudflare.com/ --http3
HTTP/3 200
date: Tue, 17 Sep 2019 12:27:07 GMT
content-type: text/html; charset=utf-8
set-cookie: __cfduid=d3fc7b95edd40bc69c7d894d296564df31568723227; expires=Wed, 16-Sep-20 12:27:07 GMT; path=/; domain=.blog.cloudflare.com; HttpOnly; Secure
x-powered-by: Express
cache-control: public, max-age=60
vary: Accept-Encoding
cf-cache-status: HIT
age: 57
expires: Tue, 17 Sep 2019 12:28:07 GMT
alt-svc: h3-23=":443"; ma=86400
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
cf-ray: 517b128df871bfe3-MAN

Finally, we also provide an example HTTP/3 command-line client (as well as a command-line server) built on top of quiche, that you can use to experiment with HTTP/3.

To get it running, first clone quiche’s GitHub repository:

$ git clone --recursive https://github.com/cloudflare/quiche

Then build it. You need a working Rust and Cargo installation for this to work (we recommend using rustup to easily setup a working Rust development environment).

$ cargo build --examples

And finally you can execute an HTTP/3 request:

$ RUST_LOG=info target/debug/examples/http3-client https://blog.cloudflare.com/

In the coming months we’ll be working on improving and optimizing our QUIC and HTTP/3 implementation, and will eventually allow everyone to enable this new feature without having to go through a waiting list. We'll continue updating our implementation as standards evolve, which may result in breaking changes between draft versions of the standards.

Here are a few new features on our roadmap that we're particularly excited about:

One important feature that QUIC enables is seamless and transparent migration of connections between different networks (such as your home WiFi network and your carrier’s mobile network as you leave for work in the morning) without requiring a whole new connection to be created.

This feature will require some additional changes to our infrastructure, but it’s something we are excited to offer our customers in the future.

Just like TLS 1.3, QUIC supports a mode of operation that allows clients to start sending HTTP requests before the connection handshake has completed. We don’t yet support this feature in our QUIC deployment, but we’ll be working on making it available, just like we already do for our TLS 1.3 support.

We are excited to support HTTP/3 and allow our customers to experiment with it while efforts to standardize QUIC and HTTP/3 are still ongoing. We'll continue working alongside other organizations, including Google and Mozilla, to finalize the QUIC and HTTP/3 standards and encourage broad adoption.

Here's to a faster, more reliable, more secure web experience for all.

Magic Transit is built on the standards and networking primitives you are familiar with, but delivered from Cloudflare’s global edge network as a service. Traffic is ingested by the Cloudflare Network with anycast and BGP, announcing your company’s IP address space and extending your network presence globally. Today, our anycast edge network spans 193 cities in more than 90 countries around the world.

Once packets hit our network, traffic is inspected for attacks, filtered, steered, accelerated, and sent onward to the origin. Magic Transit will connect back to your origin infrastructure over Generic Routing Encapsulation (GRE) tunnels, private network interconnects (PNI), or other forms of peering.

Enterprises are often forced to pick between performance and security when deploying IP network services. Magic Transit is designed from the ground up to minimize these trade-offs: performance and security are better together. Magic Transit deploys IP security services across our entire global network. This means no more diverting traffic to small numbers of distant “scrubbing centers” or relying on on-premise hardware to mitigate attacks on your infrastructure.

We’ve been laying the groundwork for Magic Transit for as long as Cloudflare has been in existence, since 2010. Scaling and securing the IP network Cloudflare is built on has required tooling that would have been impossible or exorbitantly expensive to buy. So we built the tools ourselves! We grew up in the age of software-defined networking and network function virtualization, and the principles behind these modern concepts run through everything we do.

When we talk to our customers managing on-premise networks, we consistently hear a few things: building and managing their networks is expensive and painful, and those on-premise networks aren’t going away anytime soon.

Traditionally, CIOs trying to connect their IP networks to the Internet do this in two steps:

1. Source connectivity to the Internet from transit providers (ISPs).

2. Purchase, operate, and maintain network function specific hardware appliances. Think hardware load balancers, firewalls, DDoS mitigation equipment, WAN optimization, and more.

Each of these boxes costs time and money to maintain, not to mention the skilled, expensive people required to properly run them. Each additional link in the chain makes a network harder to manage.

This all sounded familiar to us. We had an aha! moment: we had the same issues managing our datacenter networks that power all of our products, and we had spent significant time and effort building solutions to those problems. Now, nine years later, we had a robust set of tools we could turn into products for our own customers.

Magic Transit aims to bring the traditional datacenter hardware model into the cloud, packaging transit with all the network “hardware” you might need to keep your network fast, reliable, and secure. Once deployed, Magic Transit allows seamless provisioning of virtualized network functions, including routing, DDoS mitigation, firewalling, load balancing, and traffic acceleration services.

Magic Transit delivers its connectivity, security, and performance benefits by serving as the “front door” to your IP network. This means it accepts IP packets destined for your network, processes them, and then outputs them to your origin infrastructure.

Connecting to the Internet via Cloudflare offers numerous benefits. Starting with the most basic, Cloudflare is one of the most extensively connected networks on the Internet. We work with carriers, Internet exchanges, and peering partners around the world to ensure that a bit placed on our network will reach its destination quickly and reliably, no matter the destination.

Let’s walk through how a customer might deploy Magic Transit. Customer Acme Corp. owns the IP prefix 203.0.113.0/24, which they use to address a rack of hardware they run in their own physical datacenter. Acme currently announces routes to the Internet from their customer-premise equipment (CPE, aka a router at the perimeter of their datacenter), telling the world 203.0.113.0/24 is reachable from their autonomous system number, AS64512. Acme has DDoS mitigation and firewall hardware appliances on-premise.

Acme wants to connect to the Cloudflare Network to improve the security and performance of their own network. Specifically, they’ve been the target of distributed denial of service attacks, and want to sleep soundly at night without relying on on-premise hardware. This is where Cloudflare comes in.

Deploying Magic Transit in front of their network is simple:

1. Cloudflare uses Border Gateway Protocol (BGP) to announce Acme’s 203.0.113.0/24 prefix from Cloudflare’s edge, with Acme’s permission.

2. Cloudflare begins ingesting packets destined for the Acme IP prefix.

3. Magic Transit applies DDoS mitigation and firewall rules to the network traffic. After it is ingested by the Cloudflare network, traffic that would benefit from HTTPS caching and WAF inspection can be “upgraded” to our Layer 7 HTTPS pipeline without incurring additional network hops.

4. Acme would like Cloudflare to use Generic Routing Encapsulation (GRE) to tunnel traffic back from the Cloudflare Network back to Acme’s datacenter. GRE tunnels are initiated from anycast endpoints back to Acme’s premise. Through the magic of anycast, the tunnels are constantly and simultaneously connected to hundreds of network locations, ensuring the tunnels are highly available and resilient to network failures that would bring down traditionally formed GRE tunnels.

5. Cloudflare egresses packets bound for Acme over these GRE tunnels.

Let’s dive deeper on how the DDoS mitigation included in Magic Transit works.

Customers deploying Cloudflare Magic Transit instantly get access to the same IP-layer DDoS protection system that has protected the Cloudflare Network for the past 9 years. This is the same mitigation system that stopped a 942Gbps attack dead in its tracks, in seconds. This is the same mitigation system that knew how to stop memcached amplification attacks days before a 1.3Tbps attack took down Github, which did not have Cloudflare watching its back. This is the same mitigation we trust every day to protect Cloudflare, and now it protects your network.

Cloudflare has historically protected Layer 7 HTTP and HTTPS applications from attacks at all layers of the OSI Layer model. The DDoS protection our customers have come to know and love relies on a blend of techniques, but can be broken into a few complementary defenses:

1. Anycast and a network presence in 193 cities around the world allows our network to get close to users and attackers, allowing us to soak up traffic close to the source without introducing significant latency.

2. 30+Tbps of network capacity allows us to soak up a lot of traffic close to the source. Cloudflare's network has more capacity to stop DDoS attacks than that of Akamai Prolexic, Imperva, Neustar, and Radware — combined.

3. Our HTTPS reverse proxy absorbs L3 (IP layer) and L4 (TCP layer) attacks by terminating connections and re-establishing them to the origin. This stops most spurious packet transmissions from ever getting close to a customer origin server.

4. Layer 7 mitigations and rate limiting stop floods at the HTTPS application layer.

Looking at the above description carefully, you might notice something: our reverse proxy servers protect our customers by terminating connections, but our network and servers still get slammed by the L3 and 4 attacks we stop on behalf of our customers. How do we protect our own infrastructure from these attacks?

Enter Gatebot!

Gatebot is a suite of software running on every one of our servers inside each of our datacenters in the 193 cities we operate, constantly analyzing and blocking attack traffic. Part of Gatebot’s beauty is its simple architecture; it sits silently, in wait, sampling packets as they pass from the network card into the kernel and onward into userspace. Gatebot does not have a learning or warm-up period. As soon as it detects an attack, it instructs the kernel of the machine it is running on to drop the packet, log its decision, and move on.

Historically, if you wanted to protect your network from a DDoS attack, you might have purchased a specialized piece of hardware to sit at the perimeter of your network. This hardware box (let’s call it “The DDoS Protection Box”) would have been fantastically expensive, pretty to look at (as pretty as a 2U hardware box could get), and required a ton of recurring effort and money to stay on its feet, keep its licence up to date, and keep its attack detection system accurate and trained.

For one thing, it would have to be carefully monitored to make sure it was stopping attacks but not stopping legitimate traffic. For another, if an attacker managed to generate enough traffic to saturate your datacenter’s transit links to the Internet, you were out of luck; no box sitting inside your datacenter can protect you from an attack generating enough traffic to congest the links running from the outside world to the datacenter itself.

Early on, Cloudflare considered buying The DDoS Protection Box(es) to protect our various network locations, but ruled them out quickly. Buying hardware would have incurred substantial cost and complexity. In addition, buying, racking, and managing specialized pieces of hardware makes a network hard to scale. There had to be a better way. We set out to solve this problem ourselves, starting from first principles and modern technology.

To make our modern approach to DDoS mitigation work, we had to invent a suite of tools and techniques to allow us to do ultra-high performance networking on a generic x86 server running Linux.

At the core of our network data plane is the eXpress Data Path (XDP) and the extended Berkeley Packet Filter (eBPF), a set of APIs that allow us to build ultra-high performance networking applications in the Linux kernel. My colleagues have written extensively about how we use XDP and eBPF to stop DDoS attacks:

• L4Drop: XDP DDoS Mitigations

• xdpcap: XDP Packet Capture

• XDP based DoS mitigation presentation

• XDP in practice: integrating XDP into our DDoS mitigation pipeline (PDF)

• Cloudflare architecture and how BPF eats the world

At the end of the day, we ended up with a DDoS mitigation system that:

• Is delivered by our entire network, spread across 193 cities around the world. To put this another way, our network doesn’t have the concept of “scrubbing centers” — every single one of our network locations is always mitigating attacks, all the time. This means faster attack mitigation and minimal latency impact for your users.

• Has exceptionally fast times to mitigate, with most attacks mitigated in 10s or less.

• Was built in-house, giving us deep visibility into its behavior and the ability to rapidly develop new mitigations as we see new attack types.

• Is deployed as a service, and is horizontally scalable. Adding x86 hardware running our DDoS mitigation software stack to a datacenter (or adding another network location) instantly brings more DDoS mitigation capacity online.

Gatebot is designed to protect Cloudflare infrastructure from attack. And today, as part of Magic Transit, customers operating their own IP networks and infrastructure can rely on Gatebot to protect their own network.

We’ve covered how Cloudflare Magic Transit connects your network to the Internet, and how it protects you from DDoS attack. If you were running your network the old-fashioned way, this is where you’d stop to buy firewall hardware, and maybe another box to do load balancing.

With Magic Transit, you don’t need those boxes. We have a long track record of delivering common network functions (firewalls, load balancers, etc.) as services. Up until this point, customers deploying our services have relied on DNS to bring traffic to our edge, after which our Layer 3 (IP), Layer 4 (TCP & UDP), and Layer 7 (HTTP, HTTPS, and DNS) stacks take over and deliver performance and security to our customers.

Magic Transit is designed to handle your entire network, but does not enforce a one-size-fits-all approach to what services get applied to which portion of your traffic. To revisit Acme, our example customer from above, they have brought 203.0.113.0/24 to the Cloudflare Network. This represents 256 IPv4 addresses, some of which (eg 203.0.113.8/30) might front load balancers and HTTP servers, others mail servers, and others still custom UDP-based applications.

Each of these sub-ranges may have different security and traffic management requirements. Magic Transit allows you to configure specific IP addresses with their own suite of services, or apply the same configuration to large portions (or all) of your block.

Taking the above example, Acme may wish that the 203.0.113.8/30 block containing HTTP services fronted by a traditional hardware load balancer instead deploy the Cloudflare Load Balancer, and also wants HTTP traffic analyzed with Cloudflare’s WAF and content cached by our CDN. With Magic Transit, deploying these network functions is straight-forward — a few clicks in our dashboard or API calls will have your traffic handled at a higher layer of network abstraction, with all the attendant goodies applying application level load balancing, firewall, and caching logic bring.

This is just one example of a deployment customers might pursue. We’ve worked with several who just want pure IP passthrough, with DDoS mitigation applied to specific IP addresses. Want that? We got you!

When you connect your network to Cloudflare Magic Transit, you get access to the entire Cloudflare network. This means all of our network locations become your network locations. Our network capacity becomes your network capacity, at your disposal to power your experiences, deliver your content, and mitigate attacks on your infrastructure.

How expansive is the Cloudflare Network? We’re in 193 cities worldwide, with more than 30Tbps of network capacity spread across them. Cloudflare operates within 100 milliseconds of 98% of the Internet-connected population in the developed world, and 93% of the Internet-connected population globally (for context, the blink of an eye is 300-400 milliseconds).

Areas of the globe within 100 milliseconds of a Cloudflare datacenter.

Just as we built our own products in house, we also built our network in house. Every product runs in every datacenter, meaning our entire network delivers all of our services. This might not have been the case if we had assembled our product portfolio piecemeal through acquisition, or not had completeness of vision when we set out to build our current suite of services.

The end result for customers of Magic Transit: a network presence around the globe as soon you come on board. Full access to a diverse set of services worldwide. All delivered with latency and performance in mind.

We'll be sharing a lot more technical detail on how we deliver Magic Transit in the coming weeks and months.

Traditional network services don’t come cheap; they require high capital outlays up front, investment in staff to operate, and ongoing maintenance contracts to stay functional. Just as our product aims to be disruptive technically, we want to disrupt traditional network cost-structures as well.

Magic Transit is delivered and billed as a service. You pay for what you use, and can add services at any time. Your team will thank you for its ease of management; your management will thank you for its ease of accounting. That sounds pretty good to us!

We’ve worked hard over the past nine years to get our network, management tools, and network functions as a service into the state they’re in today. We’re excited to get the tools we use every day in customers’ hands.

So that brings us to naming. When we showed this to customers the most common word they used was ‘whoa.’ When we pressed what they meant by that they almost all said: ‘It’s so much better than any solution we’ve seen before. It’s, like, magic!’ So it seems only natural, if a bit cheesy, that we call this product what it is: Magic Transit.

We think this is all pretty magical, and think you will too. Contact our Enterprise Sales Team today.

Welcome to Speed Week! Each day this week, we’re going to talk about something Cloudflare is doing to make the Internet meaningfully faster for everyone.

Cloudflare has built a massive network of data centers in 180 cities in 75 countries. One way to think of Cloudflare is a global system to transport bits securely, quickly, and reliably from any point A to any other point B on the planet.

To make that a reality, we built Argo. Argo uses real-time global network information to route around brownouts, cable cuts, packet loss, and other problems on the Internet. Argo makes the network that Cloudflare relies on—the Internet—faster, more reliable, and more secure on every hop around the world.

We launched Argo two years ago, and it now carries over 22% of Cloudflare’s traffic. On an average day, Argo cuts the amount of time Internet users spend waiting for content by 112 years!

As Cloudflare and our traffic volumes have grown, it now makes sense to build our own private backbone to add further security, reliability, and speed to key connections between Cloudflare locations.

Today, we’re introducing the Cloudflare Global Private Backbone. It’s been in operation for a while now and links Cloudflare locations with private fiber connections.

This private backbone benefits all Cloudflare customers, and it shines in combination with Argo. Argo can select the best available link across the Internet on a per data center-basis, and takes full advantage of the Cloudflare Global Private Backbone automatically.

Let’s open the hood on Argo and explain how our backbone network further improves performance for our customers.

Argo is like Waze for the Internet. Every day, Cloudflare carries hundreds of billions of requests across our network and the Internet. Because our network, our customers, and their end-users are well distributed globally, all of these requests flowing across our infrastructure paint a great picture of how different parts of the Internet are performing at any given time.

Just like Waze examines real data from real drivers to give you accurate, uncongested (and sometimes unorthodox) routes across town, Argo Smart Routing uses the timing data Cloudflare collects from each request to pick faster, more efficient routes across the Internet.

In practical terms, Cloudflare’s network is expansive in its reach. Some of the Internet links in a given region may be congested and cause poor performance (a literal traffic jam). By understanding this is happening and using alternative network locations and providers, Argo can put traffic on a less direct, but faster, route from its origin to its destination.

These benefits are not theoretical: enabling Argo Smart Routing shaves an average of 33% off HTTP time to first byte (TTFB).

One other thing we’re proud of: we’ve stayed super focused on making it easy to use. One click in the dashboard enables better, smarter routing, bringing the full weight of Cloudflare’s network, data, and engineering expertise to bear on making your traffic faster. Advanced analytics allow you to understand exactly how Argo is performing for you around the world.

You can read a lot more about how Argo works in our original launch blog post.

So far, we’ve been talking about Argo at a functional level: you turn it on and it makes requests that traverse the Internet to your origin faster. How does it actually work? Argo is dependent on a few things to make its magic happen: Cloudflare’s network, up-to-the-second performance data on how traffic is moving on the Internet, and machine learning routing algorithms.

Cloudflare maintains a network of data centers around the world, and our network continues to grow significantly. Today, we have more than 180 data centers in 75 countries. That’s an additional 69 data centers since we launched Argo in May 2017.

In addition to adding new locations, Cloudflare is constantly working with network partners to add connectivity options to our network locations. A single Cloudflare data center may be peered with a dozen networks, connected to multiple Internet eXchanges (IXs), connected to multiple transit providers (e.g. Telia, GTT, etc), and now, connected to our own physical backbone. A given destination may be reachable over multiple different links from the same location; each of these links will have different performance and reliability characteristics.

This increased network footprint is important in making Argo faster. Additional network locations and providers mean Argo has more options at its disposal to route around network disruptions and congestion. Every time we add a new network location, we exponentially grow the number of routing options available to any given request.

Argo requires the huge global network we’ve built to do its thing. But it wouldn’t do much of anything if it didn’t have the smarts to actually take advantage of all our data centers and cables between them to move traffic faster.

Argo combines multiple machine learning techniques to build routes, test them, and disqualify routes that are not performing as we expect.

The generation of routes is performed on data using “offline” optimization techniques: Argo’s route construction algorithms take an input data set (timing data) and fixed optimization target (“minimize TTFB”), outputting routes that it believes satisfy this constraint.

Route disqualification is performed by a separate pipeline that has no knowledge of the route construction algorithms. These two systems are intentionally designed to be adversarial, allowing Argo to be both aggressive in finding better routes across the Internet but also adaptive to rapidly changing network conditions.

One specific example of Argo’s smarts is its ability to distinguish between multiple potential connectivity options as it leaves a given data center. We call this “transit selection”.

As we discussed above, some of our data centers may have a dozen different, viable options for reaching a given destination IP address. It’s as if you subscribed to every available ISP at your house, and you could choose to use any one of them for each website you tried to access. Transit selection enables Cloudflare to pick the fastest available path in real-time at every hop to reach the destination.

With transit selection, Argo is able to specify both:

1. Network location waypoints on the way to the origin.

2. The specific transit provider or link at each waypoint in the journey of the packet all the way from the source to the destination.

To analogize this to Waze, Argo giving directions without transit selection is like telling someone to drive to a waypoint (go to New York from San Francisco, passing through Salt Lake City), without specifying the roads to actually take to Salt Lake City or New York. With transit selection, we’re able to give full turn-by-turn directions — take I-80 out of San Francisco, take a left here, enter the Salt Lake City area using SR-201 (because I-80 is congested around SLC), etc. This allows us to route around issues on the Internet with much greater precision.

Transit selection requires logic in our inter-data center data plane (the components that actually move data across our network) to allow for differentiation between different providers and links available in each location. Some interesting network automation and advertisement techniques allow us to be much more discerning about which link actually gets picked to move traffic.

Without modifications to the Argo data plane, those options would be abstracted away by our edge routers, with the choice of transit left to BGP. We plan to talk more publicly about the routing techniques used in the future.

We are able to directly measure the impact transit selection has on Argo customer traffic. Looking at global average improvement, transit selection gets customers an additional 16% TTFB latency benefit over taking standard BGP-derived routes. That’s huge!

One thing we think about: Argo can itself change network conditions when moving traffic from one location or provider to another by inducing demand (adding additional data volume because of improved performance) and changing traffic profiles. With great power comes great intricacy.

Given our diversity of transit and connectivity options in each of our data centers, and the smarts that allow us to pick between them, why did we go through the time and trouble of building a backbone for ourselves? The short answer: operating our own private backbone allows us much more control over end-to-end performance and capacity management.

When we buy transit or use a partner for connectivity, we’re relying on that provider to manage the link’s health and ensure that it stays uncongested and available. Some networks are better than others, and conditions change all the time.

As an example, here’s a measurement of jitter (variance in round trip time) between two of our data centers, Chicago and Newark, over a transit provider’s network:

Average jitter over the pictured 6 hours is 4ms, with average round trip latency of 27ms. Some amount of latency is something we just need to learn to live with; the speed of light is a tough physical constant to do battle with, and network protocols are built to function over links with high or low latency.

Jitter, on the other hand, is “bad” because it is unpredictable and network protocols and applications built on them often degrade quickly when jitter rises. Jitter on a link is usually caused by more buffering, queuing, and general competition for resources in the routing hardware on either side of a connection. As an illustration, having a VoIP conversation over a network with high latency is annoying but manageable. Each party on a call will notice “lag”, but voice quality will not suffer. Jitter causes the conversation to garble, with packets arriving on top of each other and unpredictable glitches making the conversation unintelligible.

Here’s the same jitter chart between Chicago and Newark, except this time, transiting the Cloudflare Global Private Backbone:

Much better! Here we see a jitter measurement of 536μs (microseconds), almost eight times better than the measurement over a transit provider between the same two sites.

The combination of fiber we control end-to-end and Argo Smart Routing allows us to unlock the full potential of Cloudflare’s backbone network. Argo’s routing system knows exactly how much capacity the backbone has available, and can manage how much additional data it tries to push through it. By controlling both ends of the pipe, and the pipe itself, we can guarantee certain performance characteristics and build those expectations into our routing models. The same principles do not apply to transit providers and networks we don’t control.

Our private backbone is another tool available to us to improve performance on the Internet. Combining Argo’s cutting-edge machine learning and direct fiber connectivity between points on our large network allows us to route customer traffic with predictable, excellent performance.

We’re excited to see the backbone and its impact continue to expand.

Speaking personally as a product manager, Argo is really fun to work on. We make customers happier by making their websites, APIs, and networks faster. Enabling Argo allows customers to do that with one click, and see immediate benefit. Under the covers, huge investments in physical and virtual infrastructure begin working to accelerate traffic as it flows from its source to destination.  

From an engineering perspective, our weekly goals and objectives are directly measurable — did we make our customers faster by doing additional engineering work? When we ship a new optimization to Argo and immediately see our charts move up and to the right, we know we’ve done our job.

Building our physical private backbone is the latest thing we’ve done in our need for speed.

Welcome to Speed Week!

Activate Argo now, or contact sales to learn more!

A year ago, we launched Cloudflare Mobile SDK with a set of free features focused on measuring mobile app networking performance. Apps are dependent on network connectivity to deliver their app’s user experiences, but developers have limited visibility into how network connectivity is impacting app performance. Integrating the Mobile SDK allows developers to measure and improve the speed of their app’s network interactions.

Mobile applications interact with the Internet to do everything — to fetch the weather, your email, to step through a checkout flow. Everything that makes a smartphone magical is powered by a service on the Internet. How quickly those network interactions happen is dictated by two things: how large the payloads are for the given request/response, and what the available link bandwidth is.

Payload size is mostly application specific: a shopping app is going to request product images and similar medium sized assets, while a stock quotes app could be expected to have smaller payloads in the API responses powering it.

Available link bandwidth is usually dictated by your network provider. Everyone familiar with the feeling of trying to check out in an e-commerce app and being stymied by poor cell connectivity. But network quality is not the only thing that impacts available bandwidth; the transport protocol (at Layer 4, in OSI-model-speak) in use also has a huge impact on how quickly your phone can pull content off the Internet.

TCP is the dominant transport protocol for most applications you know and love. It’s over 40 years old, and impressive in both its simplicity and longevity (they are likely related). TCP relies on congestion control algorithms to understand how quickly to send traffic over a connection without congesting the link (filling the pipe to the point things start getting backed up).

Congestion is something to be avoided. TCP guarantees reliable delivery, and cleaning up from a congestion event often involves additional round trips and retransmits. TCP implementations are often conservative in two important dimensions: how much data they choose to send on connection establishment (called the initcwnd, or initial congestion window), and what to do when the sender senses packet loss (congestion avoidance).

Source: https://commons.wikimedia.org/wiki/File:TCP_Slow-Start_and_Congestion_Avoidance.svg

An example of the data rate on a connection over time. Congestion avoidance is illustrated in pink.

How TCP opens connections and how it responds to packet loss are critical factors in determining how much data actually gets to flow over the connection. Tuning TCP connection parameters allows more data to flow over the link without actually touching the actual physical layer (i.e. boosting your cell signal).

Unfortunately, TCP parameters governing a connection’s data rate are hidden in the kernel, out of reach of user space and the optimizing, enterprising app developer. Cloudflare Mobile SDK aims to solve this problem by shipping a replacement transport protocol implemented on top of UDP, which the SDK can speak with the Cloudflare edge.

There are three advantages to replacing TCP with a custom UDP transport protocol:

1. Performance tuning, bug fixes, and incremental updates to the protocol itself can be done without any downtime or coordination with the kernel. This is not the case with TCP.

2. Middle-boxes' (eg. corporate proxies, etc) assumptions on how TCP works have made improving TCP very difficult. UDP based protocols don't suffer from the same middle-box ossification.

3. Having tight control and coordination between the send-side Cloudflare edge and receive-side Mobile SDK makes optimizing individual connections possible, even over very dissimilar mobile netwoks.

All of these factors lead directly to reduced latency, increased throughput, and improved user experiences.

Once an app is integrated with the SDK, enabling Acceleration is straightforward. Most standard HTTP networking libraries are supported out of the box, and require no additional integration work beyond initializing the SDK with your API key.

Customers accelerating their traffic with Cloudflare Mobile SDK see significant reductions in latency, increases in throughput, and reductions in TCP related timeouts.

As an example, a transportation company enabled acceleration in their iOS app. Their users immediately saw a 7% decrease in network response time and a 13.8% drop in network timeouts. This directly translates to an increase in conversions: purchases per user increased 3% with Acceleration enabled.

We’re excited to bring Acceleration to a broader audience. Get in touch with us for early access. Mobile SDK supports both iOS and Android.

In addition to developing features to improve app performance, we’re working hard on features to better authenticate mobile devices with the APIs that power them. Why is this important? Non-humans (bots) are increasingly interacting with the APIs that power apps to scrape data, stuff credentials, and otherwise act in ways humans would not.

The Mobile SDK will soon include features to help API owners understand whether or not the user purporting to be using an app actually is a real mobile user. We’ll have a lot more detail on this soon; if you’re interested in hearing more sooner, please get in touch!

We’re excited to announce the full integration of Cloudflare Spectrum with Load Balancing. Combining Spectrum with Load Balancing enables traffic management of TCP connections utilising the same battle tested Load Balancer our customers already use for billions of HTTP requests every day.

Customers can configure load balancers with TCP health checks, failover, and steering policies to dictate where traffic should flow. This is live in the Cloudflare dashboard and API — give it a shot!

You can now configure Cloudflare’s Load Balancer health checks to probe any TCP port for an accepted connection. This is in addition to the existing HTTP and HTTPS options.

Health checks are an optional feature within Cloudflare’s Load Balancing product. Without health checks, the Cloudflare Load Balancer will distribute traffic to all origins in the first pool. While this is in itself useful, adding a health check to a Load Balancer provides additional functionality.

With a health check configured for a pool in a Load Balancer, Cloudflare will automatically distribute traffic within a pool to any origins that are marked up by the health check. Unhealthy origins will be dropped automatically. This allows for intelligent failover both within a pool and amongst pools. Health checks can be configured from multiple regions (and even all of Cloudflare’s PoPs as an Enterprise customer) to detect local and global connectivity issues from your origins.

In this example, we will configure a TCP health check for an application running on port 2408 with a refresh rate of every 30 seconds via either the dashboard or our API.

Configuring a TCP health check

# POST accounts/:account_identifier/load_balancers/monitors

{
  "description": "Spectrum Health Check",
  "type": "tcp",
  "port": 2048,
  "interval": 30,
  "retries": 2,
  "timeout": 5,
  "method": "connection_established",
}

Origin weights are beneficial should you have origins that are not of equal capacity or if you want to unequally split traffic for any other reason.

Weights configured within a load balancer pool will be honored with transport load balancing through Spectrum. If configured, Cloudflare will distribute traffic amongst the available origins within a pool according to the relative weights assigned to each origin.

For further information on weighted steering, see the knowledge base article.

All steering modes are available for transport load balancing through Spectrum: You can choose standard failover, dynamic steering, or geo steering:

• FailoverIn this mode, the Cloudflare Load Balancer will fail over amongst pools listed in a given load balancer configuration as they are marked down by health checks. If all pools are marked down, Cloudflare will send traffic to the fallback pool. The fallback pool is the last pool in the list in the dashboard or specifically nominated via a parameter in the API. If no health checks are configured, Cloudflare will send to the primary pool exclusively.

• Dynamic SteeringDynamic steering was recently introduced by Cloudflare as a way of directing traffic to the fastest pool for a given user. In this mode, the Cloudflare load balancer will select the fastest pool for the given Cloudflare Region or PoP (ENT only) through health check data. If there is no health check data for a given colo or region, the load balancer will select a pool in failover order. It is important to note that with TCP health checks, latency calculated may not be representative of true latency to origin if you are terminating TCP at a cloud provider edge location.

• Geo SteeringGeo Steering allows you to specify pools for a given Region or PoP (ENT only). In this configuration, Cloudflare will direct traffic from specified Cloudflare locations to configured pools. You may configure multiple pools, and the load balancer will use them in failover order. If this steering mode is selected and there is no configuration for a region or pool, the load balancer will use the default failover order.

Once your load balancer is configured, it’s available for use as an origin with your Spectrum application:

Configuring a Spectrum application with Load Balancing

Combining Spectrum’s ability to proxy TCP applications, our Load Balancer’s full feature set, and Cloudflare’s global network allows our customers to build performant, reliable, and secure network applications with minimal effort.

We’ve seen customers combine Spectrum and Load Balancing to build scalable gaming platforms, make their live streaming infrastructure more robust, push the envelope with interesting cryptocurrency use cases, and lots more. What will you build?

Spectrum with Load Balancing is available to all current Spectrum and Load Balancing users. Want access to Spectrum? Get in touch with our team. Spectrum is available for applications on the Enterprise plan.

Three things were required to make the Bandwidth Alliance a reality:

1. An ecosystem of like-minded companies who want to provide reduced data transfer fees to their customers.

2. A large global and well-connected network (Cloudflare has 150+ points of presence around the world and multiple peered and paid links at each location). Our network is connected to thousands of partners through transit providers, Internet exchanges, peering interconnects, and private network interconnects. Having a large network footprint allows us to meet our partners where their infrastructure is and exchange traffic with them over low-cost or free connections, instead of expensive paid transit.

3. Argo, our sophisticated traffic routing engine. Argo allows us to make decisions on how to carry traffic across our network in ways that optimize for a number of factors: latency, throughput, jitter, or in the case of the Bandwidth Alliance, cost to our partners to exchange traffic. This routing engine is the technical underpinning of the Bandwidth Alliance.

Typically, as traffic moves across the Internet, packets are exchanged between multiple networks as they move from origin to destination. The specific path taken is determined by routers along the way using Border Gateway Protocol (BGP). BGP allows routers to learn which networks have connectivity to which other networks and to pick optimal paths across those. Notably, “optimal path” in this case refers to the path with the fewest number of network hops to the destination, not the best performance or lowest monetary cost for any party involved.

Cloudflare is directly connected and peered with thousands of different networks, including all of our Bandwidth Alliance partners. Moving traffic over one of these links is very fast and very cheap, especially compared to moving the same traffic over public transit. However, because BGP and standard path discovery techniques are not built to directly understand either real world performance or monetary cost, moving traffic from Cloudflare to our Bandwidth Alliance partners and back may not actually use these fast, cheap links between us and them.

This is where Argo comes in. We can plug different routing functions into Argo, optimizing for connection parameters other than the default heuristics BGP users.

Argo consists of two logical components:

1. A control plane, that mines our network performance data and understands our network topology in order to generate good routes across the globe.

2. A data plane, that takes that routing information and makes sure data flowing across our network actually takes those paths.

Argo’s control plane, in its standard configuration, optimizes connections for minimum latency. It does this by examining subnet-to-subnet timing data collected from requests that pass between those subnets, determining which paths are fastest. The list of subnets examined is determined by looking at global routing tables; each entry in the table is a logically and physically grouped set of hosts. Argo then determines where on our network traffic destined for a given subnet should exit our network.

Full Argo is a paid service that Cloudflare customers can subscribe to. For the customers hosted on providers that are part of the Bandwidth Alliance, Cloudflare automatically enables a limited, routing-only version of Argo at no additional charge. While this does not get all the benefits of Argo's full performance feature set, it does ensure that traffic is routed to the Cloudflare location nearest where our customer is hosted so traffic can pass across a Private Network Interface (PNI) and therefore enjoy substantially lowered bandwidth costs. Customers who choose to subscribe to full-featured Argo get additional performance benefits from optimized route selection, data compression, tiered caching, and protocol optimization.

The end result of all this: low latency, highly available transit across our network from your host to your user, and all at little to no cost to Cloudflare or your cloud provider — and, in turn, a substantially reduced cost to you. We expect that as the Bandwidth Alliance comes online, Cloudflare customers could save more than $50 million per year in cloud bandwidth fees.

Our commitment through the Bandwidth Alliance is to pass those cost savings on to you, our mutual customers. Today, we’re excited to put the benefits in your hands: smaller bills and better performance.

Subscribe to the blog for daily updates on all our Birthday Week announcements.

If your app loads critical resources over the network, it's relying on your user's mobile network connection to deliver an engaging experience. Network errors occur in 3 to 12% of app sessions depending on infrastructure reliability and user geography.

How much engagement are you losing in your app to network errors? Chances are, you don't know.

We didn't either, until we built a free tool that helps Android and iOS developers visualize and understand their mobile app's network utilization.

Our SDK helps you identify slowdowns caused by balky or too frequent network calls, so you can focus your development effort on optimizing the lowest-hanging fruit.

Modern app developers already heavily instrument their apps to identify UX impacting events: they measure and collect launch time, session length, crash rates, conversion events, and lots more, using a multitude of different metrics packages and services.

Web developers look at similar data. They also pay tons of attention to their resource waterfall, mapping their critical rendering path, and understanding which resource loads are synchronous, which are not, and which block rendering. JavaScript even exposes an API to collect waterfalls in the browser programmatically.

It's time to bring the same visibility to your app's network waterfall.

Using Cloudflare Mobile SDK, you can identify top N requests, slow requests, and requests most likely to fail. You can also inspect all the third party calls your app is making through included libraries. Always suspected that ad network you're calling out to is crippling your app's performance? Now you know.

Our aim is to make this data as useful as possible. We know you're already looking at engagement data in tools like Mixpanel, Amplitude, and Heap. To this end, we're making it as easy as possible to correlate network experience data with the event and engagement data you're already tracking.

All of this is free as in beer, with no cap on active users or metrics tracked. Collecting metrics does not require you use Cloudflare as part of your infrastructure stack, and adds minimal weight to your app APK or IPA. Integration is as simple as including our library, adding one line to your AppDelegate or Gradle config, and building.

Privacy is, and was, top of mind as we built this. Cloudflare Mobile SDK does not collect any persistent identifiers (UDID, IDFA, etc.), and we will never sell the data we collect to any third party.

Interested in giving this a shot?Register for an SDK key here: mobilesdk.cloudflare.com/v2s/signupAnd get docs here: developers.cloudflare.com/mobile-sdk/overview

Today, to fix all of this, Cloudflare is launching Argo, a “virtual backbone” for the modern Internet. Argo analyzes and optimizes routing decisions across the global Internet in real-time. Think Waze, the automobile route optimization app, but for Internet traffic.

Just as Waze can tell you which route to take when driving by monitoring which roads are congested or blocked, Argo can route connections across the Internet efficiently by avoiding packet loss, congestion, and outages.

Cloudflare’s Argo is able to deliver content across our network with dramatically reduced latency, increased reliability, heightened encryption, and reduced cost vs. an equivalent path across the open Internet. The results are impressive: an average 35% decrease in latency, a 27% decrease in connection errors, and a 60% decrease in cache misses. Websites, APIs, and applications using Argo have seen bandwidth bills fall by more than half and speed improvements end users can feel.

Argo is a central nervous system for the Internet, processing information from every request we see to determine which routes are fast, which are slow, and what the optimum path from visitor to content is at that given moment. Through Cloudflare’s 115 PoPs and 6 million domains, we see every ISP and every user of the Internet pass through our network. The intelligence from this gives us a billion eyes feeding information about brownouts, faults, and packet loss globally.

Today, Argo includes two core features: Smart Routing and Tiered Cache. All customers can enable Argo today in the Traffic app in the dashboard. Argo is priced at \$5/domain monthly, plus \$0.10 per GB of transfer from Cloudflare to your visitors.

Networks on the Internet rely on legacy technologies like BGP to propagate and calculate routes from network to network, ultimately getting you from laptop-on-couch to video-on-YouTube. BGP has been around for decades, and was not designed for a world with malicious or incompetent actors lurking at every network hop.

In one comical example from 2008, a Pakistani ISP turned a botched censorship order into a global YouTube outage, bringing the fragility of core Internet routing algorithms into the public eye. In the same situation, Argo Smart Routing would detect which transit providers had valid routes to YouTube and which did not, keeping end user experience fast, reliable, and secure.

Metcalfe’s Law states that the value of a network is defined by the square of the number of nodes that make up the network. The existing Internet is incredibly valuable because of the number and diversity of nodes connected to the network. Unfortunately, this makes it difficult to pick up and start over; no Internet started from scratch, with sounder routing and traffic management, would come close to delivering the value provided by the current incarnation without a similar network footprint.

Because of our physical and virtual presence around the world, Cloudflare is uniquely positioned to rebuild the core of the Internet. Every customer we bring on increases the size of our network and the value of that network to each of our customers. Argo is Metcalfe’s Law brought to life.

Argo Smart Routing uses latency and packet loss data collected from each request that traverses our network to pick optimal paths across the Internet. Using this latency data, we’re able to determine which of our transit providers are performing best between any two points on the planet. Cloudflare now sees about 10% of all HTTP/HTTPS requests on the Internet. With Argo, each of those requests is providing the insight necessary to speed up all of its peers.

CC BY 2.0 image by Steve Jurvetson

Enabling Argo (and Smart Routing with it) results in breathtaking reductions in latency. As an example, OKCupid enabled Argo and immediately saw a 36% decrease in request latency, as measured by TTFB (Time To First Byte). Without Argo, requests back to the origin from a Cloudflare PoP traverse the public Internet, subject to vagaries of routers, cables, and computers they will touch on their journey. With Argo, requests back to the origin are tunneled over our secure overlay network, on a path to the origin we've learned the performance of from all the requests that have traversed before it.

Transit over the public Internet is like driving with paper maps; it usually works, but using a modern navigation system that takes current traffic conditions into account will almost always be faster.

Routing over intelligently determined paths also results in significant reliability gains. Argo picks the fastest, most reliable route to the origin, which means routing around flapping links and routers that refuse to do their job. In a real-world illustration of these reliability gains, OKCupid saw a 42% drop in the number of connection timeouts on their site with Argo enabled.

It’s not just OKCupid that’s happy with Argo. 50,000 customers, large and small, have been beta testing Argo over the last 12 months. On average, these Argo Smart Routing beta customers saw a 35% decrease in latency and a 27% decrease in connection timeouts.

Argo Tiered Cache uses the size of our network to reduce requests to customer origins by dramatically increasing cache hit ratios. By having 115 PoPs around the world, Cloudflare caches content very close to end users, but if a piece of content is not in cache, the Cloudflare edge PoP must contact the origin server to receive the cacheable content. This can be slow and places load on an origin server compared to serving directly from cache.

Argo Tiered Cache lowers origin load, increases cache hit ratios, and improves end user experience by first asking other Cloudflare PoPs if they have the requested content when a cache miss occurs. This results in improved performance for visitors, because distances and links traversed between Cloudflare PoPs are generally shorter and faster than the links between PoPs and origins. It also reduces load on origins, making web properties more economical to operate. Customers enabling Argo can expect to see a 60% reduction in their cache miss rate as compared to Cloudflare’s traditional CDN service.

Argo Tiered Cache also concentrates connections to origin servers so they come from a small number of PoPs rather than the full set of 115 PoPs. This results in fewer open connections using server resources. In our testing, we've found many customers save more on their cloud hosting bills than Argo costs, because of reduced bandwidth usage and fewer requests to the origin. This makes the service a “no brainer” to enable.

In addition to performance and reliability gains, Argo also delivers a more secure online experience. All traffic between Cloudflare data centers is protected by mutually authenticated TLS, ensuring any traffic traversing the Argo backbone is protected from interception, tampering, and eavesdropping.

With Argo, we’ve rebuilt things at the very core of the Internet, the algorithms that figure out where traffic should flow and how. We’ve done all this without any disruption to how the Internet works or how applications behave.

Cloudflare has built a suite of products to address lots of pains on the Internet. Argo is our newest offering.

Go ahead and enable it — you’ll find it in the Traffic tab in your dashboard.

PS. Interested in working on Argo? Drop us a line!