Cloudflare launched our free tier at the same time our company launched — fourteen years ago, on September 27, 2010. Of course, a bit has changed since then — there are now millions of Internet properties behind Cloudflare. As we’ve grown in size and amassed millions of free customers, one of the questions we often get asked is: how can Cloudflare afford to do this at such scale?

Cloudflare always has, and always will, offer a generous free version for public-facing applications (Application Services), internal private networks and people (Cloudflare One), and developer tools (Developer Platform). Counterintuitively: our free service actually helps us keep our costs lower. Not only is it mission-aligned, our free tier is business-aligned. We want to make abundantly clear: our free plan is here to stay, and we reaffirmed that commitment this week with 15 releases across our product portfolio that make the Free plan even better.

To understand the economics of Free, you need to understand our Cost of Goods Sold (COGS). Cloudflare hasn’t outsourced its network — we built it ourselves, and it spans more than 330 cities. We design and ship our own hardware across the world, we interconnect with more than 12,500 networks, and we manage over 300 Tbps of network capacity. We even have a dedicated backbone that spans the globe.

There are three major costs of running our network, which together comprise about 80% of our COGS. First and largest is bandwidth: the traffic that traverses our network. Then there is hardware: the servers that process traffic. And third are colocation costs: the power and space at the data centers where we house our servers. There are other parts of COGS, too, like our SRE team that keeps the network running, and our payment processor fees, without which we couldn’t collect revenue.

To get traffic across the Internet for a network of our scale, we need a lot of bandwidth. Typically, a network like ours would pay third-party transit networks and Internet Service Providers (ISPs) to transmit data anywhere on the Internet. But there are thousands of ISPs that we don’t have to pay at all, and hundreds that also offer us space in their data center at no cost. How did we manage that? The surprising answer: Free.

Imagine you run an ISP serving your local community. Your job is to connect your customers to the Internet. You notice that your customers are often visiting sites behind Cloudflare, which sits in front of roughly 20% of the web. You need to deliver those webpages and facilitate connections to the applications behind Cloudflare, but right now you have to pay a transit provider to reach them. Instead, you could choose to peer directly with Cloudflare and exchange traffic at no cost.

Cloudflare is one of the most peered networks in the world. We freely exchange traffic with thousands of ISPs, who in turn benefit because they can cut out a third-party transit provider to reach the millions of sites and applications behind Cloudflare.

Continuing with this hypothetical, if as an ISP, your customers pay for Internet connectivity based on data usage (a common model outside of Western Europe and the US), your revenue scales with data consumption. One simple way to increase data consumption? Make the Internet faster! Hosting Cloudflare’s servers in your facility, as close to your users as possible, reduces latency for millions of websites and apps. So it’s in your best interest to host Cloudflare’s servers in your data centers, too.

We have hundreds of ISP partnerships that look just like that. The value ISPs get from Cloudflare stems from the breadth of the web that sits behind Cloudflare, a number driven by our Free customers. This arrangement is a big part of why we have a free service, and is part of what enables us to continue to offer one. PS: If you really are an operator for a local ISP and don’t partner with us yet, please connect with us through our peering portal!

These days, we are at such a scale that the traffic our customers generate requires much more capacity than can fit within our ISP partners. To reliably serve our enterprise customers, we operate in multiple facilities in every major Internet hub city. And yet, the traffic patterns of our enterprise customers are typically very predictable. They usually follow a diurnal cycle, with peaks and troughs throughout a day. Enterprise customer traffic is prioritized and served as close to end users as possible, regardless of the time of day. But our Free customers use off-cycle headroom. That’s why we’re able to continue to offer unmetered bandwidth on the Free plan: we serve the traffic from across our network, wherever there is spare room. It might not have quite the same performance as our enterprise traffic, but it’s still reliable and fast.

There do have to be some rules for this to continue to work, however. Free traffic needs to remain a manageable proportion of our total traffic. To ensure that remains true, and that we can continue to offer unmetered traffic to Free customers at no cost, we have to be opinionated about what kind of traffic we serve for free. Our terms of service specify that large assets (like videos) are not supported on our Free plan. So we require that customers pushing large files and videos move onto one of our paid services, like Images and Stream.

The benefits of our Free plan extend well beyond direct economics.

Our Free plan gives Cloudflare access to unique threat intelligence. A wide surface area exposes our network to diverse traffic and attacks that we wouldn’t otherwise see, often allowing us to identify potential security and reliability issues at the earliest stage. Like an immune system, we learn from these attacks and adapt to improve our products for all customers. This is a special competitive advantage. Visibility into attacks allows us to build products that no one else could.

Our Free customers help us do quality assurance (QA) quickly. Free customers are often the first to try new products and features. When we launch something new, we get signal immediately and at an incredible scale. We use that signal to swiftly address bugs and iterate on our products. 

Offering a Free plan challenges us to build more intuitive products. Free customers represent a broad audience, from tech enthusiasts to those simply looking to secure their website or build an application. Building for a broad spectrum of users forces us to create more user-friendly tools for everyone.

Offering a Free service has other benefits, too. Some of our strongest customer advocates are folks that used our Free plan on their hobby projects before bringing Cloudflare with them to work. Some of them even end up working at Cloudflare!

Our Free offering is a flywheel that helps make Cloudflare’s products, team, and cost structure more efficient. We pay back these efficiencies by continuing to improve our free offerings. Just this week, we’ve announced 16 updates that make our Free plans even better:

• Free customers can audit and control the AI models accessing their content.

• Turnstile, our privacy-first CAPTCHA alternative available to everyone, gets more accurate with granular, client-level identification.

• Free customers now have access to our Cloud Access Security Broker (CASB), Data Loss Prevention (DLP), Digital Experience Monitoring (DEX), and Magic Network Monitoring (MNM) tools, for up to 50 seats.

• A new version of Leaked Credential Checks (LCC) is available to all customers to help mitigate account takeover (ATO) attacks.

• All customers can now monitor third-party scripts with Page Shield Script Monitor.

• Free customers can use API Shield’s Schema Validation to ensure only valid requests to their API make it through to the origin.

• Free customers get more robust analytics, with versions of Security Analytics and DNS GraphQL for everyone.

• All customers can now log in to the Cloudflare Dashboard using Sign in with Google.

• Free customers using our Terraform provider to configure their infrastructure will now benefit from autogenerated API SDKs.

• Cloudflare Calls managed TURN service is now GA and free up to 1,000 GB per month.

• All customers will benefit from the introduction of Zstandard compression, which improves web performance by compressing up to 42% faster than Brotli.

• Free customer traffic is now more private as we roll out Encrypted Client Hello (ECH) which obfuscates the Server Name Identifier (SNI) during a TLS handshake.

• All customers can store and query 3 days of logs from their Cloudflare Worker.

• Requests made through Service Bindings and to Tail Workers are now free.

• Cloudflare Image Optimization is now available for free to all Cloudflare customers.

• Free domains just got 45% faster with Speed Brain enabled.

We offer a Free plan out of more than goodwill — it is a core business differentiator that helps us build better products, drive growth, and keep costs low. And it helps us advance our mission. Building a better Internet is a collective effort. Today, more than 30 million Internet properties, comprising some 20% of the web, sit behind Cloudflare. Our Free plan makes that portion of the web faster, more secure, and more efficient. Free is not just a commitment — it’s a cornerstone of our strategy.

Become part of a better Internet and sign up for Cloudflare’s Free plan.

Cloudflare's network is one of the biggest, most connected, and fastest in the world. It extends to more than 250 cities. In those cities, we're often present in multiple data centers in order to connect to as many networks and bring our services as close to as many users as possible. We're always asking ourselves: how can we get closer to even more of the world's Internet users?

Today, we're taking a big step toward that goal.

Introducing Cloudflare for Offices. We are creating strategic partnerships that will enable us to extend Cloudflare's network into the world's busiest office buildings and multi-dwelling units. These buildings span the globe, and are where millions of people work every day; now, they’re going to be microseconds away from our global network.

And we're not done. We've built custom secure hardware and partnered with fiber providers to scale this model globally. It will bring a valuable new resource to the literal doorstep of building tenants.

Cloudflare has built a mutually beneficial relationship with the world's ISPs by reducing their operational costs and improving customer performance. Similarly, we expect a mutually beneficial relationship as we roll out Cloudflare for Offices. Real estate operators & service offices upgraded with this amenity increase the value and occupancy of their portfolio. IT teams can enforce a consistent security posture while enabling flexible work environments from any location their employees prefer. And employees in these smart spaces, experiencing faster Internet performance, can be more productive, seamlessly working as they choose, be it at the office, at home, or on the go.

There’s no disputing the fact that the nature of work has undergone a tremendous shift over the past 18 months. While we still don’t know what the future of work will look like exactly, here’s what we do know: it’s going to require more flexibility, all while maintaining security and performance standards that are a prerequisite for operating on today’s Internet. Enabling flexibility, and improving performance AND security (as opposed to trading one off for the other) has been a long held belief of Cloudflare. Alongside, of course, driving value for organizations.

Cloudflare for Offices — by connecting directly with enterprises — enables us to now do that for commercial office space.

There are a variety of advantages to Cloudflare for Offices. First and foremost, it eliminates the need to rely on the costly, rigid hardware solutions and multiple, regional, third parties that are often required to provide secure and performant branch office connectivity. Businesses have maintained expensive and hardware-intensive office networks since the dawn of the modern Internet.

Never have they gotten less return on that investment than through the pandemic.

The hybrid future of work will only exacerbate the high costs and complexity of maintaining and securing this outdated infrastructure. MPLS links. WANs. Hardware firewalls. VPNs. All these remain mainstays of the modern office. In the same way that we look back on maintaining server rooms for compute and storage as complete anachronisms, so too will we soon look back on maintaining all these boxes in an office. We’ve spoken to customers who now have over half of their workforce remote, and who are considering giving up their office space or increasing their presence in shared workspaces. Some are being hamstrung because of a need for MPLS to make their network operate securely. But it’s not just customers. This is a problem that we ourselves have been facing. Setting up new offices, or securing and optimizing shared workspaces, is a huge lift, physically as well as technologically.

Cloudflare for Offices simplifies this: a direct connection to Cloudflare’s network puts all office traffic behind Cloudflare’s services. Now, creating an office is as simple as plugging a cable into our box, and all the security and performance features that an office typically needs are microseconds away. It also enables the creation of custom topologies on Cloudflare's network, dramatically increasing the flexibility of your physical footprint.

"Throughout the pandemic, we've supported our over 12,000 employees to work safely and seamlessly from home or from our offices. Cloudflare solutions have been critical, and we're excited to continue to partner on efficient and strong solutions.”- Mark Papermaster, CTO and Executive Vice President, Technology and Engineering, AMD

COVID-19 hasn’t just driven a paradigm shift in where people work, however. It’s also driven a paradigm shift in how organizations think about IT security.

The old model — castle and moat — was designed during the desktop era, when most computing happened on premises. Everyone within the walls of the enterprise was considered authenticated; if you were outside the office, you needed to “tunnel” in through the moat in the castle of the office. As more and more users entered the portable era — through laptops and smartphones — then more tunnels were created.

The pandemic made it so that everyone was outside the moat, tunneling into an empty castle. Nobody was in the office anymore. The paradigm has been stretched to a parody.

Google was one of the first organizations to start to think about how things could be done differently: it proposed a model called BeyondCorp, which treated internal employees to an organization similar to how it treated external customers or suppliers to an organization. To put it simply: nobody is trusted, no matter if they’re in the office or not. If you want access to something, be prepared to prove you are who you say you are.

Fast-forward to 2021, and this model — otherwise known as Zero Trust — has become the gold standard of enterprise security, to which more and more organizations are implementing. Cloudflare’s Zero Trust solution — Cloudflare for Teams — has become increasingly popular for not just its advanced functionality and its ease of use, but because, when coupled with our enterprise connectivity offerings, allows you to run more and more of your traffic across Cloudflare’s network. We call this holistic solution Cloudflare One, and it provides your organization a virtual private network in the cloud, with all the associated security and visibility benefits.

Cloudflare for Offices is the onramp for offices onto Cloudflare One. It’s a fast, private onramp for your office network traffic straight onto the Cloudflare network — with all the security and visibility benefits that running your traffic over our network provides.

We also realize that for many organizations, Zero Trust is a journey. Not every customer is ready to go from MPLS and built-out networks to trusting the public Internet overnight. Cloudflare for Offices is a great start in the journey — by building out your own networks on top of Cloudflare, you reduce your threat vectors while being able to keep your existing topologies. This gives you the privacy and security of Cloudflare One, but with the flexibility to build Zero Trust any way you choose.

But security and visibility are not the only benefits. One of the common complaints we hear from customers about competing solutions is that performance can be extremely variable. The proximity Cloudflare has to so many people around the world is important because when employees connect using a Zero Trust solution, at least a subset (but often all) the traffic going from an end-user device needs to connect to the Zero Trust provider. Having Cloudflare equipment close means that the performance of the user device will be vastly increased as opposed to having to connect to a far off data center. You’ve probably read about what happens when Cloudflare takes control of your Last Mile connectivity and your network to your data centers. And you know that connecting to a Cloudflare data center in the same city increases performance, but imagine what happens when you’re connecting to Cloudflare in your office basement. And when you think about all the employees that you have are running on a zero trust model, that performance difference sums up to a lot of additional employee productivity.

Up until now, something like this has been extremely expensive, complicated, and oftentimes, slow.

“We see a lot of potential in the way Cloudflare is bringing its network directly to our office locations. It’s critical that we empower our employees to work productively and securely, and this makes it that much easier for us to do so no matter where our teams are working from in the future–and reducing our network costs along the way.”- Aaron Dearinger, Edge Architect, Garmin International

Cloudflare for Offices allows for customers to choose their Network as a Service: let us manage your footprint and build your network out however you like.

But it’s not just zero trust that gets a boost. Workers, Cloudflare’s serverless platform, runs on the edge from the nearest data center to the user making the request. As you might have already read: it’s fast. With more and more business and application logic being moved to Workers, your end users stand to benefit.

But it does beg the question: just how fast are we talking?

Photo by Denys Nevozhai on Unsplash

The tallest commercial office building in San Francisco is Salesforce Tower.  It’s 1,070 feet tall. A light signal running from the top of the building to the basement along a single-mode fiber cable would take no more than 6 µs (6 microseconds) to complete its journey. Cloudflare For Offices deployments in buildings like this  put customers fractions of a millisecond away from Cloudflare’s network. The edge is becoming indistinguishable in performance from local compute.

We’ve written many times before about how Cloudflare designs our hardware. But deploying Cloudflare hardware outside of data centers — and into office basements — presented a new set of challenges. Cooling, energy efficiency, and resiliency were even more important in the design. Similarly, these are going to be deployed to offices all over the world; they needed to be cost-effective. Finally, and perhaps most importantly, there is also a security aspect to this: we could not assume the same level of access control inside a building as we could inside a data center.

This is where the inherent advantages of designing and owning the hardware come to the fore. Because of it, we’re able to build exactly what we need for the environment: ranging from how resilient these devices need to be, to an appropriate level of security given where they’re going to be operating. In fact, we have been working on hardware security for the last five years in anticipation of the launch of Cloudflare for Offices. We're starting with switching, and we plan to add compute and storage capabilities in short order. Stay tuned for more details.

If you’re an organization (tenant) in a large office building, an owner/operator of multi-tenant (or multi-dwelling) real estate, or a co-working space looking to bring Cloudflare to your doorstep — with all the flexibility, performance and security enhancements, and cost savings that would entail — then we’d love for you to get in touch with us.

Cloudflare has millions of free customers. Not only is it something we’re incredibly proud of in the context of helping to build a better Internet — but it’s something that has made the Cloudflare service measurably better. One of the ways we’ve benefited is that it’s created a very strong imperative for Cloudflare to maintain a network that is as efficient as possible. There’s simply no other way to serve so many free customers.

In the spirit of this, we are very excited about the latest step in our energy-efficiency journey: turning to Arm for our server CPUs. It has been a long journey getting here — we first started testing Arm CPUs all the way back in November 2017. It’s only recently, however, that the quantum of energy efficiency improvement from Arm has become clear.  Our first deployment of an Arm-based CPU, designed by Ampere, was earlier this month – July 2021.

Our most recently deployed generation of edge servers, Gen X, used AMD Rome CPUs. Compared with that, the newest Arm based CPUs process an incredible 57% more Internet requests per watt. While AMD has a sequel, Milan (and which Cloudflare will also be deploying), it doesn’t achieve the same degree of energy efficiency that the Arm processor does — managing only 39% more requests per watt than Rome CPUs in our existing fleet. As Arm based CPUs become more widely deployed, and our software is further optimized to take advantage of the Arm architecture, we expect further improvements in the energy efficiency of Arm servers.

Using Arm, Cloudflare can now securely process over ten times as many Internet requests for every watt of power consumed, than we did for servers designed in 2013.

(In the graphic below, for 2021, the perforated data point refers to x86 CPUs, whereas the bold data point refers to Arm CPUs)

As Arm server CPUs demonstrate their performance and become more widely deployed, we hope this will inspire x86 CPUs manufacturers (such as Intel and AMD) to urgently take energy efficiency more seriously. This is especially important since, worldwide, x86 CPUs continue to represent the vast majority of global data center energy consumption.

Together, we can reduce the carbon impact of Internet use. The environment depends on it.

Since March 2020, the Internet has been the trusty sidekick that’s helped us through the pandemic. Or so it seems to those of us lucky enough to have fast, reliable (and often cheap) Internet access.

With a good connection you could keep working (if you were fortunate enough to have a job that could be done online), go to school or university, enjoy online entertainment like streaming movies and TV, games, keep up with the latest news, find out vital healthcare information, schedule a vaccination and stay in contact with loved ones and friends with whom you’d normally be spending time in person.

Without a good connection though, all those things were hard or impossible.

Sadly, access to the Internet is not uniformly distributed. Some have cheap, fast, low latency, reliable connections, others have some combination of expensive, slow, high latency and unreliable connections, still others have no connection at all. Close to 60% of the world have Internet access leaving a huge 40% without it at all.

This inequality of access to the Internet has real-world consequences. Without good access it is so much harder to communicate, to get vital information, to work and to study. Inequality of access isn’t a technical problem, it’s a societal problem.

This week, Cloudflare is announcing Project Pangea with the goal of helping reduce this inequality. We’re helping community networks get onto the Internet cheaply, securely and with good bandwidth and latency. We can't solve all the challenges of bringing fast, cheap broadband access to everyone (yet) but we can give fast, reliable transit to ISPs in underserved communities to help move in that direction. Please refer to our Pangea announcement for more details.

To understand why Project Pangea is important, you need to understand how different the experience of accessing the Internet is around the world. From a distance, the world looks blue and green. But we all know that our planet varies wildly from place to place: deserts and rainforests, urban jungles and placid rural landscapes, mountains, valleys and canyons, volcanos, salt flats, tundra, and verdant, rolling hills.

Cloudflare is in a unique position to measure the performance and reach of the Internet over this vast landscape. We have servers in more than 200 cities in over 100 countries, we process 10s of trillions of Internet requests every month. Our network and customers and their users span the globe, every country in every network.

Zoom out to the level of a city, county, state, or country, and average Internet performance can look good — or, at least, acceptable. Zoom in, however, and the inequalities start to show. Perhaps part of a county has great performance, and another limps along at barely dial-up speeds — or worse. Or perhaps a city has some neighborhoods with fantastic fiber service, and others that are underserved and struggling with spotty access.

Inequality of Internet access isn’t a distant problem, it’s not limited to developing countries, it exists in the richest countries in the world as well as the poorest. There are still many parts of the world  where a Zoom call is hard or impossible to make. And if you’re reading this on a good Internet connection, you may be surprised to learn that places with poor or no Internet are not far from you at all.

For Impact Week, we’ve analyzed Internet data in the United States, Brazil, United Kingdom, Germany, France, South Africa, Japan, and Australia to build a picture of Internet performance.

Below, you’ll find detailed maps of where the Internet is fast and slow (focusing on available bandwidth) and far away from the end user (at least in terms of the latency between the client and server). We’d have loved to have used a single metric, however, it’s hard for a single number to capture the distribution of good, bad, and non-existent Internet traffic in a region. It’s for that reason that we’ve used two metrics to represent performance: latency and bandwidth (otherwise known as throughput). The maps below are colored to show the differences in bandwidth and latency and answer part of the question: “How good is the Internet in different places around the world?”

As we like to say, we’re just getting started with this — we intend to make more of this data and analysis available in the near future. In the meantime, if you’re a local official who wants to better understand their community’s relative performance, please reach out — we’d love to connect with you. Or, if you’re interested in your own Internet performance, you can visit speed.cloudflare.com to run a personalized test on your connection.

Before we begin, a quick reminder: latency (usually measured in milliseconds or ms) is the time it takes for communications to go to an Internet destination from your device and back, whereas bandwidth is the amount of data that can be transferred in a second (it’s usually measured in megabits per second or Mbps).

Both latency and bandwidth affect the performance of an Internet connection. High latency particularly affects things like online gaming where quick responses from servers are needed, but also shows up by slowing down the loading of complex web pages, and even interrupting some streaming video. Low bandwidth makes downloading anything slow: be it images on a webpage, the new app you want to try out on your phone, or the latest movie.

Blinking your eyes takes about 100ms; but you’ll begin to notice performance changes around 60ms of latency and below 30ms is gold class performance, seeing little to no delay in video streaming or gaming.

United States

United States median throughput: 50.27Mbps

US median latency: 46.69ms

The US government has long recognized the importance of improving the Internet for underserved communities, but the Federal Communications Commission (FCC), the US agency responsible for determining where investment is most needed, has struggled to accurately map Internet access across the country.  Although the FCC has embarked on a new data collection effort to improve the accuracy of existing maps, the US government still lacks a comprehensive understanding of the areas that would most benefit from broadband investment.

Cloudflare’s data confirms the overall concerns with inconsistent access to the Internet and helps fill in some of the current gaps.  A glance at the two maps of the US below will show that, even zoomed out to county level, there is inequality across the country. High latency and low bandwidth stand out as red areas.

US locations with the lowest latency (best) and highest latency (worst) are as follows.

When thinking about bandwidth, 5 to 10Mbps are generally good enough for video conferencing, but ultra-HD TV watching might consume up to 20Mbps easily. For context, the Federal Communications Commission (FCC) defines the minimum bandwidth for “Advanced Service” at 25 Mbps.

The best performing (i.e., the highest bandwidth) in the US tells an interesting story. New York City comes out on top, but if you were to zoom in on the city you’d find pockets of inequality. You can read more about our partnership with NYC Mesh in the Project Pangea post and how they are helping bring better Internet to underserved parts of the Big Apple. Notice how the tyranny of averages can disguise a problem.

Contrary to popular discourse about access to the Internet as a product of the rural-urban divide, we found that poor performance was not unique to rural areas. Los Angeles, Milwaukee, Florida’s Orange County, Fairfax, San Bernardino, Knox County, and even San Francisco have pockets of uniformly poor performance, often while adjoining ZIP codes have stronger performance.

Even in areas with excellent Internet connectivity, the same connectivity to the same resources can cost wildly different amounts. Internet prices for end-users correlates with the number of ISPs in an area, i.e. the greater the consumer choice, the better the price. President Biden's recent competition Executive Order, called out the lack of choice for broadband, noting “More than 200 million U.S. residents live in an area with only one or two reliable high-speed internet providers, leading to prices as much as five times higher in these markets than in markets with more options.”

The following cities have the greatest choice of Internet providers:

One might expect less populated areas to have uniformly slower performance. There are, however, pockets of poor performance even in densely populated areas such as Los Angeles (California), Milwaukee (Wisconsin), Orange County (Florida), Fairfax (Virginia),  San Bernardino (California), Knox County (Tennessee), and even San Francisco (California).

In as many as 9% of ZIP codes, average latency exceeds 150ms, the acceptable threshold of performance to run a videoconferencing service such as Zoom.

Australia

Australia median throughput: 33.34Mbps

Australia median latency: 42.04ms

In general, Australia seems to suffer very poor broadband speeds, with speeds that are not capable of sustaining households watching video streaming, and possibly struggling with multiple video calls. The problem isn’t just a rural one either, while the inner cities showed good broadband speed, often with fiber-to-the-building Internet access, suburban areas suffered. Larger suburban areas like the Illawarra had similar speeds to more rural centers like Wagga Wagga, showing this is more than just an urban divide.

The irony is that, from a latency perspective, Australia actually performs quite well.

Japan

Japan median throughput: 61.4Mbps

Japan median latency: 31.89ms

Japan’s Internet has consistently low latency, including in distant areas such as Okinawa prefecture, 1,000 miles away from Tokyo.

However, it's a different story when it comes to bandwidth. Several prefectures in Kyushu Island, Okinawa Prefecture, and Western Honshu have performance falling behind the rest of the country. Unsurprisingly, the best Internet performance is seen in Tokyo, with the highest concentration of people and data centers.

United Kingdom

United Kingdom median throughput: 53.8Mbps

United Kingdom median latency: 34.12ms

The United Kingdom has good latency throughout most of the country, however bandwidth is a different story. The best performance is seen in inner London as well as some other larger cities like Manchester. London and Manchester are also the homes of the UK's largest Internet exchange points. More effort to localize data into other cities, like Edinburgh, would be an important step to improving performance for those regions.

Germany

Germany median throughput: 48.79Mbps

Germany median latency: 42.1ms

Germany has some of the best performance centered on Frankfurt am Main, which is one of the major Internet hubs of the world, however what was formerly East Germany, has higher latency, and slower speeds, leaning to a poorer Internet performance.

France

France median throughput: 48.51Mbps

France median latency: 54.2ms

Paris has long been the Internet hub in France. Marseille has started to grow as a hub, especially with the large number of submarine cables landing. Other interconnection hubs in Lyon and Bordeaux are where we’ll start to see growth as Internet hubs. These four cities are where we also see the best performance, with the highest speeds and lowest latencies, giving the best Internet performance.

Brazil

Brazil median throughput: 26.28Mbps

Brazil median latency: 49.25ms

Much of Brazil has good, low latency Internet performance, given geographic proximity to the major Internet hubs in São Paulo and Rio de Janeiro. Much of the Amazon has low speeds and high latency, for those parts that are actually connected to the Internet.

Campinas is one stand out, with some of the best performing Internet across Brazil, and is also the site of a recent Cloudflare data center launch.

South Africa

South Africa median throughput: 6.4Mbps

South Africa median latency: 59.78ms

Johannesburg has been the historical hub for South Africa’s Internet. This is where many Internet giants have built data centers, and it shows in latency as distance from Johannesburg. South Africa has grown to have two more Internet hubs in Cape Town and Durban. Internet performance also follows these three cities. However, much of South Africa’s Internet performance lacks the ability for video streaming and video conferencing in high definition.

One question we had as we went through a lot of this data: does ISP concentration impact Internet performance?

On one hand, there’s a case to be made that more ISP competition results in no one vendor being able to invest sufficient resources to build out a fast network. On the other hand, well, classical economics would suggest that monopolies are bad, right?

To investigate the question further, we did a deep dive into Alabama in the United States, the 24th most populous state in the US. We tracked two key metrics across 65 counties: Internet performance as defined by average download speed, and ISP concentration, as measured by the largest ISP’s traffic share.

Here is the raw data:

Across most of Alabama, we see very high ISP concentration. For the majority of counties, the largest ISP has 80% (or higher) share of traffic, while all the other ISPs combined operate at considerably smaller scale. In only three counties (Marion, Escambia and Etowah) does each ISP carry less than 50% of user traffic. Interestingly, Etowah is one of the best performing in the state, while Henry, a county where 98% of Internet traffic is concentrated behind a single ISP is the worst performing.

Where it gets interesting is when you plot the data, tracking the non-dominant ISP by traffic share (which is simply 100% less the traffic share of the dominant ISP) against the performance (as measured by download speed) and then use a linear line of best fit to find the relationship. Here’s what you get:

As you can see, there is a strong positive relationship between the non-dominant ISP’s traffic share and the average download speed. As the non-dominant ISP increases its traffic share, Internet speeds tend to improve. The conclusion is clear: if you want to improve Internet performance in a region, foster more competition between multiple Internet service providers.

There is more to the story, however, than just concentration. Alabama, like a lot of other regions that aren’t served well by ISPs, faces another performance challenge: poor routing, also sometimes known as “tromboning”.

Consider Tuskegee in Alabama, home to a local university.

In Tuskegee, choice is limited. Consumers only have a single choice for high-speed broadband. But even once an off-campus student has local access to the Internet, it isn’t truly local: Tuskegee students on a different ISP than their university will likely see their traffic detour all the way through Atlanta (two hours northeast by car!) before making its way back to school.

This doesn’t happen in isolation: today, the largest ISPs only exchange traffic with other networks in a handful of cities, notably Seattle, San Jose, Los Angeles, Dallas, Chicago, Atlanta, Miami, Ashburn, and New York City.

If you’re in one of these big cities, you’re unlikely to suffer from tromboning. But if you’re not? Your Internet traffic can often have to travel further away before looping back, similar to the shape of a trombone, reducing your Internet performance. Tromboning contributes to inefficiency and drives up the cost of Internet access. An increasing amount of traffic is wastefully carried to cities far away, instead of keeping the data local.

You can visualize how your Internet traffic is flowing, by using tools like traceroute.

As an example, we ran tests using RIPE Atlas probes to Facebook from Alabama, and unfortunately found extremes where traffic can sometimes take a highly circuitous route — traffic going to Atlanta, then Ashburn, Paris, Amsterdam, before making its way back to Alabama. The path begins on AT&T's network and goes to Atlanta where it enters the network for Telia (an IP transit provider), crosses the Atlantic, meets Facebook, and then comes back.

Traceroute to 157.240.201.35 (157.240.201.35), 48 byte packets
1- 192.168.6.1 1.435ms 0.912ms 0.636ms
2-  99.22.36.1 99-22-36-1.lightspeed.dctral.sbcglobal.net AS7018 1.26ms 1.134ms 1.107ms
3-  99.173.216.214 AS7018 3.185ms 3.173ms 3.099ms
4-  12.122.140.70 cr84.attga.ip.att.net AS7018 11.572ms 13.552ms 15.038ms
5 - * * *
6- 192.205.33.42 AS7018 8.695ms 9.185ms 8.703ms
7-  62.115.125.129 ash-bb2-link.ip.twelve99.net AS1299 23.53ms 22.738ms 23.012ms
8-  62.115.112.243 prs-bb1-link.ip.twelve99.net AS1299 115.516ms 115.52ms 115.211ms
9-  62.115.134.96 adm-bb3-link.ip.twelve99.net AS1299 113.487ms 113.405ms 113.25ms
10-  62.115.136.195 adm-b1-link.ip.twelve99.net AS1299 115.443ms 115.703ms 115.45ms
11- 62.115.148.231 facebook-ic331939-adm-b1.ip.twelve99-cust.net AS1299 134.149ms 113.885ms 114.246ms
12- 129.134.51.84 po151.asw02.ams2.tfbnw.net AS32934 113.27ms 113.078ms 113.149ms
13-  129.134.48.101 po226.psw04.ams4.tfbnw.net AS32934 114.529ms 114.439ms 117.257ms
14-  157.240.38.227 AS32934 113.281ms 113.365ms 113.448ms
15- 157.240.201.35 edge-star-mini-shv-01-ams4.facebook.com AS32934 115.013ms 115.223ms 115.112ms

The intent here isn’t to shame AT&T, Telia, or Facebook — nor is this challenge unique to them. Facebook's content is undoubtedly cached in Atlanta and the request from Alabama should go no further than that. While many possible conditions within and between these three networks could have caused this tromboning, in the end, the consumer suffers.

The solution? Have more major ISPs exchange in more cities and with more networks. Of course, there’d be an upfront cost involved in doing so, even if it would reduce cost more over the long run.

As William Gibson famously observed: the future is here, but it’s just not evenly distributed.

One of the clearest takeaways from the data and analysis presented here is that Internet access varies tremendously across geographies. But it’s not just a case of the developed world vs the developing, or even rural vs urban. There are underserved urban communities and regions of the developed world that do not score as highly as you might expect.

Furthermore, our case study of Alabama shows that the structure of the ISP market is incredibly important to promoting performance. We found a strong positive correlation between more competition and faster performance. Similarly, there’s a lot of opportunity for more networks to interconnect in more places, to avoid bad routing.

Finally, if we want to get the other 40% of the world online, we are going to need more initiatives that drive up access and drive down cost. There’s plenty of scope to help — and we’re excited to be launching Project Pangea to help.

When web hosting services first emerged in the mid-1990s, you paid for everything on a separate meter: bandwidth, storage, CPU, and memory. Over time, customers grew to hate the nickel-and-dime nature of these fees. The market evolved to a fixed-fee model. Then came Amazon Web Services.

AWS was a huge step forward in terms of flexibility and scalability, but a massive step backward in terms of pricing. Nowhere is that more apparent than with their data transfer (bandwidth) pricing. If you look at the (ironically named) AWS Simple Monthly Calculator you can calculate the price they charge for bandwidth for their typical customer. The price varies by region, which shouldn't surprise you because the cost of transit is dramatically different in different parts of the world.

AWS charges customers based on the amount of data delivered — 1 terabyte (TB) per month, for example. To visualize that, imagine data is water. AWS fills a bucket full of water and then charges you based on how much water is in the bucket. This is known as charging based on “stocks.”

On the other hand, AWS pays for bandwidth based on the capacity of their network. The base unit of wholesale bandwidth is priced as one Megabit per second per month (1 Mbps). Typically, a provider like AWS, will pay for bandwidth on a monthly fee based on the number of Mbps that their network uses at its peak capacity. So, extending the analogy, AWS doesn't pay for the amount of water that ends up in their customers' buckets, but rather the capacity based on the diameter of the “hose” that is used to fill them. This is known as paying for “flows.”

You can translate between flow and stock pricing by knowing that a 1 Mbps connection (think of it as the "hose") can transfer 0.3285 TB (328GB) if utilized to its fullest capacity over the course of a month (think of it as running the "hose" at full capacity to fill the "bucket" for a month).¹ AWS obviously has more than 1 Mbps of capacity — they can certainly transfer more than 0.3285 TB per month — but you can use this as the base unit of their bandwidth costs, and compare it against what they charge a customer to deliver 1 Terabyte (1TB), in order to figure out the AWS bandwidth markup.

One more subtlety to be as accurate as possible. Wholesale bandwidth is also billed at the 95th percentile. That effectively cuts off the peak hour or so of use every day. That means a 1 Mbps connection running at 100% can actually likely transfer closer to 0.3458 TB (346GB) per month.

Two more factors are important: utilization and regional costs. AWS can't run all their connections at 100% utilization 24x7 for a month. Instead, they'll have some average utilization per transit connection in any month. It’s reasonable to estimate that they likely run at between 20% and 40% average utilization. That would be a typical average utilization range for the industry. The higher their utilization, the more efficient they are, the lower their costs, and the higher their effective customer markup will be.

To be conservative, we’ve assumed that AWS’s average utilization is the bottom of that range (20%), but you can download the raw data and adjust the assumptions however you think makes sense.

We have a good sense of the wholesale prices of bandwidth in different regions around the world based on what Cloudflare sees in the market when we buy bandwidth ourselves. We’d imagine AWS gets at least as good of pricing as we do. We’ve included a rough estimate of these prices in the calculation, rounding up on the wholesale price wherever there was a question (which makes AWS look better).

Based on these assumptions, here's our best estimate of AWS’s effective markup for egress bandwidth on a per-region basis.

Don’t rest easy, South Korea with your merely 357% markup. The general rule of thumb appears to be that the older a market is, the more Amazon wrings from its customers in egregious egress markups — and the Seoul availability zone is only a bit over four years old. Winter, unfortunately, inevitably seems to come to AWS customers.

Remember, this is for the transit bandwidth that AWS is paying for. For the bandwidth that they exchange with a network like Cloudflare, where they are directly connected (settlement-free peered) over a private network interface (PNI), there are no meaningful incremental costs and their effective margins are nearly infinite. Add in the effect of rebates Amazon collects from colocation providers who charge cross connect fees to customers, and the effective markup is likely even higher.

Some other cloud providers take into account that their costs are lower when passing over peering connections. Both Microsoft Azure and Google Cloud will substantially discount egress charges for their mutual Cloudflare customers. Members of the Bandwidth Alliance — Alibaba, Automattic, Backblaze, Cherry Servers, Dataspace, DNS Networks, DreamHost, HEFICED, Kingsoft Cloud, Liquid Web, Scaleway, Tencent, Vapor, Vultr, Wasabi, and Zenlayer — waive bandwidth charges for mutual Cloudflare customers.

At this point, the majority of hosting providers in the industry either substantially discount or entirely waive egress fees when sending traffic from their network to a peer like Cloudflare. AWS is the notable exception in the industry. It's worth noting that we invited AWS to be a part of the Bandwidth Alliance, and they politely declined.

It seems like a no-brainer that if we’re not paying for the bandwidth costs, and the hosting provider isn’t paying for the bandwidth costs, customers shouldn’t be charged for the bandwidth costs at the same rate as if the traffic was being sent over the public Internet. Unfortunately, Amazon’s supposed obsession over doing the right thing for customers doesn’t extend to egress charges.

Amazon’s mission statement is: “We strive to offer our customers the lowest possible prices, the best available selection, and the utmost convenience.” And yet, when it comes to egress, their prices are far from the lowest possible.

During the last ten years, industry wholesale transit prices have fallen an average of 23% annually. Compounded over that time, wholesale bandwidth is 93% less expensive than 10 years ago. However, AWS’s egress fees over that same period have fallen by only 25%.

And, since 2018, the egress fees AWS charges in North America and Europe have not dropped a penny even as wholesale prices in those markets over the same time period have fallen by more than half.

Another oddity of AWS’s pricing is that they charge for data transferred out of their network but not for data transferred into their network. If the only time you’ve paid for bandwidth is with your residential Internet connection, then this may make some sense. Because of some technical limitations of the cable network, download bandwidth is typically higher than upload bandwidth on cable modem connections. But that’s not how wholesale bandwidth is bought or sold.

Wholesale bandwidth isn’t like your home cable connection. Instead, it’s symmetrical. That means that if you purchase a 1 Mbps (1 Megabit per second) connection, then you have the capacity to send 1 Megabit out and receive another 1 Megabit in every second. If you receive 1 Mbps in and simultaneously 1 Mbps out, you pay the same price as if you receive 1 Mbps in and 0 Mbps out or 0 Mbps in and 1 Mbps out. In other words, ingress (data sent to AWS) doesn’t cost them any more or less than egress (data sent from AWS). And yet, they charge customers more to take data out than put it in. It’s a head scratcher.

We’ve tried to be charitable in trying to understand why AWS would charge this way. Disappointingly, there just doesn't seem to be an innocent explanation. As we dug in, even things like writes versus reads and the wear they put on storage media, as well as the challenges of capacity planning for storage capacity, suggest that AWS should charge less for egress than ingress.

But they don’t.

The only rationale we can reasonably come up with for AWS’s egress pricing: locking customers into their cloud, and making it prohibitively expensive to get customer data back out. So much for being customer-first.

AWS may object that this doesn't take into account the cost of things like metro dark fiber between data centers, amortized optical and other networking equipment, and cross connects. In our experience, those costs amount to a rounding error of less than one cent per Mbps when operating at AWS-like scale. And these prices have been falling at a similar rate to the decline in the price of bandwidth over the past 10 years. Yet AWS’s egress prices have barely budged.

All the data above is derived from what’s published on AWS’s simple pricing calculator. There’s no doubt that some large customers are able to negotiate lower prices. But these are the prices charged to small businesses and startups by default. And, when we’ve reviewed pricing even with large AWS customers, the egress fees remain egregious.

We have a lot of mutual customers who use Cloudflare and AWS. They’re a great service, and we want to support our mutual customers and provide services in a way that meets their needs and is always as secure, fast, reliable, and efficient as possible. We remain hopeful that AWS will do the right thing, lower their egress fees, join the Bandwidth Alliance — following the lead of the majority of the rest of the hosting industry — and pass along savings from peering with Cloudflare and other networks to all their customers.

.......

¹Here’s the calculation to convert a 1 Mbps flow into TB stocks: 1 Mbps @ 100% for 1 month = (1 million bits per second) * (60 seconds / minute) * (60 minutes / hour) * (730 hours on average/month) divided by (eight bits / byte) divided by 10^12 (to convert bytes to Terabytes) = 0.3285 TB/month.

We're also mindful of events like the Spectre and Meltdown vulnerabilities and have been working with outside parties on research into mitigation and exploitation which we hope to publish later this year.

We looked very seriously at ARM-based CPUs and continue to keep our software up to date for the ARM architecture so that we can use ARM-based CPUs when the requests per watt is interesting to us.

In the meantime, we've deployed AMD's EPYC processors as part of Gen X server platform and for the first time are not using any Intel components at all. This week, we announced details of this tenth generation of servers. Below is a recap of why we're excited about the design, specifications, and performance of our newest hardware.

Every server can run every service. This architectural decision has helped us achieve higher efficiency across the Cloudflare network. It has also given us more flexibility to build new software or adopt the newest available hardware.

Notably, Intel is not inside. We are not using their hardware for any major server components such as the CPU, board, memory, storage, network interface card (or any type of accelerator).

This time, AMD is inside.

Compared with our prior server (Gen 9), our Gen X server processes as much as 36% more requests while costing substantially less. Additionally, it enables a ~50% decrease in L3 cache miss rate and up to 50% decrease in NGINX p99 latency, powered by a CPU rated at 25% lower TDP (thermal design power) per core.

To identify the most efficient CPU for our software stack, we ran several benchmarks for key workloads such as cryptography, compression, regular expressions, and LuaJIT. Then, we simulated typical requests we see, before testing servers in live production to measure requests per watt.    

Based on our findings, we selected the single socket 48-core AMD 2nd Gen EPYC 7642.

The single AMD EPYC 7642 performed very well during our lab testing, beating our Gen 9 server with dual Intel Xeon Platinum 6162 with the same total number of cores. Key factors we noticed were its large L3 cache, which led to a low L3 cache miss rate, as well as a higher sustained operating frequency.

Partnering with AMD, we tuned the 2nd Gen EPYC 7642 processor to achieve additional 6% performance. We achieved this by using power determinism and configuring the CPU's Thermal Design Power (TDP).

Finally, we described how we use Secure Memory Encryption (SME), an interesting security feature within the System on a Chip architecture of the AMD EPYC line. We were impressed by how we could achieve RAM encryption without significant decrease in performance. This reduces the worry that any data could be exfiltrated from a stolen server.

We enjoy designing hardware that improves the security, performance and reliability of our global network, trusted by over 26 million Internet properties.

Want to help us evaluate new hardware? Join us!

“Every server can run every service.”

We designed and built Cloudflare’s network to be able to grow capacity quickly and inexpensively; to allow every server, in every city, to run every service; and to allow us to shift customers and traffic across our network efficiently. We deploy standard, commodity hardware, and our product developers and customers do not need to worry about the underlying servers. Our software automatically manages the deployment and execution of our developers’ code and our customers’ code across our network. Since we manage the execution and prioritization of code running across our network, we are both able to optimize the performance of our highest tier customers and effectively leverage idle capacity across our network.

An alternative approach might have been to run several fragmented networks with specialized servers designed to run specific features, such as the Firewall, DDoS protection or Workers. However, we believe that approach would have resulted in wasted idle resources and given us less flexibility to build new software or adopt the newest available hardware. And a single optimization target means we can provide security and performance at the same time.

We use Anycast to route a web request to the nearest Cloudflare data center (from among 200 cities), improving performance and maximizing the surface area to fight attacks.

Once a datacenter is selected, we use Unimog, Cloudflare’s custom load balancing system, to dynamically balance requests across diverse generations of servers. We load balance at different layers: between cities, between physical deployments located across a city, between external Internet ports, between internal cables, between servers, and even between logical CPU threads within a server.

As demand grows, we can scale out by simply adding new servers, points of presence (PoPs), or cities to the global pool of available resources. If any server component has a hardware failure, it is gracefully de-prioritized or removed from the pool, to be batch repaired by our operations team. This architecture has enabled us to have no dedicated Cloudflare staff at any of the 200 cities, instead relying on help for infrequent physical tasks from the ISPs (or data centers) hosting our equipment.

We recently turned up our tenth generation of servers, “Gen X”, already deployed across major US cities, and in the process of being shipped worldwide. Compared with our prior server (Gen 9), it processes as much as 36% more requests while costing substantially less. Additionally, it enables a ~50% decrease in L3 cache miss rate and up to 50% decrease in NGINX p99 latency, powered by a CPU rated at 25% lower TDP (thermal design power) per core.

Notably, for the first time, Intel is not inside. We are not using their hardware for any major server components such as the CPU, board, memory, storage, network interface card (or any type of accelerator). Given how critical Intel is to our industry, this would until recently have been unimaginable, and is in contrast with prior generations which made extensive use of their hardware.

Intel-based Gen 9 server

This time, AMD is inside.

We were particularly impressed by the 2nd Gen AMD EPYC processors because they proved to be far more efficient for our customers’ workloads. Since the pendulum of technology leadership swings back and forth between providers, we wouldn’t be surprised if that changes over time. However, we were happy to adapt quickly to the components that made the most sense for us.

CPU efficiency is very important to our server design. Since we have a compute-heavy workload, our servers are typically limited by the CPU before other components. Cloudflare’s software stack scales quite well with additional cores. So, we care more about core-count and power-efficiency than dimensions such as clock speed.

We selected the AMD EPYC 7642 processor in a single-socket configuration for Gen X. This CPU has 48-cores (96 threads), a base clock speed of 2.4 GHz, and an L3 cache of 256 MB. While the rated power (225W) may seem high, it is lower than the combined TDP in our Gen 9 servers and we preferred the performance of this CPU over lower power variants. Despite AMD offering a higher core count option with 64-cores, the performance gains for our software stack and usage weren’t compelling enough.

We have deployed the AMD EPYC 7642 in half a dozen Cloudflare data centers; it is considerably more powerful than a dual-socket pair of high-core count Intel processors (Skylake as well as Cascade Lake) we used in the last generation.

Readers of our blog might remember our excitement around ARM processors. We even ported the entirety of our software stack to run on ARM, just as it does with x86, and have been maintaining that ever since even though it calls for slightly more work for our software engineering teams. We did this leading up to the launch of Qualcomm’s Centriq server CPU, which eventually got shuttered. While none of the off-the-shelf ARM CPUs available this moment are interesting to us, we remain optimistic about high core count offerings launching in 2020 and beyond, and look forward to a day when our servers are a mix of x86 (Intel and AMD) and ARM.

We aim to replace servers when the efficiency gains enabled by new equipment outweigh their cost.

The performance we’ve seen from the AMD EPYC 7642 processor has encouraged us to accelerate replacement of multiple generations of Intel-based servers.

Compute is our largest investment in a server. Our heaviest workloads, from the Firewall to Workers (our serverless offering), often require more compute than other server resources. Also, the average size in kilobytes of a web request across our network tends to be small, influenced in part by the relative popularity of APIs and mobile applications. Our approach to server design is very different than traditional content delivery networks engineered to deliver large object video libraries, for whom servers focused on storage might make more sense, and re-architecting to offer serverless is prohibitively capital intensive.

Our Gen X server is intentionally designed with an “empty” PCIe slot for a potential add on card, if it can perform some functions more efficiently than the primary CPU. Would that be a GPU, FPGA, SmartNIC, custom ASIC, TPU or something else? We’re intrigued to explore the possibilities.

In accompanying blog posts over the next few days, our hardware engineers will describe how AMD 7642 performed against the benchmarks we care about. We are thankful for their hard work.

Since we are typically limited by CPU, Gen X represented an opportunity to grow components such as RAM and SSD more slowly than compute.

For memory, we continued to use 256GB of RAM, as in our prior generation, but rated higher at 2933MHz. For storage, we continue to have ~3TB, but moved to 3x1TB form factor using NVME flash (instead of SATA) with increased available IOPS and higher endurance, which enables full disk encryption using LUKS without penalty. For the network card, we continue to use Mellanox 2x25G NIC.

We moved from our multi-node chassis back to a simple 1U form factor, designed to be lighter and less error prone during operational work at the data center. We also added multiple new ODM partners to diversify how we manufacture our equipment and to take advantage of additional global warehousing.

Our newest generation of servers give us the flexibility to continue to build out our network even closer to every user on Earth. We’re proud of the hard work from across engineering teams on Gen X, and are grateful for the support of our partners. Be on the lookout for more blogs about these servers in the coming days.

So far in 2019, we’ve added a score of new locations: Amman, Antananarivo*, Arica*, Asunción, Baku, Bengaluru, Buffalo, Casablanca, Córdoba*, Cork, Curitiba, Dakar*, Dar es Salaam, Fortaleza, Geneva, Göteborg, Guatemala City, Hyderabad, Kigali, Kolkata, Male*, Maputo, Nagpur, Neuquén*, Nicosia, Nouméa, Ottawa, Port-au-Prince, Porto Alegre, Querétaro, Ramallah, and Thessaloniki.

When Cloudflare launched in 2010, we focused on putting servers at the Internet’s crossroads: large data centers with key connections, like the Amsterdam Internet Exchange and Equinix Ashburn. This not only provided the most value to the most people at once but was also easier to manage by keeping our servers in the same buildings as all the local ISPs, server providers, and other people they needed to talk to streamline our services.

This is a great approach for bootstrapping a global network, but we’re obsessed with speed in general. There are over five hundred cities in the world with over one million inhabitants, but only a handful of them have the kinds of major Internet exchanges that we targeted. Our goal as a company is to help make a better Internet for all, not just those lucky enough to live in areas with affordable and easily-accessible interconnection points. However, we ran up against two broad, nasty problems: a) running out of major Internet exchanges and b) latency still wasn’t as low as we wanted. Clearly, we had to start scaling in new ways.

One of our first big steps was entering into partnerships around the world with local ISPs, who have many of the same problems we do: ISPs want to save money and provide fast Internet to their customers, but they often don’t have a major Internet exchange nearby to connect to. Adding Cloudflare equipment to their infrastructure effectively brought more of the Internet closer to them. We help them speed up millions of Internet properties while reducing costs by serving traffic locally. Additionally, since all of our servers are designed to support all our products, a relatively small physical footprint can also provide security, performance, reliability, and more.

Though it may be obvious and easy to overlook, continuing to build out existing locations is also a key facet of building a global network. This year, we have significantly increased the computational capacity at the edge of our network. Additionally, by making it easier to interconnect with Cloudflare, we have increased the number of unique networks directly connected with us to over 8,000. This makes for a faster, more reliable Internet experience for the >1 billion IPs that we see daily.

To make these capacity upgrades possible for our customers, efficient infrastructure deployment has been one of our keys to success. We want our infrastructure deployment to be targeted and flexible.

The next Cloudflare customer through our door could be a small restaurant owner on a Pro plan with thousands of monthly pageviews or a fast-growing global tech company like Discord. As a result, we need to always stay one step ahead and synthesize a lot of data all at once for our customers.

To accommodate this expansion, our Capacity Planning team is learning new ways to optimize our servers. One key strategy is targeting exactly where to send our servers. However, staying on top of everything isn’t easy - we are a global anycast network, which introduces unpredictability as to where incoming traffic goes. To make things even more difficult, each city can contain as many as five distinct deployments. Planning isn’t just a question of what city to send servers to, it’s one of which address.

To make sense of it all, we tackle the problem with simulations. Some, but not all, of the variables we model include historical traffic growth rates, foreseeable anomalous spikes (e.g., Cyber Day in Chile), and consumption states from our live deal pipeline, as well as product costs, user growth, end-customer adoption. We also add in site reliability, potential for expansion, and expected regional expansion and partnerships, as well as strategic priorities and, of course, feedback from our fantastic Systems Reliability Engineers.

Knowing where to send a server is only the first challenge of many when it comes to a global network. Just like our user base, our supply chain must span the entire world while also staying flexible enough to quickly react to time constraints, pricing changes including taxes and tariffs, import/export restrictions and required certifications - not to mention local partnerships many more dynamic location-specific variables. Even more reason we have to stay quick on our feet, there will always be unforeseen roadblocks and detours even in the most well-prepared plans. For example, a planned expansion in our Prague location might warrant an expanded presence in Vienna for failover.

Once servers arrive at our data centers, our Data Center Deployment and Technical Operations teams work with our vendors and on-site data center personnel (our “Remote Hands” and “Smart Hands”) to install the physical server, manage the cabling, and handle other early-stage provisioning processes.

Our architecture, which is designed so that every server can support every service, makes it easier to withstand hardware failures and efficiently load balance workloads between equipment and between locations.

If working at a rapidly expanding, globally diverse company interests you, we’re hiring for scores of positions, including in the Infrastructure group. If you want to help increase hardware efficiency, deploy and maintain servers, work on our supply chain, or strengthen ISP partnerships, get in touch.

*Represents cities where we have data centers with active Internet ports and where we are configuring our servers to handle traffic for more customers (at the time of publishing)

Cloudflare is excited to announce the addition of ten new data centers across the United States, Bahrain, Russia, Vietnam, Pakistan and France (Réunion). We're delighted to help improve the performance and security of over 12 million domains across these diverse countries that collectively represent about half a billion Internet users.

Our global network now spans 165 cities, with 46 new cities added just this year, and several dozen additional locations being actively worked on.

Our expansion begins in the United States, where Cloudflare's 36th and 37th data centers in the nation serve Charlotte (North Carolina) and Columbus (Ohio) respectively. They are promising markets for interconnection, and join our existing deployments in Ashburn, Atlanta, Boston, Chicago, Dallas, Denver, Detroit, Houston, Indianapolis, Jacksonville, Kansas City, Las Vegas, Los Angeles, McAllen, Memphis, Miami, Minneapolis, Montgomery, Nashville, Newark, Norfolk, Omaha, Philadelphia, Portland, Richmond, Sacramento, Salt Lake City, San Diego, San Jose, Seattle, St. Louis, Tallahassee, and Tampa.

Cloudflare's Manama (Bahrain) data center, our 158th globally, further expands our Middle East coverage. A growing hub for cloud computing, including public sector adoption (with the Kingdom's "Cloud First" policy), Bahrain is attracting talent and investment in innovative companies.

Cloudflare's new St. Petersburg deployment serves as a point of redundancy to our existing Moscow facility, while also expanding our surface area to withstand DDoS attacks and reducing latency for local Internet users. (Hint: If you live in Novosibirsk or other parts of Russia, stay tuned for upcoming Cloudflare deployments near you).

Hànội and Hồ Chí Minh City, the two most populated cities in Vietnam with an estimated population of 8 million and 9 million respectively, now host Cloudflare's 160th and 161st data center.

On November 19, 1997, the Internet officially became available in Vietnam. Since then, several telecommunication companies - including VNPT, FPT, Viettel, CMC, VDC, and NetNam - have played a critical role in integrating the use of Internet into the government systems, business environment, school facilities, and many other organizations. With our new data centers in place, we are delighted to help provide a faster and safer Internet experience.  

The world's sixth most populous country, Pakistan is a land of delicious food, breathtaking natural beauty, poetry and, of course cricket. Its natural beauty is exemplified by being the home of 5 out of 14 mountains which are at least 8,000m high, including K2, the second highest peak in the world. Pakistan's rich history includes the 5,000 year old lost civilization of Mohenjo-daro, with incredible design from complex architecture on a grid-layout to advanced water and sewage systems.

Nanga Parbat - Creative Commons Attribution-Share Alike 3.0 Unported

Today, Cloudflare is unveiling three new data centers housed in Pakistan, one in each of the most populous cities - Karachi and Lahore - alongside an additional data center in the capital city, Islamabad. We are already seeing latency per request decrease by over 3x and as much as 150ms, and expect this to further improve as we tune routing for all our customers.

Latency from PTCL to Cloudflare customers reduces by over 3x across Pakistan. Courtesy: Cedexis

8,000 miles away, the final stop in today's expansion is Sainte-Marie in the Réunion island, the overseas department France off the coast of Magadascar (which can also expect some Cloudflare servers very soon!)

Even beyond these, we are working on at least six new cities in each of Africa, Asia, Europe, North America, and South America. Guess 20 upcoming locations to receive Cloudflare swag.

Hot off the presses! Cloudflare just completed provisioning our Luxembourg City and Chișinău data centers, expanding our Europe network to 41 cities, and our global network to 151 cities across 74 countries. In the coming days, we'll ramp up traffic from across millions of websites using Cloudflare, and get routes optimized across all networks. Cloudflare is a participant at the Chișinău Internet Exchange (KIVIX), Luxembourg Commercial Internet eXchange (LU-CIX), and Moldova Internet Exchange (MD-IX), amongst ~180 other interconnection points.

This has been an exciting month, with 31 cities added just in March, for an average of one per day! Collectively, they provide additional resilience and performance across countries spanning a population of over one billion people. To recap, here's the list of our newest data centers: Beirut, Phnom Penh, Kathmandu, Istanbul, Reykjavík, Riyadh, Macau, Baghdad, Houston, Indianapolis, Montgomery, Pittsburgh, Sacramento, Mexico City, Tel Aviv, Durban, Port Louis, Cebu City, Edinburgh, Riga, Tallinn, Vilnius, Calgary, Saskatoon, Winnipeg, Jacksonville, Memphis, Tallahassee, Bogotá, Luxembourg and Chișinău!

We are very excited to surpass a milestone of 150 cities, or our sixth cohort of twenty-five cities. If the first five cohorts took 45 months, 15 months, 16 months, 8 months and 18 months, the most recent cohort was completed in 24 days.

As we evaluate new cities, our search begins by identifying cities (or ISPs) where we deliver a significant amount of traffic and/or can improve performance for a significant number of Internet users. Then, we assess the facilities we deploy at to ensure they can comply with a high standard for their information security practices, data center resilience and diversity of interconnection.

Our logistics team works to compile hardware from our the latest generation (which continues to evolve!) and manages the intricacies of shipping, often to a completely new city or country. Infrastructure Engineers work with our local partners to complete the physical install of hardware. Finally, entirely remotely, our engineering team provisions servers into service, while giving us the live analytics we need to ensure the integrity of our hardware, and supporting smooth operations. When a new city goes live, we often see a "bump up" in traffic as reduced latency facilitates increased page views and interconnection. From there on, our deployment grows, through the confluence of new customers signing up, new products being adopted, and increasingly, customers running their code at our edge - and we continue to invest in upgrades across our network.

Our aspiration for 2018 is for 95% of the world's Internet users to live in a country with a Cloudflare data center. (On a lighter note, I can't wait for us to turn up an upcoming data center beginning with the letter 'U', so that we can span every letter of the alphabet. See you soon, Mongolia!).

None of our deployments would be possible without the support of our many partners, team members and customers, who bring all of us closer to the promise of a better Internet.

This map reflects the network as of the publish date of this blog post. For the most up to date directory of locations please refer to our Network Map on the Cloudflare site.

Good things come in threes! Following the launch of three data centers each in the Baltics (Riga, Tallinn, Vilnius) and in the Canadian Prairies (Calgary, Saskatoon, Winnipeg), we're thrilled to announce three new data centers in the Southern United States!

Located in Jacksonville (Florida), Memphis (Tennessee), and Tallahassee (Florida), they represent the 146th, 147th and 148th cities across our growing global network, and our 40th, 41st and 42nd cities just in North America. They join existing Cloudflare facilities in the US, including other Florida / Tennessee deployments such as Miami, Tampa and Nashville. Just in March, we've added deployments in 28 new cities worldwide, which help reduce latency to millions of Internet properties using Cloudflare, while expanding our capacity to withstand new and familiar attacks.

Photo of Jacksonville Beach by Lance Asper / Unsplash

Whether you're doing the Memphis Main Street Crawl, experiencing history through a visit to Tallahassee's Mission San Luis de Apalachee, or just relaxing by the stunning beaches of Jacksonville, you'll be close to the nearest Cloudflare data center.

This map reflects the network as of the publish date of this blog post. For the most up to date directory of locations please refer to our Network Map on the Cloudflare site.

Cloudflare announces the turn up of our newest data centers located in Riga (Latvia), Tallinn (Estonia) and Vilnius (Lithuania). They represent the 140th, 141st and 142nd cities across our growing global network, and our 37th, 38th, 39th cities in Europe. We are very excited to help improve the security and performance of over 7 million Internet properties across 72 countries including the Baltic states.

We will be interconnecting with local networks over multiple Internet exchanges: Baltic Internet Exchange (BALT-IX), Lithuanian Internet eXchange Point (LIXP), LITIX, Tallinn Internet Exchange (TLLIX), Tallinn Governmental Internet Exchange (RTIX), Santa Monica Internet Local Exchange (SMILE-LV), and potentially, the Latvian Internet Exchange (LIX-LV).

If you are an entrepreneur anywhere in the world selling your product in these markets, or a Baltic entrepreneur reaching a global audience, we've got your back.

Photo by Siim Lukka / UnsplashLatvia, Estonia and Lithuania join the list of other countries with shorelines along the Baltic Sea and Cloudflare data centers. That list includes Denmark, Finland, Germany, Poland, Russia and Sweden.

Of the five countries that in the drainage basin but do not border the sea, Cloudflare has deployments located in three (Czech Republic, Norway, Ukraine) and is evaluating deployments in the remaining two.

CC BY 2.0 image by Kārlis Dambrāns

If you haven't already, consider visiting the Baltic States. Attractions include Estonia's over 2,000 islands (such as Saaremaa and Muhu), Lithuania's Trakai Island Castle (dating back to the 14th century!), Latvia's Art Nouveau district, and much more.

This map reflects the network as of the publish date of this blog post. For the most up to date directory of locations please refer to our Network Map on the Cloudflare site.

Our newest data centers in Durban (South Africa) and Port Louis (Mauritius) expand the Cloudflare network to 137 cities globally. We are delighted to reach this special milestone, and even more excited to help improve the performance and security of over 7 million Internet properties (and growing!) across 69 countries.

Just in March, so far, we've launched new data centers across Beirut, Phnom Penh, Kathmandu, Istanbul, Reykjavík, Riyadh, Macau, Baghdad, Houston, Indianapolis, Montgomery, Pittsburgh, Sacramento, Mexico City and Tel Aviv!

Just three years (and about 100 cities ago!), we launched our very first Africa deployment in Johannesburg (South Africa). It was an exciting day for members of our team to facilitate an especially substantial latency improvement for our customers.

Since then, we’ve turned up additional deployments in Cairo (Egypt), Cape Town (South Africa), Djibouti (Djibouti), Luanda (Angola), and Mombasa (Kenya).

Durban is our third deployment in South Africa, where mobile adoption continues to drive traffic growth amongst 20 million Internet users. Other countries with three (or more) Cloudflare data centers are Australia, Canada, China, Germany and United States (with two European states joining this list very soon!). In addition to better serving KwaZulu-Natal, our Durban data center will provide additional redundancy and expand the surface area for Cloudflare to withstand potential DDoS attacks. We are participants at two interconnection points: NAP Africa IX Durban and DINX (Durban Internet Exchange).

Photo by Jörg Angeli / Unsplash

Port Louis (Mauritius), which used to be served out of our Mombasa data center, will see the latency-equivalent of 1,500 miles shaved off on every web request. Mauritius welcomes 1.3 million visitors from world over to experience the idyllic island - as many visitors as the local population! Even if you’re traveling on vacation (and choose to go online!), we’d love to ensure that you access a Cloudflare data center close to you. The reduction in latency and improvement in availability result in higher throughput to visitors. We do already do more traffic to Port Louis today than we did from Johannesburg in late 2014.

Throughput to Mauritius Telecom visitors increases for millions of Cloudflare customers

Several additional Africa data centers are actively in the works, with many expected to go live in the second half of 2018. We are especially keen to partner with ISPs in Central and West Africa, including in countries such as Nigeria, where four of the ten most popular Internet properties are protected by Cloudflare.

Correctly guess at least five of the upcoming cities being turned up in March, and we’ll send you some Cloudflare gear.

This map reflects the network as of the publish date of this blog post. For the most up to date directory of locations please refer to our Network Map on the Cloudflare site.

When Cloudflare launched, three of the original five cities in our network - Chicago, Ashburn and San Jose - were located in the United States. Since then, we have grown the breadth of the global network considerably to span 66 countries, and even expanded the US footprint to twenty five locations. Even as a highly international business, the United States continues to be home to a number of our customers and the majority of Cloudflare employees.

Today, we expand our network in the United States even further by adding five new locations: Houston (Texas), Indianapolis (Indiana), Montgomery (Alabama), Pittsburgh (Pennsylvania) and Sacramento (California) as our 129th, 130th, 131st, 132nd and 133rd data centers respectively. They represent states that collectively span nearly 100 million people. In North America alone, the Cloudflare network now spans 37 cities, including thirty in the US.

In each of these new locations, we connect with at least one major local Internet service provider and also openly peer using at least one major Internet exchange. We are participants at CyrusOne IX Houston, Midwest IX Indianapolis, Montgomery Internet Exchange, Pittsburgh IX, and the upcoming Sacramento IX.

These deployments improves performance, security and reliability for our customers, even while expanding the edge (and the compute capability it enables). In the not too distant future, we’d like to deploy at cell towers across major metro markets (and beyond!) to support the next generation of 5G-enabled applications.

With the launch of our next data center, Cloudflare will have deployments located in all of the ten most populous North American metropolitan areas.

This map reflects the network as of the publish date of this blog post. For the most up to date directory of locations please refer to our Network Map on the Cloudflare site.

Cloudflare's newest data center is located in Baghdad, Iraq, in the region often known as the cradle of civilization. This expands our growing Middle East presence, while serving as our 45th data center in Asia, and 128th data center globally.

Even while accelerating over 7 million Internet properties, this deployment helps our effort to be closer to every Internet user. Previous, ISPs such as Earthlink were served from our Frankfurt data center. Nearly 40 million people live in Iraq.

One of the world's largest producers of the sweet date palm, Iraq's cuisine dates back over 10,000 years and includes favorites such as,

• Kleicha: Date-filled cookies flavored with cardamom, saffron and rose water

• Mezza: a selection of appetizers to begin the meal

• Iraqi Dolma: stuffed vegetables with a tangy sauce

• Iraqi Biryani: cooked rice with spices, beans, grilled nuts and meat / vegetables

• Masgouf: whole baked fish marinated in oil, salt, pepper, turmeric and tamarind

Baghdad is the first of eight deployments joining the Cloudflare global network just this week. Stay tuned!

This map reflects the network as of the publish date of this blog post. For the most up to date directory of locations please refer to our Network Map on the Cloudflare site.

We are very excited to announce Cloudflare’s 126th data center in Riyadh, Saudi Arabia (only hours after launching in Reykjavík!). This joins our existing Middle East facilities to provide even stronger coverage and resilience for over 7 million Internet properties across the region.

Our newest deployment was made possible in partnership with Zain, which now experiences reduced latency for every Internet user accessing every Internet facing application using Cloudflare. At least four additional Middle East deployments are already in the works.

Photo by Mohammed Alamri / Unsplash

Over 30 million people live in Saudi Arabia, which is also the 13th largest country by area at over 830,000 square miles. In 2020, alongside the launch of entirely new “economic cities”, we might witness the opening of the world’s tallest skyscraper at a staggering 1,000m height, located in Jeddah. More modestly, but in much less than two years from now, we also expect to place a Cloudflare data center there.

Saudi Arabia has an incredibly young demographic, as over half of the population is less than 25. Additional 4G LTE deployments, while also paving the way for 5G, should drive increased Internet usage.

Stay tuned as we continue to add new deployments in the Middle East, and around the world.

Cloudflare is excited to turn up our newest data center in Phnom Penh, Cambodia, making over 7 million Internet properties even faster. This is our 122nd data center globally, and our 41st data center in Asia. By the end of 2018, we expect that 95% of the world's population will live in a country with a Cloudflare data center, as we grow our global network to span 200 cities.

Home to over 16 million people, Cambodia has a relatively low base of Internet penetration (~25%) today, but is seeing an increasing number of Internet users coming online. For perspective, Cambodia has approximately the same number of Internet users as Lebanon (where we just turned up our 121st data center!) or Singapore (from where we used to serve a portion of Cambodian visitors).

Photo by Florian Hahn / Unsplash

In the coming weeks, we’ll further optimize our routing for Cloudflare customers and expect to see a growing number of ISPs pick up our customers’ traffic on a low latency path.

Latency from a Cambodian ISP (SINET) to Cloudflare customers decreases 10x

Next up, in fact, thousands of feet further up, we head to the mountains for Cloudflare’s 123rd data center. Following that, two upcoming Cloudflare data centers are located well south of the Equator, and a continent away.

Fire the Gric Cannon! Hot on the heels of several birthday week product announcements, we continue to expand our global network.

Cloudflare is excited to announce the launch of our newest data center in Zagreb, Croatia, furthering the breadth of our network to 118 cities across 58 countries. Our Europe network alone now spans 33 cities across 25 countries (with at least ten new cities being planned). [For trivia fans: Our list of data centers beginning with the letter Z now spans four cities, with the others being Zhengzhou, Zhuzhou and Zurich].

_CC BY 2.0 image by Mario Fajt, sobrecroacia.com_

With a rich history going back almost a thousand years, Zagreb is sometimes called the City of Museums. Visitors can join the Saturday spica to Dolac market to try out the traditional paprenjak biscuit, hop on the shortest cable car in the world, explore Maksimir Park and more in this charming European city.

Croatia is home to over 3 million Internet users, with Internet penetration approaching 75%, which is high but still significantly lower than the European average of ~85%. Our newest deployment improves the security and performance of over 6 million Internet properties using Cloudflare for visitors from across the country (plus: many others in the region).

As we have explained before, a significant portion of Eastern Europe interconnection happens in Frankfurt or Amsterdam, over 1,000km away. Multiplied by billions of requests, this represents latency we are committed to help reduce as we get closer to our goal of being within 10ms of 99% of the world’s population.

If you happen to live in the third city which is a national capital by the Sava river, much like Belgrade and Zagreb, stay tuned. We might have servers headed your way soon! Se vidimo kmalu!

Even as the luckiest amongst us across the US West Coast dashed off to Oregon to be closer to the solar eclipse path of totality, Cloudflare engineers were busy turning up our newest data center in Portland.

This deployment serves as our 27th data center in North America alone, and our 117th globally. It also provides additional redundancy to our Seattle and San Jose data centers, while increasing our capacity to run services and fight growing attacks.

The Silicon Forest corridor around Portland holds a special place in the hearts of the Cloudflare team. It is both new (by way of our latest edge deployment bringing us closer to millions of Oregon Internet users), and familiar (since we have had an internal data center in this region for many years, enabling services such as analytics and enterprise logs).

The greater Portland area has played a pivotal area in building high-technology products from companies such as Tektronix (test and measurement equipment), Mentor Graphics (electronic design automation) and Intel (with nearly 20,000 employees across Oregon).

At our Portland data center, we locally interconnect with ISPs such as Comcast (into their regional area networks such as Beaverton and Troutdale), and also to the Northwest Access Exchange (a volunteer non-profit internet exchange fostering peering).

This is part of a much broader expansion, as we are actively working on at least a dozen additional data centers within North America alone, and over 50 new cities around the world. If managing the moving parts of building a large global network sounds interesting, come join us!

A Cloudflare está muito feliz de anunciar o nosso mais recente data center: Rio de Janeiro, Brasil. Este é o nosso oitavo data center na América do Sul, e com ele a rede da Cloudflare se expande por 116 cidades em 57 países. Este lançamento vai acelerar e proteger mais de seis milhões de sites e aplicações web pelo Brasil, também provendo redundância para o nosso data center em São Paulo. Provendo acesso à nossa rede para mais parceiros através do Ponto de Troca de Tráfego (IX-RJ), nós estamos chegando mais perto dos usuários da Internet em todo o Brasil.

Rio de Janeiro plays a great role in the history of Internet in Brazil. In 1988, the National Laboratory of Scientific Computation, headquartered in Rio de Janeiro connected to the University of Maryland via Bitnet, a network to exchange messages. The next year, the Federal University of Rio de Janeiro also connected to Bitnet, becoming the third institution (with São Paulo State Foundation for Research Support) to have access to this technology.

O Rio de janeiro tem papel central na história da Internet no Brasil. Em 1988, o Laboratório Nacional de Computação Científica (LNCC), conectou-se à Universidade de Mariland através da Bitnet, que era uma rede que permitia o envio de e-mail entre as instituições acadêmicas. Em 1989, a Universidade Federal do Rio de Janeiro também se conectou na Bitnet através de outra universidade americana, se tornando a terceira instituição Brasileira a se conectar na Internet (a FAPESP também já estava na rede).

_CC BY-NC 2.0 image by Lau Rey_

Today, the city of Rio de Janeiro is very well connected. Internet access can be found all over, and better connectivity can boost entrepreneurship. In some Favelas (slums), the residents are creating their own ISPs, providing Internet access to some users that big ISPs are not able to reach.

Hoje, a cidade do Rio de Janeiro é muito bem conectada. Acesso à internet pode ser encontrado em todo lugar, inclusive incentivando o empreendedorismo. Em algumas favelas os próprios moradores criaram seus provedores de internet via Wi-Fi, e estão proporcionando a inclusão digital em áreas onde os grandes provedores não chegam.

We have an additional eight datacenters in progress across Latin America. If managing the many moving parts of building a large global network interest you, come join our team!

Nós temos mais oito datacenters a caminho na América Latina. Se você se interessa em gerenciar uma rede de alcance global, venha fazer parte do nosso time!

-The Cloudflare team