There’s an important debate happening in Europe that could affect the future of the Internet. The European Commission is considering new rules for how networks connect to each other on the Internet. It’s considering proposals that – no hyperbole – will slow the Internet for consumers and are dangerous for the Internet.

The large incumbent telcos are complaining loudly to anyone who wants to listen that they aren’t being adequately compensated for the capital investments they’re making. These telcos are a set of previously regulated monopolies who still constitute the largest telcos by revenue in Europe in today's competitive market. They say traffic volumes, largely due to video streaming, are growing rapidly, implying they need to make capital investments to keep up. And they call for new charges on big US tech companies: a “fair share” contribution that those networks should make to European Internet infrastructure investment.

In response to this campaign, in February the European Commission released a set of recommended actions and proposals “aimed to make Gigabit connectivity available to all citizens and businesses across the EU by 2030.” The Commission goes on to say that “Reliable, fast and secure connectivity is a must for everybody and everywhere in the Union, including in rural and remote areas.” While this goal is certainly the right one, our agreement with the European Commission’s approach, unfortunately, ends right there. A close reading of the Commission’s exploratory consultation that accompanies the Gigabit connectivity proposals shows that the ultimate goal is to intervene in the market for how networks interconnect, with the intention to extract fees from large tech companies and funnel them to large incumbent telcos.

This debate has been characterised as a fight between Big Tech and Big European Telco. But it’s about much more than that. Contrary to its intent, these proposals would give the biggest technology companies preferred access to the largest European ISPs. European consumers and small businesses, when accessing anything on the Internet outside Big Tech (Netflix, Google, Meta, Amazon, etc), would get the slow lane. Below we’ll explain why Cloudflare, although we are not currently targeted for extra fees, still feels strongly that these fees are dangerous for the Internet:

• Network usage fees would create fast lanes for Big Tech content, and slow lanes for everything else, slowing the Internet for European consumers;

• Small businesses, Internet startups, and consumers are the beneficiaries of Europe’s low wholesale bandwidth prices. Regulatory intervention in this market would lead to higher prices that would be passed onto SMEs and consumers;

• The Internet works best – fastest and most reliably – when networks connect freely and frequently, bringing content and service as close to consumers as possible. Network usage fees artificially disincentivize efforts to bring content close to users, making the Internet experience worse for consumers.

Understanding why the debate in Europe matters for the future of the Internet requires understanding how Internet traffic gets to end users, as well as the steps that can be taken to improve Internet performance.

At Cloudflare, we know a lot about this. According to Hurricane Electric, Cloudflare connects with other networks at 287 Internet exchange points (IXPs), the second most of any network on the planet. And we’re directly connected to other networks on the Internet in more than 285 cities in over 100 countries. So when we see a proposal to change how networks interconnect, we take notice. What the European Commission is considering might appear to be targeting the direct relationship between telcos and large tech companies, but we know it will have much broader effects.

There are different ways in which networks exchange data on the Internet. In some cases, networks connect directly to exchange data between users of each network. This is called peering. Cloudflare has an open peering policy; we’ll peer with any other network. Peering is one hop between networks – it’s the gold standard. Fewer hops from start to end generally means faster and more reliable data delivery. We peer with more than 12,000 networks around the world on a settlement-free basis, which means neither network pays the other to send traffic. This settlement-free peering is one of the aspects of Cloudflare’s business that allows us to offer a free version of our services to millions of users globally, permitting individuals and small businesses to have websites that load quickly and efficiently and are better protected from cyberattacks. We’ll talk more about the benefits of settlement-free peering below.

Figure 1: Traffic takes one of three paths between an end-user’s ISP and the content or service they are trying to access. Traffic could go over direct peering which is 1:1 between the ISP and the content or service provider; it could go through IX Peering which is a many:many connection between networks; or it could go via a transit provider, which is a network that gets compensated for delivering traffic anywhere on the Internet.

When networks don’t connect directly, they might pay a third-party IP transit network to deliver traffic on their behalf. No network is connected to every other network on the Internet, so transit networks play an important role making sure any network can reach any other network. They’re compensated for doing so; generally a network will pay their transit provider based on how much traffic they ask the transit provider to deliver. Cloudflare is connected to more than 12,000 other networks, but there are over 100,000 Autonomous Systems (networks) on the Internet, so we use transit networks to reach the “long tail”. For example, the Cloudflare network (AS 13335) provides the website cloudflare.com to any network that requests it. If a user of a small ISP with whom Cloudflare doesn’t have direct connections requests cloudflare.com from their browser, it’s likely that their ISP will use a transit provider to send that request to Cloudflare. Then Cloudflare would respond to the request, sending the website content back to the user via a transit provider.

In Europe, transit providers play a critical role because many of the largest incumbent telcos won’t do settlement-free direct peering connections. Therefore, many European consumers that use large incumbent telcos for their Internet service interact with Cloudflare’s services through third party transit networks. It isn’t the gold standard of network interconnection (which is peering, and would be faster and more reliable) but it works well enough most of the time.

Cloudflare would of course be happy to directly connect with EU telcos because we have an open peering policy. As we’ll show, the performance and reliability improvement for their subscribers and our customers’ content and services would significantly improve. And if the telcos offered us transit – the ability to send traffic to their network and onwards to the Internet – at market rates, we would consider use of that service as part of competitive supplier selection. While it’s unfortunate that incumbent telcos haven’t offered services at market-competitive prices, overall the interconnection market in Europe – indeed the Internet itself – currently works well. Others agree. BEREC, the body of European telecommunications regulators, wrote recently in a preliminary assessment:

BEREC's experience shows that the internet has proven its ability to cope with increasing traffic volumes, changes in demand patterns, technology, business models, as well as in the (relative) market power between market players. These developments are reflected in the IP interconnection mechanisms governing the internet which evolved without a need for regulatory intervention. The internet’s ability to self-adapt has been and still is essential for its success and its innovative capability.

There is a competitive market for IP transit. According to market analysis firm Telegeography’s State of the Network 2023 report, “The lowest [prices on offer for] 100 GigE [IP transit services in Europe] were $0.06 per Mbps per month.” These prices are consistent with what Cloudflare sees in the market. In our view, the Commission should be proud of the effective competition in this market, and it should protect it. These prices are comparable to IP transit prices in the United States and signal, overall, a healthy Internet ecosystem. Competitive wholesale bandwidth prices (transit prices) mean it is easier for small independent telcos to enter the market, and lower prices for all types of Internet applications and services. In our view, regulatory intervention in this well-functioning market has significant down-side risks.

Large incumbent telcos are seeking regulatory intervention in part because they are not willing to accept the fair market prices for transit. Very Large Telcos and Content and Application Providers (CAPs) – the term the European Commission uses for networks that have the content and services consumers want to see – negotiate freely for transit and peering. In our experience, large incumbent telcos ask for paid peering fees that are many multiples of what a CAP could pay to transit networks for a similar service. At the prices offered, many networks – including Cloudflare – continue to use transit providers instead of paying incumbent telcos for peering. Telcos are trying to use regulation to force CAPs into these relationships at artificially high prices.

If the Commission’s proposal is adopted, the price for interconnection in Europe would likely be set by this regulation, not the market. Once there’s a price for interconnection between CAPs and telcos, whether that price is found via negotiation, or more likely arbitrators set the price, that is likely to become the de facto price for all interconnection. After all, if telcos can achieve artificially high prices from the largest CAPs, why would they accept much lower rates from any other network – including transits – to connect with them? Instead of falling wholesale prices spurring Internet innovation as is happening now in Europe and the United States, rising wholesale prices will be passed onto small businesses and consumers.

If network fees become a reality, the current Internet experience for users in Europe will deteriorate. Notwithstanding existing net neutrality regulations, we already see large telcos relegate content from transit providers to more congested connections. If the biggest CAPs pay for interconnection, consumer traffic to other networks will be relegated to a slow and/or congested lane. Networks that aren’t paying would still use transit providers to reach the large incumbent telcos, but those transit links would be second class citizens to the paid traffic. Existing transit links will become (more) slow and congested. By targeting only the largest CAPs, a proposal based on network fees would perversely, and contrary to intent, cement those CAPs’ position at the top by improving the consumer experience for those networks at the expense of all others. By mandating that the CAPs pay the large incumbent telcos for peering, the European Commission would therefore be facilitating discrimination against services using smaller networks and organisations that cannot match the resources of the large CAPs.

Indeed, we already see evidence that some of the large incumbent telcos treat transit networks as second-class citizens when it comes to Internet traffic. In November 2022, HWSW, a Hungarian tech news site, reported on recurring Internet problems for users of Magyar Telekom, a subsidiary of Deutsche Telekom, because of congestion between Deutsche Telekom and its transit networks:

Network problem that exists during the fairly well-defined period, mostly between 4 p.m. and midnight Hungarian time, … due to congestion in the connection (Level3) between Deutsche Telekom, the parent company that operates Magyar Telekom's international peering routes, and Cloudflare, therefore it does not only affect Hungarian subscribers, but occurs to a greater or lesser extent at all DT subsidiaries that, like Magyar Telekom, are linked to the parent company. (translated by Google Translate)

Going back many years, large telcos have demonstrated that traffic reaching them through transit networks is not a high priority to maintain quality. In 2015, Cogent, a transit provider, sued Deutsche Telekom over interconnection, writing, “Deutsche Telekom has interfered with the free flow of internet traffic between Cogent customers and Deutsche Telekom customers by refusing to increase the capacity of the interconnection ports that allow the exchange of traffic”.

Beyond the effect on consumers, the implementation of Network Usage Fees would seem to violate the European Union’s Open Internet Regulation, sometimes referred to as the net neutrality provision. Article 3(3) of the Open Internet Regulation states:

Providers of internet access services shall treat all traffic equally, when providing internet access services, without discrimination, restriction or interference, and irrespective of the sender and receiver, the content accessed or distributed, the applications or services used or provided, or the terminal equipment used. (emphasis added)

Fees from certain sources of content in exchange for private paths between the CAP and large incumbent telcos would seem to be a plain-language violation of this provision.

Let’s now talk about the ecosystem that leads to a thriving Internet. We first talked about transit, now we’ll move on to peering, which is quietly central to how the Internet works. “Peering” is the practice of two networks directly interconnecting (they could be backbones, CDNs, mobile networks or broadband telcos to exchange traffic. Almost always, networks peer without any payments (“settlement-free”) in recognition of the performance benefits and resiliency we’re about to discuss. A recent survey of over 10,000 ISPs shows that 99.99% of their exchanged traffic is on settlement-free terms. The Internet works best when these peering arrangements happen freely and frequently.

These types of peering arrangements and network interconnection also significantly improve latency for the end-user of services delivered via the Internet. The speed of an Internet connection depends more on latency (the time it takes for a consumer to request data and receive the response) than on bandwidth (the maximum amount of data that is flowing at any one time over a connection). Latency is critical to many Internet use-cases. A recent technical paper used the example of a mapping application that responds to user scrolling. The application wouldn’t need to pre-load unnecessary data if it can quickly get a small amount of data in response to a user swiping in a certain direction.

In recognition of the myriad benefits, settlement-free peering between CDNs and terminating ISPs is the global norm in the industry. Most networks understand that through settlement-free peering, (1) customers get the best experience through local traffic delivery, (2) networks have increased resilience through multiple traffic paths, and (3) data is exchanged locally instead of backhauled and aggregated in larger volumes at regional Internet hubs. By contrast, paid peering is rare, and is usually employed by networks that operate in markets without robust competition. Unfortunately, when an incumbent telco achieves a dominant market position or has no significant competition, they may be less concerned about the performance penalty they impose on their own users by refusing to peer directly.

As an example, consider the map in Figure 2. This map shows the situation in Germany, where most traffic is exchanged via transit providers at the Internet hub in Frankfurt. Consumers are losing in this situation for two reasons: First, the farther they are from Frankfurt, the higher latency they will experience for Cloudflare services. For customers in northeast Germany, for example, the distance from Cloudflare’s servers in Frankfurt means they will experience nearly double the latency of consumers closer to Cloudflare geographically. Second, the reliance on a small number of transit providers exposes their traffic to congestion and reliability risks. The remedy is obvious: if large telcos would interconnect (“peer”) with Cloudflare in all five cities where Cloudflare has points of presence, every consumer, regardless of where they are in Germany, would have the same excellent Internet experience.

We’ve shown that local settlement-free interconnection benefits consumers by improving the speed of their Internet experience, but local interconnection also reduces the amount of traffic that aggregates at regional Internet hubs. If a telco interconnects with a large video provider in a single regional hub, the telco needs to carry their subscribers’ request for content through their network to the hub. Data will be exchanged at the hub, then the telco needs to carry the data back through their “backbone” network to the subscriber. (While this situation can result in large traffic volumes, modern networks can easily expand the capacity between themselves at almost no cost by adding additional port capacity. The fibre-optic cable capacity in this “backbone” part of the Internet is not constrained.)

Figure 3. A hypothetical example where a telco only interconnects with a video provider at a regional Internet hub, showing how traffic aggregates at the interconnection point.

Local settlement-free peering is one way to reduce the traffic across those interconnection points. Another way is to use embedded caches, which are offered by most CDNs, including Cloudflare. In this scenario, a CDN sends hardware to the telco, which installs it in their network at local aggregation points that are private to the telco. When their subscriber requests data from the CDN, the telco can find that content at a local infrastructure point and send it back to the subscriber. The data doesn’t need to aggregate on backhaul links, or ever reach a regional Internet hub. This approach is common. Cloudflare has hundreds of these deployments with telcos globally.

Figure 4. A hypothetical example where a telco has deployed embedded caches from a video provider, removing the backhaul and aggregation of traffic across Internet exchange points

In conclusion, it’s our view that despite the unwillingness of many large European incumbents to peer on a settlement-free basis, the IP interconnection market is healthy, which benefits European consumers. We believe regulatory intervention that forces content and application providers into paid peering agreements would have the effect of relegating all other traffic to a slow, congested lane. Further, we fear this intervention will do nothing to meet Europe’s Digital Decade goals, and instead will make the Internet experience worse for consumers and small businesses.

There are many more companies, NGOs and politicians that have raised concerns about the impact of introducing network usage fees in Europe. A number of stakeholders have spoken out already about the dangers of regulating the Internet interconnection system; from digital rights groups to the Internet Society, European Video on Demand providers and commercial broadcasters, Internet Exchanges and mobile operators to several European governments and Members of the European Parliament.

If you agree that major intervention in how networks interconnect in Europe is unnecessary, and even harmful, consider reading more about the European Commission’s consultation. While the consultation itself may look intimidating, anyone can submit a narrative response (deadline: 19 May). Consider telling the European Commission that their goals of ubiquitous connectivity are the right ones but that the approach they are considering is going into the wrong direction.

In 2018, we launched the Cloudflare Peering Portal, which allows network operators to see where your traffic is coming from and to identify the best possible places to interconnect with Cloudflare. We’re excited to announce that we’ve made it even easier to interconnect with Cloudflare through this portal by removing Cloudflare-specific logins and allowing users to request sessions in the portal itself!

We’re going to walk through the changes we’ve made to make peering easier, but before we do that, let’s talk a little about peering: what it is, why it’s important, and how Cloudflare is making peering easier.

Put succinctly, peering is the act of connecting two networks together. If networks are like towns, peering is the bridges, highways, and streets that connect the networks together. There are lots of different ways to connect networks together, but when networks connect, traffic between them flows to their destination faster. The reason for this is that peering reduces the number of Border Gateway Protocol (BGP) hops between networks.

For a quick refresher, Border Gateway Protocol (or BGP for short) is a protocol that propagates instructions on how networks should forward packets so that traffic can get from its origin to its destination. BGP provides packets instructions on how to get from one network to another by indicating which networks the packets need to go through to get to the destination, prioritizing the paths with the smallest number of hops between origin and destination. BGP sees networks as Autonomous Systems (AS), and each AS has its own number. For example, Cloudflare’s ASN is 13335.

In the example below, AS 1 is trying to send packets to AS 3, but there are two possible paths the packets can go:

The BGP decision algorithm will select the path with the least number of hops, meaning that the path the packets will take is AS 1 → AS 2 → AS 3.

When two networks peer with each other, the number of networks needed to connect AS 1 and AS 3 is reduced to one, because AS 1 and AS 3 are directly connected with each other. But “connecting with” another network can be kind of vague, so let’s be more specific. In general, there are three ways that networks can connect with other networks: directly through Private Network Interconnects (PNI), at Internet Exchanges (IX), and through transit networks that connect with many networks.

A private network interconnect (PNI) sounds complicated, but it’s really simple at its core: it’s a cable connecting two networks to each other. If networks are in the same datacenter facility, it’s often easy for these two networks to connect and by doing so over a private connection, they can get dedicated bandwidth to each other as well as reliable uptime. Cloudflare has a product called Cloudflare Network Interconnect (CNI) that allows other networks to directly connect their networks to Cloudflare in this way.

An Internet exchange (IX) is a building that specifically houses many networks in the same place. Each network gets one or more ports, and plugs into what is essentially a shared switch so that every network has the potential to interconnect. These switches are massive and house hundreds, even thousands of networks. This is similar to a PNI, but instead of two networks directly connecting to each other, thousands of networks connect to the same device and establish BGP sessions through that device.

Fabienne Serriere: An optical patch panel of the AMS-IX Internet exchange point, in Amsterdam, Netherlands. CC BY-SA 3.0

At Internet Exchanges, traffic is generally exchanged between networks free of charge, and it’s a great way to interconnect a network with other networks and save money on bandwidth between networks.

Transit networks are networks that are specifically designed to carry packets between other networks. These networks peer at Internet Exchanges and directly with many other networks to provide connectivity for your network without having to get PNIs or IX presence with networks. This service comes at a price, and may impact network performance as the transit network is an intermediary hop between your network and the place your traffic is trying to reach. Transit networks aren’t peering, but they do peering on your behalf.

No matter how you may decide to connect your network to Cloudflare, we have an open peering policy, and strongly encourage you to connect your networks directly to Cloudflare. If you’re interested, you can get started by going through the Cloudflare Peering Portal, which has now been made even easier. But let’s take a second to talk about why peering is so important.

Peering is important on the Internet for three reasons: it distributes traffic across many networks reducing single points of failure of the Internet, it often reduces bandwidth prices on the Internet making overall costs lower, and it improves performance by removing network hops. Let's talk about each of those benefits.

Peering improves the overall uptime of the Internet by distributing traffic across multiple networks, meaning that if one network goes down, traffic from your network will still be able to reach your users. Compare that to connecting to a transit network: if the transit network has an issue, your network will be unreachable because that network was the only thing connecting your network to the rest of the Internet (unless you decide to pay multiple transit providers). With peering, any individual network failure will not completely impact the ability for your users to reach your network.

Peering helps reduce your network bandwidth costs because it distributes the cost you pay an Internet Exchange for a port across all the networks you interconnect with at the IX. If you're paying \$1000/month for a port at an IX, and you're peered with 100 networks there, you're effectively paying \$10/network, as opposed to paying \$1000/month to connect to one transit network. Furthermore, many networks including Cloudflare have open peering policies and settlement free peering, which means we don't charge you to send traffic to us or the other way round, making peering even more economical.

Peering also improves performance for Internet traffic by bringing networks closer together, reducing the time it takes for a packet to go from one network to another. The more two networks peer with each other, the more physical places on the planet they can exchange traffic directly, meaning that users everywhere see better performance.

Here’s an example. Janine is trying to order food from Acme Food Services, a site protected by Cloudflare. She lives in Boston and connects to Cloudflare via her ISP. Acme Food Services has their origin in Boston as well, so for Janine to see the fastest performance, her ISP should connect to Cloudflare in Boston and then Cloudflare should route her traffic directly to the Acme origin in Boston. Unfortunately for Janine, her ISP doesn’t peer with Cloudflare in Boston, but instead peers with Cloudflare in New York: meaning that when Janine connects to Acme, her traffic is going through her ISP to New York before it reaches Cloudflare, and then all the way back to Boston to the Acme origins!

But with proper peering, we can ensure that traffic is routed over the fastest possible path to ensure Janine connects to Cloudflare in Boston and everything stays local:

Fortunately for Janine, Cloudflare peers with over 10,000 networks in the world in over 275 locations, so high latency on the network is rare. And every time a new network peers with us, we help make user traffic even faster. So now let’s talk about how we’ve made peering even easier.

Cloudflare, along with many other networks, rely on PeeringDB as a source of truth for which networks are present on the Internet. PeeringDB is a community-maintained database of all the networks that are present on the Internet and what datacenter facilities and IXs they are present at, as well as what IPs are used for peering at each public location. Many networks, including Cloudflare, require you to have an account on PeeringDB before you can initiate a peering session with their network.

You can now use that same PeeringDB account to log into the Cloudflare Peering Portal directly, saving you the need to make a specific Cloudflare Peering Portal account.

When you log into the Cloudflare Peering Portal, simply click on the PeeringDB login button and enter your PeeringDB credentials. Cloudflare will then use this login information to determine what networks you are responsible for and automatically load data for those networks.

From here you can see all the places your network exchanges traffic with Cloudflare. You can see all the places you currently have direct peering with us, as well as locations for potential peering: places you could peer with us but currently don’t. Wouldn’t it be great if you could just click a button and configure a peering session with Cloudflare directly from that view? Well now you can!

Starting today, you can now request peering sessions with Cloudflare at Internet Exchanges right from the peering portal, making it even easier to get connected with Cloudflare. When you’re looking at potential peering sessions in the portal, you’ll now see a button that will allow you to verify your peering information is correct, and if it is to proceed with a peering request:

Once you click that button, a ticket will go immediately to our network team to configure a peering session with you using the details already provided in PeeringDB. Our network team looks at whether we already have existing connections with your network at that location, and also what the impact to your Internet traffic will be if we peer with you there. Once we’ve evaluated these variables, we’ll proceed to establish a BGP session with you at the location and inform you via email that you’ve already provided via PeeringDB. Then all you have to do is accept the BGP sessions, and you’ll be exchanging traffic with Cloudflare!

It has never been easier to peer with Cloudflare, and our simplified peering portal will make it even easier to get connected. Visit our peering portal today and get started on the path of faster, cheaper connectivity to Cloudflare!

Our logs show that at 21:12 UTC on Monday, a Nigerian ISP, MainOne, accidentally misconfigured part of their network causing a "route leak". This resulted in Google and a number of other networks being routed over unusual network paths. Incidents like this actually happen quite frequently, but in this case, the traffic flows generated by Google users were so great that they overwhelmed the intermediary networks — resulting in numerous services (but predominantly Google) unreachable.

You might be surprised to learn that an error by an ISP somewhere in the world could result in Google and other services going offline. This blog post explains how that can happen and what the Internet community is doing to try to fix this fragility.

When traffic is routed outside of regular and optimal routing paths, this is known as a “route leak”. An explanation of how they happen requires a little bit more context.

Every network and network provider on the Internet has their own Autonomous System (AS) number. This number is unique and indicates the part of the Internet that that organization controls. Of note for the following explanation Google’s primary AS Number is 15169. That's Google's corner of the Internet and where Google traffic should end up... by the fastest path.

A Typical view of how Google/AS15169’s routes are propagated to Tier-1 Networks.

As seen above, Google is directly connected to most of the Tier-1 networks (the largest networks link large swathes of the Internet). When everything is working as it should be, Google’s AS Path, the route packets take from network to network to reach their destination, is actually very simple. For example, in the diagram above, if you were a customer of Cogent and you were trying to get to Google, the AS Path that you would see is “174 6453 15169”. That string of numbers is like a sequence of waypoints: start on AS 174 (Cogent), go to Tata (AS 6453), then go to Google (AS 15169). So, Cogent subscribers reach Google via Tata, a huge Tier-1 provider.

During the incident, MainOne misconfigured their routing as reflected in the AS Path : “20485 4809 37282 15169”. As a result of this misconfiguration, any networks that MainOne peered with (i.e. were directly connected to) potentially had their routes leaked through this erroneous path. For example, the Cogent customer in the paragraph above (AS 174) wouldn’t have gone via Tata (AS 6453) as they should have. Instead, they were routed first through TransTelecom (a Russian Carrier, AS 20485), then to China Telecom CN2 (a cross border Chinese carrier, AS 4809), then on to MainOne (the Nigerian ISP that misconfigured everything, AS 37282), and only then were they finally handed off to Google (AS 15169). In other words,  a user in London could have had their traffic go from Russia to China to Nigeria — and only then got to Google.

The root cause of this was MainOne misconfiguring their routing. As mentioned earlier, incidents like this actually happen quite frequently. The impact of this misconfiguration should have been limited to MainOne and its customers.

However, what took this from relatively isolated and turned it into a much broader one is because CN2 — China Telecom’s premium cross-border carrier — was not filtering the routing that MainOne provided to them. In other words, MainOne told CN2 that it had authority to route Google’s IP addresses. Most networks verify this, and if it is incorrect, filter it out. CN2 did not — it simply trusted MainOne. As a result of this, MainOne’s misconfiguration propagated to a substantially larger network. Compounding this, it is likely that the Russian network TransTelecom behaved similarly towards CN2 as CN2 had behaved towards MainOne — they trusted without any verification of the routing paths that CN2 gave to them.

This demonstrates how much trust is involved in the underlying connections that make up the Internet. It's a network of networks (an internet!) that works by cooperation between different entities (countries and companies).

This is how a routing mistake made in Nigeria then propagated through China and then through Russia. Given the amount of traffic involved, the networks were overwhelmed and Google was unreachable.

It is worth explicitly stating: the fact that Google traffic was routed through Russia and China before going getting to Nigeria and only then hitting the correct destination made it appear to some people that the misconfiguration was nefarious. We do not believe this to be the case. Instead, this incident reflects a mistake that was not caught by appropriate network filtering. There was too much trust and not enough verification across a number of networks: this is a systemic problem that makes the Internet more vulnerable to mistakes than it should be.

Along with Cloudflare’s internal systems, BGPMon.net and ThousandEyes detected the incident. BGPMon has several methods to detect abnormalities; in this case, they were able to detect that the providers in the paths to reach Google were irregular.

Cloudflare’s systems immediately detected this and took auto-remediation action.

Thankfully our systems detected it and mitigated it! pic.twitter.com/qFiDkrn2Kd

— Jerome Fleury (@Jerome_UZ) November 13, 2018

Some more information about Cloudflare’s Auto Remediation system: https://blog.cloudflare.com/the-internet-is-hostile-building-a-more-resilient-network/

Cloudflare is committed to working with others to drive more secure and stable routing. We recently wrote about RPKI and how we’ll start enforcing “Strict” RPKI validation, and will continue to strive for secure internet routing.  We hope that all networks providing transit services can ensure proper filtering of their customers, end hijacks and route-leaks and implement best practices like BCP-38.

It can be a big deal for Internet users when Cloudflare rolls into town. After our recent Mongolia launch, we received lots of feedback from happy customers that all of a sudden, Internet performance noticeably improved.

As a result, it's not a surprising that we regularly receive requests from all over the world to either peer with our network, or to host a node. However, potential partners are always keen to know just how much traffic will be served over that link. What performance benefits can end-users expect? How much upstream traffic will the ISP save? What new bandwidth will they have available for traffic management?

Starting today, ISPs and hosting providers can request a login to the Cloudflare Peering Portal to find the answers to these questions. After validating ownership of your ASN, the Cloudflare network team will provide a login to the newly launched Peering Portal - Beta. You can find more information at: cloudflare.com/partners/peering-portal/

If you're new to the core infrastructure of the Internet, the best way to understand peering is to frame the problems it solves:

1. Bandwidth costs money

2. Internet users don't like slow websites

3. Network operators have limited resources

Consider what happens if you request a site hosted on Cloudflare from home:

• The domain resolves to a Cloudflare IP

• Your browser sends an HTTP request to that IP

• Your ISP's routers consult their routing table and route the packets upstream to their transit provider (COSTS MONEY)

• The packets traverse multiple hops to the nearest Cloudflare POP (TAKES TIME)

• Those packets are taking up some of the ISPs capacity, making it unavailable for other competing packets (TRAFFIC MANAGEMENT)

As Cloudflare continues to grow, we represent an ever-increasing share of the Internet's traffic. A Network Administrator reviewing his or her network, will see this represented as an increasing cost of bandwidth, of slower than optimal request times and of bandwidth not able to be allocated for other competing traffic.

To address these problems, large networks, hosting providers and CDNs will peer with each other. That is, to interconnect their networks directly. Put another way, cut out the middleman. If I request a site on Cloudflare and my ISP is already directly connected to the Cloudflare network, the packet will traverse fewer hops, my website will load faster, my ISP will pay less to their transit provider and they can allocate that bandwidth for other sites.

There are a few different types of peering. The simplest way to conceptualize is plugging in a dedicated cable to exchange data directly between two networks. Practically, there a few ways it happens:

If Cloudflare and the potential peering partner have equipment in the same data center, we can setup a "Private Network Interconnect", which involves dedicated cabling and network configuration.

PNI: Private Network Interconnect

An Internet exchange point (IxP) is a physical location through which Internet companies such as Internet Service Providers (ISPs) and CDNs connect with each other. It typically involves a switching-fabric, that Cloudflare is already connected to. If our potential peering partner is also present in the IxP, setting up peering is a simple as network configuration, with no additional cabling required.

Internet Exchange Point (IxP)

There are scenarios where neither PNI nor IxP peering is feasible, such as when Cloudflare and our potential partners are not present in the same physical location. In these cases, our partner can request to host Cloudflare equipment in their own data center racks. Once commissioned, the equipment effectively operates as if it were a Cloudflare data center (PoP). The cache will populate and all of our performance and reliability services will execute right there, close to our peering partner's customer.

Hosting a node

Peering still has some costs - particularly an administrative burden. The Network Administrators have to agree the details. Commercial teams may require contractual agreements. The link has to be monitored. Therefore, most large networks will have a peering policy and prioritize peering with the networks with which they exchange the most data and will thus generate the most mutual value.

This is especially true for hosting nodes. In addition to the configuration and contractual agreements, there is the logistical effort of receiving and configuring equipment.

The Peering Portal - Beta, released today, allows any ASN (Autonomous System Number) network to view detailed statistics on transit data between it and Cloudflare. Existing partners can use it to review existing session data and statistics and prospective partners can use it to estimate potential transit cost savings, performance improvements and the freeing up of upstream bandwidth to use for other traffic.

The Peering Portal allows both existing and prospective partners to see:

This view shows both growth over time, and the relative traffic for each location.

Time series traffic data

Peering sessions outline how many sessions are established in various locations.

Session Statistics

At a glance view of where data flows in- and out- of connections to Cloudflare from your ASN.

Relative Traffic Weight by Peering Location

We plan to add many more statistics over time.

So, whether you're a current peering partner who'd like more insight into your current traffic with Cloudflare, or a future partner who'd like to explore the performance, financial and traffic management benefits of peering with Cloudflare, or hosting a node visit cloudflare.com/partners/peering-portal and request a login.

You can also review our peering policy here.

The North American Network Operators Group (NANOG) is the loci of modern Internet innovation and the day-to-day cumulative network-operational knowledge of thousands and thousands of network engineers. NANOG itself is a non-profit membership organization; but you don’t need to be a member in order to attend the conference or join the mailing list. That said, if you can become a member, then you’re helping a good cause.

The next NANOG conference starts in a few days (February 6-8 2017) in Washington, DC. Nearly 900 network professionals are converging on the city to discuss a variety of network-related issues, both big and small; but all related to running and improving the global Internet. For this upcoming meeting, Cloudflare has three network professionals in attendance. Two from the San Francisco office and one from the London office.

With the conference starting next week, it seemed a great opportunity to introduce readers of the blog as to why a NANOG conference is so worth attending.

While it seems obvious how to do some network tasks (you unpack the spiffy new wireless router from its box, you set up its security and plug it in); alas the global Internet is somewhat more complex. Even seasoned professionals could do with a recap on how traceroute actually works, or how DNSSEC operates, or this years subtle BGP complexities, or be enlightened about Optical Networking. All this can assist you with deployments within your networks or datacenter.

If there’s one thing that keeps the Internet (a network-of-networks) operating, it’s peering. Peering is the act of bringing together two or more networks to allow traffic (bits, bytes, packets, email messages, web pages, audio and video streams) to flow efficiently and cleanly between source and destination. The Internet is nothing more than a collection of individual networks. NANOG provides one of many forums for diverse network operators to meet face-to-face and negotiate and enable those interconnections.

While NANOG isn’t the only event that draws networks together to discuss interconnection, it’s one of the early forums to support these peering discussions.

In this day-and-age we are brutally aware that security is the number-one issue when using the Internet. This is something to think about when you choose your email password, lock screen password on your laptop, tablet or smartphone. Hint: you should always have a lock screen!

At NANOG the security discussion is focused on a much deeper part of the global Internet, in the very hardware and software practices, that operate and support the underlying networks we all use on a daily basis. An Internet backbone (rarely seen) is a network that moves traffic from one side of the globe to the other (or from one side of a city to the other). At NANOG we discuss how that underlying infrastructure can operate efficiently, securely, and be continually strengthened. The growth of the Internet over the last handful of decades has pushed the envelope when it comes to hardware deployments and network-complexity. Sometimes it only takes one compromised box to ruin your day. Discussions at conferences like NANOG are vital to the sharing of knowledge and collective improvement of everyone's networks.

Above the hardware layer (from a network stack point of view) is the Domain Name System (DNS). DNS has always been a major subject of discussion within the NANOG community. It’s very much up to the operational community to make sure that when you type a website name into web browser or type in someone’s email address into your email program that there’s a highly efficient process to convert from names to numbers (numbers, or IP address, are the address book and routing method of the Internet). DNS has had its fair share of focus in the security arena and it comes down to network operators (and their system administrator colleagues) to protect DNS infrastructure.

Nearly everyone knows that bad news sells. It’s a fact. To be honest, the same is the case in the network operator community. However, within NANOG, those stories of disasters are nearly always told from a learning and improvement point of view. There’s simply no need to repeat a failure, no-one enjoys it a second time around. Notable stories have included subjects like route-leaks, BGP protocol hiccups, peering points, and plenty more.

We simply can’t rule out failures within portions of the network; hence NANOG has spent plenty of time discussing redundancy. The internet operates using routing protocols that explicitly allow for redundancy in the paths that traffic travels. Should a failure occur (a hardware failure, or a fiber cut), the theory is that the traffic will be routed around that failure. This is a recurring topic for NANOG meetings. Subsea cables (and their occasional cuts) always make for good talks.

While we learned twenty or more years ago how to type into Internet routers on the command line, those days are quickly becoming history. We simply can’t scale if network operational engineers have to type the same commands into hundreds (or thousands?) of boxes around the globe. We need automation. This is where NANOG has been a leader in this space. Cloudflare has been active in this arena and Mircea Ulinic presented our experience for Network Automation with Salt and NAPALM at the previous NANOG meeting. Mircea (and Jérôme Fleury) will be giving a follow-up in-depth tutorial on the subject at next week’s meeting.

The first NANOG conference was held June 1994 in Ann Arbor, Michigan and the conference has grown significantly since then. While it’s fun to follow the history, it’s maybe more important to realize that NANOG has covered a multitude of subjects since that start. Go scan the archives at nanog.org and/or watch some of the online videos.

Let’s not forget the advantages of spending time with other operators within a relaxed setting. After all, sometimes the big conversations happen when spending time over a beer discussing common issues. NANOG has long understood this and it’s clear that the Tuesday evening Beer ’n Gear social is set up specifically to let network geeks both grab a drink (soft drinks included) and poke around with the latest and greatest network hardware on show. The social is as much about blinking lights on shiny network boxes as it is about tracking down that network buddy.

Oh; there’s a fair number of vendor giveaways (so far there’s 15 hardware and software vendors signed up for next week’s event). After all, who doesn’t need a new t-shirt?

But there’s more to the downtime and casual hallway conversations. For myself (the author of this blog), I know that sometimes the most important work is done within the hallways during breaks in the meeting vs. standing in front of the microphone presenting at the podium. The industry has long recognized this and the NANOG organizers were one of the early pioneers in providing full-time coffee and snacks that cover the full conference agenda times. Why? Because sometimes you have to step out of the regular presentations to meet and discuss with someone from another network. NANOG knows its audience!

NANOG isn’t the only forum to discuss network operational issues, however it’s arguably the largest. It started off as a “North American” entity; however, in the same way that the Internet doesn’t have country barriers, NANOG meetings (which take place in the US, Canada and at least once in the Caribbean) have fostered an online community that has grown into a global resource. The mailing list (well worth reviewing) is a bastion of networking discussions.

In a different realm, the Internet Engineering Task Force (IETF) focuses on protocol standards. Its existence is why diverse entities can communicate. Operators participate in IETF meetings; however, it’s a meeting focused outside of the core operational mindset.

Central to the Internet’s existence is ICANN. Meeting three times a year at locations around the globe, it focused on the governance arena and in domain names and related items. Within the meetings there’s an excellent Tech Day.

In the numbers arena ARIN is an example of a regional routing registry (an RIR) that runs members meeting. An RIR deals with allocating resources like IP addresses and AS numbers. ARIN focuses on the North American area and sometimes holds its meetings alongside NANOG meetings.

ARIN’s counterparts in other parts of the world also hold meetings. Sometimes they simply focus on resource policy and sometimes they also focus on network operational issues. For example RIPE (in Europe, Central Asia and the Middle East) runs a five-day meeting that covers operational and policy issues. APNIC (Asia Pacific), AFRINIC (Africa), LACNIC (Latin America & Caribbean) all do similar variations. There isn’t one absolute method and that's a good thing. It’s worth pointing out that APNIC holds it’s members meetings once a year in conjunction with APRICOT which is the primary operations meeting in the Asia Pacific region.

While NANOG is somewhat focused on North America, there are also the regional NOGs. These regional NOGs are vital to help the education of network operators globally. Japan has JANOG, Southern Africa has SAFNOG, MENOG in the Middle East, AUSNOG & NZNOG in Australia & New Zealand, DENOG in Germany, PHNOG in the Philippines, and just to be different, the UK has UKNOF (“Forum” vs. “Group”). It would be hard to list them all; but each is a worthwhile forum for operational discussions.

Peering specific meetings also exist. Global Peering Forum, European Peering Forum, and Peering Forum de LACNOG for example. Those focus on bilateral meetings within a group of network operators or administrators and specifically focus on interconnect agreements.

In the commercial realms there’s plenty of other meetings that are attended by networks like Cloudflare. PTC and International Telecoms Week (ITW) are global telecom meetings specifically designed to host one-to-one (bilateral) meetings. They are very commercial in nature and less operational in focus.

As you would guess, you will find our network team at RIR meetings, sometimes at IETF meetings, sometimes at ICANN meetings, often at various regional NOG meetings (like SANOG in South East Asia, NoNOG in Norway, RONOG in Romania, AUSNOG/NZNOG in Australia/New Zealand and many other NOGs). We get around; however, we also run a global network and we need to interact with many many networks around the globe. These meetings provide an ideal opportunity for one-to-one discussions.

If you've heard something you like from Cloudflare at one of these operational-focused conferences, then check out our jobs listings (in various North American cities, London, Singapore, and beyond!)

CC BY-SA 2.0 image by Quinn Dombrowski

Two years ago, we previewed the relative cost of bandwidth that we see in different parts of the world. Bandwidth is the largest recurring cost of providing our service. Compared with Europe and North America, there were considerably higher Internet costs in Australia, Asia and Latin America. Even while bandwidth costs tend to trend down over time, driven by competition and decreases in the costs of underlying hardware, we thought it might be interesting to provide an update.

Since August 2014, we have tripled the number of our data centers from 28 to 86, with more to come. CloudFlare hardware is also deployed in new regions such as the Middle East and Africa. Our network spans multiple countries in each continent, and, sometimes, multiple cities in each country.

Traffic across 86 data centers in the CloudFlare network

There are approximately thirteen networks called “Tier 1 networks” (e.g., Telia, GTT, Tata, Cogent) who sell “transit” to access any of thousands of other networks on the Internet using their global backbones, including networks who are not their customers. We connect to networks by either purchasing transit from a global "Tier 1 network" (or major regional network), or by exchanging traffic directly with a carrier or ISP using “peering”. Typically, peered traffic is exchanged without settlement between the peered parties.

We try to make it as easy as possible for networks to interconnect with us. CloudFlare has an “open peering” policy, and participates at nearly 150 internet exchanges, more than any other company.

As a benchmark, let's assume the cost of transit in Europe and North America is 10 units (per Mbps). With that benchmark in place, without disclosing exact pricing, we can compare regions by transit cost, percentage of peering, and their effective blended cost (transit + peering).

Europe Transit vs Peering (Last 30 Days)

Based on our benchmark, the transit cost is 10 units. The region has a large number of Internet exchanges, typically non-profit, where we peer around 60% of our traffic. This makes for an effective regional cost of 4 units.

With perhaps the notable exception of the incumbent in Germany, many networks are supportive of open interconnection. CloudFlare already participates at 40 European internet exchanges, and is in the process of joining at least five more.

North America Transit vs Peering (Last 30 Days)

The cost of transit in North America is equal to the cost in Europe, or 10 units. We peer around 40% of our traffic, resulting in an effective regional cost of 6 units.

The level of peering in North America is less than in Europe, but a significant improvement over two years ago. The share of peered traffic is expected to grow. Some material changes have occurred and are occurring in the North American market, such as Frontier acquiring Verizon FiOS customers in three U.S. States and Charter preparing to merge with Time Warner Cable. We can see these changes making an impact to the regional interconnection landscape.

Notably, our peering has particularly grown in smaller regional locations, closer to the end visitor, leading to an improvement in performance. This could be through private peering, or via an interconnection point such as the Midwest Internet Cooperative Exchange (MICE) in Minneapolis.

Africa Transit vs Peering (Last 30 Days)

Transit prices in Africa are amongst the highest in the world at 14 times the benchmark or 140 units, with notable variance across the continent, from Cairo to Mombasa to Johannesburg. Fortunately, of the traffic that we are currently able to serve locally in Africa, we manage to peer about 90% (with a mix of carriers and ISPs), making for an effective cost of 14 units.

Our African deployments help us avoid the significant latency of serving websites from London, Paris or Marseille. A particularly promising but challenging region where we hope to deploy a CloudFlare data center is West Africa - specifically Nigeria, which is already at just under 100 million Internet users.

Middle East Transit vs Peering (Last 30 Days)

CloudFlare currently has four data centers in the Middle East, each of which are cache deployments with strategic ISP partners to serve their respective customers. We are able to peer all the traffic currently served from these data centers. While these collectively provide significant coverage, there is additional traffic (reaching Europe) that we would like to localize in the region. We hope that the remaining ISPs, such as Saudi Telecom Company, deploy similar caches, and enhance the performance of their customers.

Because we can peer 100% of our traffic in the Middle East, our effective pricing for bandwidth in the region is 0 units. There are, of course, other costs to delivering our service beyond bandwidth. However, by driving up peering rates in the Middle East we’ve been able to make our service in the Middle East extremely cost competitive.

Asia Transit vs Peering (Last 30 Days)

In Asia (excluding the Middle East), transit costs 7 times times the benchmark, or 70 units. However, we peer about 60% of our traffic, resulting in an effective cost of 28 units.

Beyond the major meeting points in Hong Kong, Singapore and Tokyo, a significant portion of our interconnection is localized to take place closer to visitors in cities such as Bangkok, Chennai, Kuala Lumpur, Mumbai, Osaka, New Delhi, Seoul, and Taipei. These statistics do not include our network of strategically located data centers inside of mainland China, where the dynamics of interconnection are entirely unique.

Two Asian locations stand out as being especially expensive: Seoul and Taipei. In these markets, with powerful incumbents (Korea Telecom and HiNet), transit costs 15x as much as in Europe or North America, or 150 units.

South Korea is perhaps the only country in the world where bandwidth costs are going up. This may be driven by new regulations from the Ministry of Science, ICT and Future Planning, which mandate the commercial terms of domestic interconnection, based on predetermined “Tiers” of participating networks. This is contrary to the model in most parts of the world, where networks self-regulate, and often peer without settlement. The government even prescribes the rate at which prices should decrease per year (-7.5%), which is significantly slower than the annual drop in unit bandwidth costs elsewhere in the world. We are only able to peer 2% of our traffic in South Korea.

If you include HiNet and Korea Telecom in our blended bandwidth pricing, and take into account peering, our effective price is 28 units. If you exclude HiNet and Korea Telecom, our effective price is 14 units.

South America Transit vs Peering (Last 30 Days)

Transit prices in South America are very high, costing 17 times the benchmark, or 170 units. We peer about 60% of traffic in South America, making for an effective cost of 68 units.

One of the reasons that transit prices are high is that the Tier 1 networks which are newer entrants to this region have yet to pick up significant market share. While markets such as Brazil are less expensive and have greater peering, costs are highest in countries such as Peru and Argentina where, in each, a single incumbent provider, respectively Telefonica and Telecom Argentina, controls access for the last mile delivery of content to the majority of Internet users.

As we try to increase our share of peered traffic, one of the challenges we face is that many Internet exchanges (e.g., NAP Colombia) only permit domestically incorporated and licensed networks to publicly peer, or in another case, require a unanimous vote of all members on an IX to permit a new participant, effectively creating a separation between “international content” and “domestic content”.

If you include Telecom Argentina and Telefonica, our blended cost of bandwidth in South America is 68 units. If you exclude these two providers then our blended cost is 17 units.

Oceania Transit vs Peering (Last 30 Days)

Transit prices in Oceania (Australia and New Zealand) are lower than they used to be, but continue to be extremely high in relative terms, costing 17 times the benchmark from Europe, or 170 units. We peer 50% of our traffic, resulting in an effective cost of 85 units.

If you exclude Optus and Telstra, then the price falls to 17 units — because we peer with nearly everyone else.

Relative Cost of Bandwidth

CloudFlare has always optimized where we serve customers to take into account our effective costs. If you are a free customer using an excessive amount of expensive transit, we would serve you from fewer regions. The good news is that, over the last five years, we’ve been able to negotiate reasonable transit pricing or settlement-free peering with the vast majority of the world’s networks. That allows us to continue to provide the free version of our service as well as to keep prices low for all our paid services.

Today, however, there are six expensive networks (HiNet, Korea Telecom, Optus, Telecom Argentina, Telefonica, Telstra) that are more than an order of magnitude more expensive than other bandwidth providers around the globe and refuse to discuss local peering relationships. To give you a sense, these six networks represent less than 6% of the traffic but nearly 50% of our bandwidth costs.

While we’ve tried to engage all these providers to reduce their extremely high costs and ensure that even our Free customers can be served across their networks, we’ve hit an impasse. To that end, unfortunately, we’ve made the decision that the only thing that will change these providers’ pricing is to make it clear how out of step they are with the rest of the world. To demonstrate this, we’ve moved our Free customers off these six transit providers. Free customers will still be accessible across our network and served from another regional cache with more reasonable bandwidth pricing.

Ironically, this actually increases the cost to several of these providers because they now need to backhaul traffic to another CloudFlare data center and pay more in the process. For instance, if Telstra were to peer with CloudFlare then they would only have to move traffic over about 30 meters of fiber optic cable between our adjoining cages in the same data center. Now Telstra will need to backhaul traffic to Free customers to Los Angeles or Singapore over expensive undersea cables. Their behavior is irrational in any competitive market and so it is not a surprise that each of these providers is a relative monopolist in their home market.

If you’re a Free CloudFlare customer who cares about optimizing the best possible performance from one of these six providers then we encourage you to reach out to them and encourage them to follow a core principle of a free and open Internet and not abuse their monopoly position. We are committed to serving all our customers across every network that peers with us. To that end, help us convince these six networks to be on the right side of a free and open Internet by reaching out to your ISP.

• Ask HiNet to peer with CloudFlare in Taipei

• Ask Korea Telecom to peer with CloudFlare in Seoul

• Ask Optus to peer with CloudFlare across Australia

• Ask Telecom Argentina to peer with CloudFlare in Buenos Aires

• Ask Telefonica to peer with CloudFlare across South America

• Ask Telstra to peer with CloudFlare across Australia

We’ll post updates as we negotiate with these six networks and are hopeful that we’ll soon be able to serve all our customers across all the networks we interconnect with.

Здоровенькі були! Cloudflare just turned up our newest datacenter in Kyiv, the capital and largest city of Ukraine.

Kyiv is an old city with more than 1,000 years of history. It was the capital of Kyivan Rus’, an ancient country which is considered to be the ancestor of modern Ukraine, Belarus and Russia. If you visit the city by plane, you may be almost blinded by the shining golden domes of numerous old churches and cathedrals - and once there, be sure to try the famous Ukrainian beet soup, “Borscht”. Cloudflare decided to contribute to the long history of Kyiv with our 22nd data center in Europe, and our 78th data center globally.

Frankfurt is arguably the biggest point of interconnection in the world, and is home to Deutscher Commercial Internet Exchange (DE-CIX) which plays an absolutely critical role and sees close to 5Tbps in traffic. While this is great if you live near Frankfurt, it is also where most traffic is exchanged for other parts of Germany, large parts of Europe (think Austria, Bulgaria, Czech Republic, Denmark, Estonia, Finland, Greece, Hungary, Italy, Kazakhstan, Lithuania, Montenegro, Norway, Poland, Romania, Serbia, Turkey, Ukraine, etc.), and even countries such as Saudi Arabia, thereby introducing latency for website visitors.

Each new Cloudflare PoP localizes our customers’ content, improving performance and avoiding the need to go back to Frankfurt. In Germany alone, Cloudflare’s network now spans four cities (Frankfurt, Dusseldorf, Berlin, Hamburg), with more expansion to follow (Nächster Halt, München!). The Kyiv data center expands our surface area to withstand potential attacks, and serves as an additional point of redundancy.

For Kyiv, and some of our upcoming cities, we’ll begin by serving traffic to ISPs that have agreed to peer with us and prefer picking up our traffic locally. If you don’t see your preferred ISP, let them know, and we will too, as our effort to improve the performance of millions of websites continues.

Photo sources: Flickr (Bert Kaufmann); Google Maps

At CloudFlare we are dedicated to peering. So much so that we just joined our 100th Internet Exchange point!

Image courtesy of Martin Levy

According to Wikipedia:

“In computer networking, peering is a voluntary interconnection of administratively separate Internet networks for the purpose of exchanging traffic between the users of each network”

In reality this normally means a physical place where two different networks (they could be backbones, CDNs, mobile networks or broadband ISPs) connect their respective networks together to exchange traffic. Over the last fifteen years, there has been a major expansion in network interconnections, running parallel to the enormous expansion of the global Internet. This expansion includes new data centre facilities being developed to house network equipment. Some of those data centres have attracted massive numbers of networks, in no small part due to the thriving Internet Exchanges Points (both new and existing) that operate within them. London with the LINX and LONAP exchanges, Amsterdam with AMS-IX and NL-IX exchanges, Frankfurt with DE-CIX and ECIX exchanges are good examples for Europe. In North America there are plenty of IXPs run by Equinix, TelX, Coresite and a myriad of independent operators. The same can be said for Asia Pacific, South America, Oceania and more recently the African continent. Over the last fifteen years IXPs have been deployed in nearly every corner of the globe.

Peering has become a core requirement of the global Internet. With around 54,000 unique Autonomous Networks (ASNs) making up the global Internet, nearly 6,000 of those networks participate in the peering ecosystem; some big, some small.

A volunteer-based organisation runs PeeringDB, a database of peering networks, peering points, peering locations and more. It’s 100% user-driven data and provides networks with a directory of IXPs and contact information for networks willing to peer. As of January 2016 there’s around 5,700 networks and around 630 IXPs listed in the database. Keeping your PeeringDB information maintained is a must for anybody wanting to peer; here are some tips.

There’s almost nothing stopping any network from peering, especially if there’s another network within its physical reach. Once you have two, three (or more) networks in the same locale, then it makes perfect sense to start an IXP.

CloudFlare operates 75+ PoPs globally and nearly all of them are located close to where a local Internet Exchange exists. By connecting to the local IXP (or multiple local IXPs) we provide our customers an unprecedented level of web and DNS service delivery.

At their core, IXPs are large Layer 2 LANs. Sometimes built with one Ethernet switch, sometimes many switches interconnected together across one or more physical buildings. With either configuration the goal is to connect networks together physically. An IXP is no different in basic concept to your home network, with the only real difference being scale. IXPs can range from 100s of Megabits/second to many Terabits/second of exchanged traffic. Independent of size, their primary goal is to make sure that many networks’ routers are connected together cleanly and efficiently. In comparison, at home you would normally only have one router and many computers or mobile devices.

Across the IXP's local network those different ISPs are able to create one-to-one connections using the BGP protocol. This protocol was created to allow disparate networks to announce to each other their own IP addresses plus those that they have provided connectivity to downstream (i.e. their customers). Once two networks set up a BGP session, their respective routes are exchanged and hence traffic can flow directly between them. Any previous intermediary network or networks are removed from the equation.

BGP allows groups of IP addresses to be announced as a single entity (called a CIDR block). An IP address is a 32 bit number for IPv4 and 128 bit number for IPv6. As the global Internet is made up of billions and billions of IP addresses, the BGP protocol consolidates IP addresses into CIDR network addresses. Within the IPv4 Internet these networks could contain anywhere from 256 contiguous IP addresses (called a /24) up to 16,777,216 contiguous addresses (a /8). A /24 means 2^(32-24) addresses. In IPv6, the smallest network block is either a /64 (18,446,744,073,709,551,616 IP addresses - or 2^(128-64) addresses) or a /48 (a staggering 1,208,925,819,614,629,174,706,176 individual addresses). It’s best to think of a /48 as just 65,536 /64s - the basic building block of IPv6 networking.

Getting back to BGP and the global Internet, the reason it works is that cooperative networks announce their IP address blocks (CIDR blocks) to peering neighbours. Here’s an example of one of those BGP announcements:

8.8.8.0/24         *[BGP/170] 15:29:05, MED 0, localpref 200, from 80.249.208.247
                     AS path: 15169 I, validation-state: unverified
                     to 80.249.208.247 via ae3.0
                   > to 80.249.209.100 via ae3.0

CloudFlare can see this route within its network because our network peers directly with the source of that specific CIDR block. What’s going on here is the other ISP’s router (aka our peering partner) is telling our router that our network can send traffic towards 8.8.8.0/24 (the network block that encompasses Google’s 8.8.8.8 DNS resolver service) to the IXP fabric. Our router sends traffic to the IP addresses of 80.249.208.247 or 80.249.209.100 as the next hop on the exchange fabric. (Fabric is a fancy word for Ethernet switch). The AS path shows that it’s a direct hop to AS15169/Google, meaning we have direct connectivity to Google over this specific Internet Exchange Point without an intermediary network. In reality, CloudFlare and Google interconnect via peering points in every continent (nearly 65+ locations globally).

This single example is repeated again-and-again with 1,000s of other networks across all of CloudFlare’s locations. Each peering session and peering point are dedicated to improve CloudFlare’s network offering towards its customers.

Without IXPs, traffic going from one ISP to another would rely on an intermediary backbone ISP to carry the traffic from source to destination. In some situations there’s no problem with doing this: it’s how a large portion of international Internet traffic flows, as it’s cost prohibitive to maintain direct connections to each-and-every ISP in the world. However, relying on a backbone ISP to carry local traffic can be adverse to performance, sometimes due to the backbone carrier being connected to another backbone whom it hands the traffic to in a completely different city. This situation can lead to what’s known as tromboning, where traffic from one city destined to another ISP in the same city can travel vast distances to be exchanged and then return again. In many developing countries this is a very common.

By connecting to a local Internet Exchange Point and peering with other local ISPs, traffic becomes finely tuned and traffic is exchanged locally and directly. This gives the best performance for our web and DNS services towards those destination networks that peer with us. At some of our PoPs we serve more than 90% of outgoing traffic over Internet Exchange Points, helping to keep latency for network traffic as low a possible.

As well as a performance benefit, exchanging traffic directly over IXPs improves redundancy. Because CloudFlare can setup peering with those other ISPs over more than one exchange, this means both networks have possibly diverse paths in which to exchange traffic.

If for example an ISP only has connectivity to one or two backbones and one of those backbones suffers an outage, traffic destined for ISPs on the Internet Exchange would not be affected as it is exchanged directly.

CloudFlare operates a network that protects its customers from DDoS attacks and alas that DDoS traffic (created somewhere on the global Internet) ends up flowing towards CloudFlare’s network. As we have written about before here and here, we have built a massive network to address DDoS traffic and filter it prior to reaching our customers’ web services. Peering plays a big part in that story. In the same way as we want to optimise the path out of our network towards the end user (providing the best experience when viewing content delivered by our network), we also want to optimise the path of nefarious DDoS traffic inbound to our network.

Why? Removing an intermediary network that’s in the path of a DDoS attack can actually help keep the Internet operating cleanly. We've built a network that will handle DDoS attack traffic, if that traffic reaches us via direct peering sessions then experience shows we have the ability to handle the traffic levels (and packet-per-second levels) associated with it.

If there’s a intermediary network in that path we sometimes observe collateral congestion or packet loss within that network. Not a good thing. A direct path (over a peering connection) means there’s the shortest path between the nefarious source and our filtering and sinking of that DDoS traffic. Without peering, the Internet could not handle the types of DDoS attacks that are much too common these days.

Just about anyone! There’s 100s and 100s of IXPs operating globally. Some countries only have one IXP, some (such as Brazil) have nearly 50 individual IXPs. Alas there are quite a few countries where no IXP exists at all.

Many operating models exist for IXPs. CloudFlare connects to IXPs with a variety of these models.

The membership model (preferred in Europe) centres around a group of networks that are members of a non-profit organisation that operates an IXP in one or more cities. Membership payments along with a per-connection fee keep the IXP operating. With a membership based IXP, the fee structure is publicly published and all members pay equally. This model does exist in some North American cities, though to a much lesser extent. It also exists in most other continents.

Next up is a purely commercial model where the operator of an IXP is a commercial for-profit entity. With the commercial model, pricing can be negotiated and in many cases the commercial entity also operates a data centre facility where costs for colocation (space, power, security, remote-hands, cross-connects et al.) eclipse the cost of the IXP itself, Negotiating downwards towards no cost is quite common. After all in the commercial world, a loss-leader mentality can be common.

Sometimes there are 100% volunteer-based IXPs with very little structure. These exist in many cities as a secondary IXP to a primary (read: larger) IXP. Sometimes a friendly group of ISPs get together (over a few beers?) and just decide it would be fun to operate an IXP. We love these people. You’ve read many times about our love of innovation, entrepreneurship, random hacking and experimenting for the good of the Internet. That also happens in the IXP space. Sometimes these IXPs get big and have to convert to a paid-for model. That’s success in its rawest form!

Not be outdone by commercial, membership or even volunteer models, some governments insist on getting into the IXP game. In most cases this is in countries where an excessive number of telecommunication regulations exist. As you can imagine (if you've read this blog before), we aren’t big fans of telecommunication regulations. That said, some of these IXPs surprisingly work quite well. Egypt’s regulatory body (NTRA – National Telecommunications Regulatory Authority) operates CAIX (the Cairo Internet eXchange). It’s a success story for the Egyptian Internet. India’s various Internet Exchanges have not been as successful. These seven IXP locations are operated by the Indian government’s NIXI (National Internet Exchange of India) company. When reading the statistics, from a distance, it shows bandwidth levels very low when compared with the size of the Indian Internet itself. CloudFlare operates three PoPs in India and by design they are not connected to the NIXI exchanges.

Finally, there’s the Pseudo-IXP. This is an exchange operated by a telecom operator (sometimes the incumbent) whose sole purpose is to provide connectivity back to its own network and charge accordingly for the pleasure. Thailand has nearly a dozen of these. Each telecom company or mobile operation has its own IXP, which misses the point of interconnects! We aren't a fan of this model, and most forward-thinking networks aren't fans of this model either.

When a network wants to expand its footprint and connect to an IXP in another city or country (or even continent), it was normal practice to extend the Layer 3 network by placing a router in that remote location. There was expense associated with those implementations. A router has to be purchased and shipped to the remote site. The remote site has to be acquired (renting space, power, etc.) and some form of remote-hands are needed to rack and stack the network equipment. Finally, there needed to be a contract with the IXP and a connection engineered and paid for.

Then along came the concept of remote peering, which extends the Layer 2 switch fabric (in a very controlled manner) via a third party telecom operator to another city, country or even continent, allowing remote networks to become true members of an IXP. Europe took this concept and went wild with it. The main IXPs in London, Amsterdam & Frankfurt got telecom partners signed up to provide remote peering. Their number of connected networks (and traffic levels) increased instantly. Other parts of the world got the remote peering bug and in this day and age you can virtually (pun intended) connect to IXPs all over the world via remote Layer 2 connections.

Remote peering is an obvious idea and to be honest it helps many ISPs reach peers in neighbouring cities or countries via a direct peering relationship. For CloudFlare our aim is to serve traffic as locally as possible and hence we don’t use remote peering. We try to not peer with remotely connected networks if we have a PoP closer to them. We are relentless in this desire to keep traffic local.

Similar to the concept of remote peering, some IXPs expand their footprint across a country or in some cases across multiple countries, this means an ISP can peer with another ISP in a different country via a local connection to the IXP. This is for practical reasons the same as remote peering, with the difference that the long-range link is between the IXP switches, instead of the connection between the ISP and the IXP. There are a few examples of where CloudFlare peers on distributed exchanges, in order to reach both local participants and ISPs in other surrounding countries that we’ve not yet finished building PoPs in.

The world isn’t perfect and in some markets local peering cannot be achieved (sometimes due to a lack of any local IXPs). In situations where we cannot peer locally on an IXP we aim to connect to ISPs privately. This is the same idea as peering over an IXP except we connect directly to the ISP’s router from our router.

In some cases CloudFlare will help foster the creation of a new IXP or IXPs, such as the many new MegaIX exchange points coming to North America. After all, when you improve local Internet connectivity, you actually improve global connectivity. The Internet is a global entity and every part plays an important role.

While some marketplaces have only one IXP (for example Dublin in Ireland with INEX), some markets have seen redundancy, duplication or fragmentation between IXPs. This is in most cases a good thing. Redundancy or even competition between commercial entities always makes the Internet stronger. The Internet was designed to handle disruptions to its underlying infrastructure. IXPs are just one more part of that infrastructure. We will sometimes connect to three (or four) IXPs in a single city. Four may seem excessive, but for the sake of good redundancy, four is a great number.

When there is more than one IXP in a city there’s a tendency for some ISPs to prefer one IXP over another, or in some cases even running their own. By CloudFlare being present at many exchanges, it gives us a unique and redundant view of the local Internet, it also allows us to have the best coverage of that local market. It brings the added advantage of being able to peer with certain regional networks over multiple exchanges and hence allowing for the best routing redundancy.

After five years of operating a network that's grown to over 75 PoPs globally we have joined our 100th Internet Exchange Point. We have a network that’s growing each and every day. Reaching 100 may be just a number; however, only one or two networks have had the ability to grow to that size. Our customers reap the benefits of that global reach. We won’t stop at 100. Far from it! Want to help us grow our network and increase our global peering? Check out available roles here.

Are you an ISP? Do you want to peer with us? Visit as13335.peeringdb.com or talk to us about an on-net cache.

CC BY 2.0 by Kendrick Erickson

Over the last few months, there’s been increased attention on networks and how they interconnect. CloudFlare runs a large network that interconnects with many others around the world. From our vantage point, we have incredible visibility into global network operations. Given our unique situation, we thought it might be useful to explain how networks operate, and the relative costs of Internet connectivity in different parts of the world.

The Internet is a vast network made up of a collection of smaller networks. The networks that make up the Internet are connected in two main ways. Networks can connect with each other directly, in which case they are said to be “peered”, or they can connect via an intermediary network known as a “transit provider”.

At the core of the Internet are a handful of very large transit providers that all peer with one another. This group of approximately twelve companies are known as Tier 1 network providers. Whether directly or indirectly, every ISP (Internet Service Provider) around the world connects with one of these Tier 1 providers. And, since the Tier 1 providers are all interconnected themselves, from any point on the network you should be able to reach any other point. That's what makes the Internet the Internet: it’s a huge group of networks that are all interconnected.

To be a part of the Internet, CloudFlare buys bandwidth, known as transit, from a number of different providers. The rate we pay for this bandwidth varies from region to region around the world. In some cases we buy from a Tier 1 provider. In other cases, we buy from regional transit providers that either peer with the networks we need to reach directly (bypassing any Tier 1), or interconnect themselves with other transit providers.

CloudFlare buys transit wholesale and on the basis of the capacity we use in any given month. Unlike some cloud services like Amazon Web Services (AWS) or traditional CDNs that bill for individual bits delivered across a network (called "stock"), we pay for a maximum utilization for a period of time (called "flow"). Typically, we pay based on the maximum number of megabits per second we use during a month on any given provider.

Traffic levels across CloudFlare's global network over the last 3 months. Each color represents one of our 28 data centers.

Most transit agreements bill the 95th percentile of utilization in any given month. That means you throw out approximately 36 not-necessarily-contiguous hours worth of peak utilization when calculating usage for the month. Legend has it that in its early days, Google used to take advantage of these contracts by using very little bandwidth for most of the month and then ship its indexes between data centers, a very high bandwidth operation, during one 24-hour period. A clever, if undoubtedly short-lived, strategy to avoid high bandwidth bills.

Another subtlety is that when you buy transit wholesale you typically only pay for traffic coming in (“ingress") or traffic going out (“egress”) of your network, not both. Generally you pay which ever one is greater.

CloudFlare is a caching proxy so egress (out) typically exceeds ingress (in), usually by around 4-5x. Our bandwidth bill is therefore calculated on egress so we don't pay for ingress. This is part of the reason we don't charge extra when a site on our network comes under a DDoS attack. An attack increases our ingress but, unless the attack is very large, our ingress traffic will still not exceed egress, and therefore doesn’t increase our bandwidth bill.

While we pay for transit, peering directly with other providers is typically free — with some notable exceptions recently highlighted by Netflix. In CloudFlare's case, unlike Netflix, at this time, all our peering is currently "settlement free," meaning we don't pay for it. Therefore, the more we peer the less we pay for bandwidth. Peering also typically increases performance by cutting out intermediaries that may add latency. In general, peering is a good thing.

The chart above shows how CloudFlare has increased the number of networks we peer with over the last three months (both over IPv4 and IPv6). Currently, we peer around 45% of our total traffic globally (depending on the time of day), across nearly 3,000 different peering sessions. The chart below shows the split between peering and transit and how it's improved over the last three months as we’ve added more peers.

We don't disclose exactly what we pay for transit, but I can give you a relative sense of regional differences. To start, let's assume as a benchmark in North America you'd pay a blended average across all the transit providers of \$10/Mbps (megabit per second per month). In reality, we pay less than that, but it can serve as a benchmark, and keep the numbers round as we compare regions. If you assume that benchmark, for every 1,000Mbps (1Gbps) you'd pay $10,000/month (again, acknowledge that’s higher than reality, it’s just an illustrative benchmark and keeps the numbers round, bear with me).

While that benchmark establishes the transit price, the effective price for bandwidth in the region is the blended price of transit (\$10/Mbps) and peering (\$0/Mbps). Every byte delivered over peering is a would-be transit byte that doesn't need to be paid for. While North America has some of the lowest transit pricing in the world, it also has below average rates of peering. The chart below shows the split between peering and transit in the region. While it's gotten better over the last three months, North America still lags behind every other region in the world in terms of peering.

While we peer nearly 40% of traffic globally, we only peer around 20-25% in North America. Assuming the price of transit is the benchmark \$10/Mbps in North America without peering, with peering it is effectively \$8/Mbps. Based only on bandwidth costs, that makes it the second least expensive region in the world to provide an Internet service like CloudFlare. So what's the least expensive?

Europe's transit pricing roughly mirrors North America's so, again, assume a benchmark of \$10/Mbps. While transit is priced similarly to North America, in Europe there is a significantly higher rate of peering. CloudFlare peers 50-55% of traffic in the region, making the effective bandwidth price \$5/Mbps. Because of the high rate of peering and the low transit costs, Europe is the least expensive region in the world for bandwidth.

The higher rate of peering is due in part to the organization of the region's “peering exchanges”. A peering exchange is a service where networks can pay a fee to join, and then easily exchange traffic between each other without having to run individual cables between each others' routers. Networks connect to a peering exchange, run a single cable, and then can connect to many other networks. Since using a port on a router has a cost (routers cost money, have a finite number of ports, and a port used for one network cannot be used for another), and since data centers typically charge a monthly fee for running a cable between two different customers (known as a "cross connect"), connecting to one service, using one port and one cable, and then being able to connect to many networks can be very cost effective.

The value of an exchange depends on the number of networks that are a part of it. The Amsterdam Internet Exchange (AMS-IX), Frankfurt Internet Exchange (DE-CIX), and the London Internet Exchange (LINX) are three of the largest exchanges in the world. (Note: these links point to PeeringDB.com which provides information on peering between networks. You'll need to use the username/password guest/guest in order to login.)

In Europe, and most other regions outside North America, these and other exchanges are generally run as non-profit collectives set up to benefit their member networks. In North America, while there are Internet exchanges, they are typically run by for-profit companies. The largest of these for-profit exchanges in North America are run by Equinix, a data center company, which uses exchanges in its facilities to increase the value of locating equipment there. Since they are run with a profit motive, pricing to join North American exchanges is typically higher than exchanges in the rest of the world.

CloudFlare is a member of many of Equinix's exchanges, but, overall, fewer networks connect with Equinix compared with Europe's exchanges (compare, for instance, Equinix Ashburn, which is their most popular exchange with about 400 networks connected, versus 1,200 networks connected to AMS-IX). In North America the combination of relatively cheap transit, and relatively expensive exchanges lowers the value of joining an exchange. With less networks joining exchanges, there are fewer opportunities for networks to easily peer. The corollary is that in Europe transit is also cheap but peering is very easy, making the effective price of bandwidth in the region the lowest in the world.

Asia’s peering rates are similar to Europe. Like in Europe, CloudFlare peers 50-55% of traffic in Asia. However, transit pricing is significantly more expensive. Compared with the benchmark of \$10/Mbps in North America and Europe, Asia's transit pricing is approximately 7x as expensive (\$70/Mbps, based on the benchmark). When peering is taken into account, however, the effective price of bandwidth in the region is \$32/Mbps.

There are three primary reasons transit is so much more expensive in Asia. First, there is less competition, and a greater number of large monopoly providers. Second, the market for Internet services is less mature. And finally, if you look at a map of Asia you’ll see a lot of one thing: water. Running undersea cabling is more expensive than running fiber optic cable across land so transit pricing offsets the cost of the infrastructure to move bytes.

Latin America is CloudFlare's newest region. When we opened our first data center in Valparaíso, Chile, we delivered 100 percent of our traffic over transit, which you can see from the graph above. To peer traffic in Latin America you need to either be in a "carrier neutral" data center — which means multiple network operators come together in a single building where they can directly plug into each other's routers — or you need to be able to reach an Internet exchange. Both are in short supply in much of Latin America.

The country with the most robust peering ecosystem is Brazil, which also happens to be the largest country and largest source of traffic in the region. You can see that as we brought our São Paulo, Brazil data center online about two months ago we increased our peering in the region significantly. We've also worked out special arrangements with ISPs in Latin America to set up facilities directly in their data centers and peer with their networks, which is what we did in Medellín, Colombia.

While today our peering ratio in Latin America is the best of anywhere in the world at approximately 60 percent, the region's transit pricing is 8x (\$80/Mbps) the benchmark of North America and Europe. That means the effective bandwidth pricing in the region is \$32/Mbps, or approximately the same as Asia.

Australia is the most expensive region in which we operate, but for an interesting reason. We peer with virtually every ISP in the region except one: Telstra. Telstra, which controls approximately 50% of the market, and was traditionally the monopoly telecom provider, charges some of the highest transit pricing in the world — 20x the benchmark (\$200/Mbps). Given that we are able to peer approximately half of our traffic, the effective bandwidth benchmark price is \$100/Mbps.

To give you some sense of how out-of-whack Australia is, at CloudFlare we pay about as much every month for bandwidth to serve all of Europe as we do to for Australia. That’s in spite of the fact that approximately 33x the number of people live in Europe (750 million) versus Australia (22 million).

If Australians wonder why Internet and many other services are more expensive in their country than anywhere else in the world they need only look to Telstra. What's interesting is that Telstra maintains their high pricing even if only delivering traffic inside the country. Given that Australia is one large land mass with relatively concentrated population centers, it's difficult to justify the pricing based on anything other than Telstra's market power. In regions like North America where there is increasing consolidation of networks, Australia's experience with Telstra provides a cautionary tale.

The chart above shows the relative cost of bandwidth assuming a benchmark transit cost of \$10/Megabits per second (Mbps) per month (which we know is higher than actual pricing, it’s just a benchmark) in North America and Europe.

While we keep our pricing at CloudFlare straight forward, charging a flat rate regardless of where traffic is delivered around the world, actual bandwidth prices vary dramatically between regions. We’ll continue to work to decrease our transit pricing, and increasing our peering in order to offer the best possible service at the lowest possible price. In the meantime, if you’re an ISP who wants to offer better connectivity to the increasing portion of the Internet behind CloudFlare’s network, we have an open policy and are always happy to peer.

In the coming weeks, connectivity to CloudFlare in Australia is going to a new level. As part of CloudFlare’s ongoing upgrades program, we established connections to three new Internet exchanges: the Megaport Internet exchanges in Sydney, Brisbane, and Melbourne. These connections doubled the number of Australian Internet exchanges we reach and marked the first exchanges outside of Sydney that Cloudflare participates in.

When two ISPs peer, they agree to exchange traffic directly between each other rather than sending it a third party. By doing this, both partners avoid congested paths between transit providers, and they avoid paying to ship traffic—it's win-win!

What peering exchanges mean for CloudFlare is that we can significantly increase our service performance to users on ISPs that peer with us. Take Australia for example, for users who are currently on ISPs peering at Megaport, instead of CloudFlare sending traffic to the transit providers of those ISPs, we can now route the traffic directly to them. The result is lower latency, and traffic taking paths that are often less congested.

Low latency is crucial for internet speed due to the nature of TCP, the fundamental protocol on which the internet is built. TCP operates in such a way that any packet loss from a congested transit link will significantly slow a connection, and, conversely, connections with reduced latency will hugely amplify performance for end users. Therefore, by moving traffic to less congested, more direct "pipes" on the internet, CloudFlare is creating a faster web.

CloudFlare understands the importance of peering, and our team has put considerable time and effort into finding peering partners as our network expands. We peer as much as possible, both by participating at internet exchanges, and by establishing direct interconnects with ISPs.

As I write, we’re in the process of deploying equipment to many new data centers around the globe, and extending our network to reach more peering exchanges. In addition to the twenty-eight plus internet exchanges where CloudFlare already peers, we will soon be participating at: Terremark São Paulo, PTT-SP (São Paulo Brazil), EspanIX, FranceIX, MIX, NetNod, JPNAP, and, of course, Megaport. We’re also constantly commissioning new private interconnects with a range of eyeball networks.

CloudFlare is building a better web, and part of that project includes reducing the distance packets travel between you and your ISP. As the months go by, our network is expanding around the globe, and more of our traffic is sent through peering partners. The result? Faster and more reliable content delivery to our users.

CloudFlare's commitment to finding the most direct path over the internet to deliver your traffic, and, as the reach of our network expands, you can expect that our service will only get better.

CloudFlare maintains an open peering policy. Our peering details can be found here. Please contact us if you are an ISP on any of the IXs we participate in and would like to peer.

CloudFlare's traffic grows in a number of different ways. The most obvious is that we sign up more websites. We also grow as the natural traffic to sites using CloudFlare increases as they get more popular themselves. Another way we grow is less obvious but extremely cool: as CloudFlare makes the web faster, visitors end up surfing more pages.

We had a really good example of this in just the last couple days. Our networking team was able to work some routing magic to more efficiently get traffic to our Singapore data center. This, on average, saved about 25 milliseconds -- 0.025 seconds, a tiny sliver of time, about 1/12th the time it takes you to a blink of your eyes -- for requests from countries in the region including Thailand, Malaysia, and Indonesia.

The graph above shows traffic over the last 9 days from TOT, one of the largest ISPs in Thailand. You can see from the graph that traffic rises and falls depending on the time of day, which is normal, but if you look at the peaks you'll see they step up dramatically in the last two days.

Digging into the details, there was approximately a 30% increase across bandwidth, hits, and page views in the region after the improved routing. The increase holds even if you control for the day of the week, remove new sites that signed up over the period, and compare other regions that also benefited from the routing so as to control for other potential explanations like weather, news events, or anything else would have had more people surfing the web in Thailand in the last few days. In other words, just eliminating 25ms in latency resulted in a 30% increase in traffic. That's really cool.

The very nature of the way that TCP, the protocol of the Internet, works means that any performance benefit tends to be amplified. Google, Amazon and the other Internet giants have known for a long time that faster means higher engagement and more Internet use. At CloudFlare, we have network engineers that have helped build the networks for some of those Internet giants now at work tuning connections to save milliseconds for the rest of the web. We'll continue to add data centers and improve routing toward our mission of making a faster, safer web for everyone.

CloudFlare is now connected to HKIX (Hong Kong Internet Exchange). HKIX is the largest Internet Exchange in Asia transferring around 200Gbit to its 160 ISP, Carrier and Content networks. HKIX is fundamental to the local Hong Kong ISP Market, with every provider in Hong Kong connected, as well as excellent regional coverage with networks from Thailand to Japan to Australia.

So, what does a connection to HKIX mean for CloudFlare users? Faster performance for your Hong Kong web surfers. Being a part of the HKIX allows us to deliver traffic to all Hong Kong web surfers within Hong Kong. This will improve web browsing performance by keeping the traffic local and the latency low.

Connecting to HKIX is one of the steps CloudFlare is working on to bring the Internet closer to you. Watch for more news to come over 2012!