We make no secret about how passionate we are about building a world-class global network to deliver the best possible experience for our customers. This means an unwavering and continual dedication to always improving the breadth (number of cities) and depth (number of interconnects) of our network.

This is why we are pleased to announce that Cloudflare is now connected to over 12,000 Internet networks in over 300 cities around the world!

The Cloudflare global network runs every service in every data center so your users have a consistent experience everywhere—whether you are in Reykjavík, Guam or in the vicinity of any of the 300 cities where Cloudflare lives. This means all customer traffic is processed at the data center closest to its source, with no backhauling or performance tradeoffs.

Having Cloudflare’s network present in hundreds of cities globally is critical to providing new and more convenient ways to serve our customers and their customers. However, the breadth of our infrastructure network provides other critical purposes. Let’s take a closer look at the reasons we build and the real world impact we’ve seen to customer experience:

Our network allows us to sit approximately 50 ms from 95% of the Internet-connected population globally. Nevertheless, we are constantly reviewing network performance metrics and working with local regional Internet service providers to ensure we focus on growing underserved markets where we can add value and improve performance. So far, in 2023 we’ve already added 12 new cities to bring our network to over 300 cities spanning 122 unique countries!

In May, we activated a new data center in Campos dos Goytacazes, Brazil, where we interconnected with a regional network provider, serving 100+ local ISPs. While it's not too far from Rio de Janeiro (270km) it still cut our 50th and 75th percentile latency measured from the TCP handshake between Cloudflare's servers and the user's device in half and provided a noticeable performance improvement!

A larger number of local interconnections facilitates direct connections between network providers, content delivery networks, and regional Internet Service Providers. These interconnections enable faster and more efficient data exchange, content delivery, and collaboration between networks.

Currently there are approximately 74,000¹ AS numbers routed globally. An Autonomous System (AS) number is a unique number allocated per ISP, enterprise, cloud, or similar network that maintains Internet routing capabilities using BGP. Of these approximate 74,000 ASNs, 43,000² of them are stub ASNs, or only connected to one other network. These are often enterprise, or internal use ASNs, that only connect to their own ISP or internal network, but not with other networks.

It’s mind blowing to consider that Cloudflare is directly connected to 12,372 unique Internet networks, or approximately 1/3rd of the possible networks to connect globally! This direct connectivity builds resilience and enables performance, making sure there are multiple places to connect between networks, ISPs, and enterprises, but also making sure those connections are as fast as possible.

A previous example of this was shown as we started connecting more locally. As seen in this blog post the local connections even increased how much our network was being used: better performance drives further usage!

At Cloudflare we ensure that infrastructure expansion strategically aligns to building in markets where we can interconnect deeper, because increasing our network breadth is only as valuable as the number of local interconnections that it enables. For example, we recently connected to a local ISP (representing a new ASN connection) in Pakistan, where the 50th percentile improved from ~90ms to 5ms!

Network expansion may be driven by reducing latency and improving interconnections, but it’s equally valuable to our existing network infrastructure. Increasing our geographic reach strengthens our redundancy, localizes failover and helps further distribute compute workload resulting in more effective capacity management. This improved resilience reduces the risk of service disruptions and ensures network availability even in the event of hardware failures, natural disasters, or other unforeseen circumstances. It enhances reliability and prevents single points of failure in the network architecture.

Ultimately, our commitment to strategically expanding the breadth and depth of our network delivers improved latency, stronger interconnections and a more resilient architecture - all critical components of a better Internet! If you’re a network operator, and are interested in how, together, we can deliver an improved user experience, we’re here to help! Please check out our Edge Partner Program and let’s get connected.

........

¹CIDR Report

²Origin ASs announced via a single AS path

Internet outages are more common than most people think, and may be caused by misconfigurations, power outages, extreme weather, or infrastructure damage. Note that such outages are distinct from state-imposed shutdowns that also happen all too frequently, generally used to deal with situations of unrest, elections or even exams.

On the morning of January 4, 2022, citizens of The Gambia woke up to a country-wide Internet outage. Gamtel (the main state-owned telecommunications company of the West Africa country), announced that it happened due to "technical issues on the backup links" — we elaborate more on this below.

Cloudflare Radar shows that the outage had a significant impact on Internet traffic in the country and started after 01:00 UTC (which is the same local time), lasting until ~09:45 — a disruption of over 8 hours.

Looking at  BGP (Border Gateway Protocol) updates from Gambian ASNs around the time of the outage, we see a clear spike at 01:10 UTC. These update messages are BGP signaling that the Gambian ASNs are no longer routable.

It is important to know that BGP is a mechanism to exchange routing information between autonomous systems (networks) on the Internet. The routers that make the Internet work have huge, constantly updated lists of the possible routes that can be used to deliver every network packet to their final destinations. Without BGP, the Internet routers wouldn't know what to do, and the Internet wouldn't work. As we saw in our blog post in 2021 about how Facebook disappeared from the Internet, the Internet is literally a network of networks, and it’s bound together by BGP.

The Gambia’s Internet access is solely dependent on a single provider, Gamtel. Because The Gambia’s international Internet connectivity via the ACE submarine cable was unavailable, it was reliant on the “backup links” referenced above - terrestrial connectivity via Senegal and the provider Sonatel. This is visible in BGP data. If we look at the ASNs that are allocated to networks in The Gambia (AS25250, AS37309, AS37503, AS37552, AS37524, AS37323, AS328488, AS328140), and put those into a regular expression on BGP routing tools like route-views as so:

route-views>show ip bgp regexp .*_(25250|37309|37503|37552|37524|37323|328488|328140)

We are able to see all the possible upstream ASN paths from these networks to the rest of the Internet.

Looking at the “Path” results, we see that AS8346 (Sonatel) and AS25250 (Gamtel) are in the path for all the Gambian networks.

Visualized, you can see the dependency on this network path for The Gambia’s Internet access.

No interruptions were seen in Sonatel (AS8346), so this indicates that the single network path between Sonatel and Gamtel (AS25250) is a critical point for connectivity. A failure in either of these networks could result in The Gambia going offline again.

Yesterday’s outage in The Gambia outage illustrates something we frequently reference here in the blog: the Internet is literally a network of networks. A significant amount of  Internet traffic is carried by a complex network of undersea fiber-optic cables that connect countries and continents — all the cable systems used have landing points in two or more countries. So a problem in one country can easily affect others.

Going back to The Gambia, Gamtel explained in a January 5, 2022, press release that there was “a primary link failure at ACE” — the cable system that serves 24 countries, from Europe to Africa. “The ACE cable repair is expected to be completed in mid-January, 2022,” explained the company.

The full ACE (Africa Coast to Europe) submarine cable system. From NSRC

The “backup failure” here was “due to a faulty card at Toubakota, in Senegal”. That problem affects “both the Karang and Seleti links [points of cable connections from Senegal to The Gambia] as both North and South links converges there”. “Thus, the reason for the complete isolation on the Sonatel link”, concludes Gamtel.

Recognizing the critical importance of reliable Internet connectivity, The Gambia Public Utilities Regulatory Authority also issued a statement noting “The Authority, operators, MOICI, and the Government are exploring other options of making sure that the Gambia has a second fibre cable backup considering the impact that these failures are having on our national security, economy, and social activities.”

In addition to our physical locations, our customers can now interconnect with us at any of 23 metro areas across five continents using software-defined layer 2 networking technology. Following the recent release of CNI (which includes PNI support for Magic Transit), customers can now order layer 3 DDoS protection in any of the markets below, without requiring physical cross connects, providing private and secure links, with simpler setup.

We’re very excited to announce that five of the world's premier interconnect platforms are available at launch. Console Connect by PCCW Global in 14 locations, Megaport in 14 locations, PacketFabric in 15 locations, Equinix ECX Fabric™ in 8 locations and Zayo Tranzact in 3 locations, spanning North America, Europe, Asia, Oceania and Africa.

What is an Interconnection Platform?

Like much of the networking world, there are many terms in the interconnection space for the same thing: Cloud Exchange, Virtual Cross Connect Platform and Interconnection Platform are all synonyms. They are platforms that allow two networks to interconnect privately at layer 2, without requiring additional physical cabling. Instead the customer can order a port and a virtual connection on a dashboard, and the interconnection ‘fabric’ will establish the connection. Since many large customers are already connected to these fabrics for their connections to traditional Cloud providers, it is a very convenient method to establish private connectivity with Cloudflare.

Cloudflare has an extensive peering infrastructure and already has private links to thousands of other networks. Virtual private interconnection is particularly attractive to customers with strict security postures and demanding performance requirements, but without the added burden of ordering and managing additional physical cross connects and expanding their physical infrastructure.

SecureSimilar to physical PNI, traffic does not pass across the Internet. Rather, it flows from the customer router, to the Interconnection Platform’s network and ultimately to Cloudflare. So while there is still some element of shared infrastructure, it’s not over the public Internet.

EfficientModern PNIs are typically a minimum of 1Gbps, but if you have the security motivation without the sustained 1Gbps data transfer rates, then you will have idle capacity. Virtual connections provide for “sub-rate” speeds, which means less than 1Gbps, such as 100Mbps, meaning you only pay for what you use. Most providers also allow some level of “burstiness”, which is to say you can exceed that 100Mbps limit for short periods.

PerformanceBy avoiding the public Internet, virtual links avoid Internet congestion.

PriceThe major cloud providers typically have different pricing for egressing data to the Internet compared to an Interconnect Platform. By connecting to your cloud via an Interconnect Partner, you can benefit from those reduced egress fees between your cloud and the Interconnection Platform. This builds on our Bandwidth Alliance to give customers more options to continue to drive down their network costs.

Less OverheadBy virtualizing, you reduce physical cable management to just one connection into the Interconnection Platform. From there, everything defined and managed in software. For example, ordering a 100Mbps link to Cloudflare can be a few clicks in a Dashboard, as would be a 100Mbps link into Salesforce.

Data Center IndependenceIs your infrastructure in the same metro, but in a different facility to Cloudflare? An Interconnection Platform can bring us together without the need for additional physical links.

1. In any of our physical facilities

2. In any of the 23 metro areas where we are currently connected to an Interconnection Platform (see below)

3. If you’d like to connect virtually in a location not yet listed below, simply get in touch via our interconnection page and we’ll work out the best way to connect.

The metro areas below have currently active connections. New providers and locations can be turned up on request.

Our customers have been asking for direct on-ramps to our global network for a long time and we’re excited to deliver that today with both physical and virtual connectivity of the world’s leading interconnection Platforms.

Already a Cloudflare customer and connected with one of our Interconnection partners? Then contact your account team today to get connected and benefit from improved reliability, security and privacy of Cloudflare Network Interconnect via our interconnection partners.

Are you an Interconnection Platform with customers demanding direct connectivity to Cloudflare? Head to our partner program page and click “Become a partner”. We’ll continue to add platforms and partners according to customer demand.

"Equinix and Cloudflare share the vision of software-defined, virtualized and API-driven network connections. The availability of Cloudflare on the Equinix Cloud Exchange Fabric demonstrates that shared vision and we’re excited to offer it to our joint customers today."– Joseph Harding, Equinix, Vice President, Global Product & Platform MarketingSoftware Developer

"Cloudflare and Megaport are driven to offer greater flexibility to our customers. In addition to accessing Cloudflare’s platform on Megaport’s global internet exchange service, customers can now provision on-demand, secure connections through our Software Defined Network directly to Cloudflare Network Interconnect on-ramps globally. With over 700 enabled data centres in 23 countries, Megaport extends the reach of CNI onramps to the locations where enterprises house their critical IT infrastructure. Because Cloudflare is interconnected with our SDN, customers can point, click, and connect in real time. We’re delighted to grow our partnership with Cloudflare and bring CNI to our services ecosystem — allowing customers to build multi-service, securely-connected IT architectures in a matter of minutes."– Matt Simpson, Megaport, VP of Cloud Services

“The ability to self-provision direct connections to Cloudflare’s network from Console Connect is a powerful tool for enterprises as they come to terms with new demands on their networks. We are really excited to bring together Cloudflare’s industry-leading solutions with PCCW Global’s high-performance network on the Console Connect platform, which will deliver much higher levels of network security and performance to businesses worldwide.”– Michael Glynn, PCCW Global, VP of Digital Automated Innovation

"Our customers can now connect to Cloudflare via a private, secure, and dedicated connection via the PacketFabric Marketplace. PacketFabric is proud to be the launch partner for Cloudflare's Interconnection program. Our large U.S. footprint provides the reach and density that Cloudflare customers need."– Dave Ward, PacketFabric CEO

Our logs show that at 21:12 UTC on Monday, a Nigerian ISP, MainOne, accidentally misconfigured part of their network causing a "route leak". This resulted in Google and a number of other networks being routed over unusual network paths. Incidents like this actually happen quite frequently, but in this case, the traffic flows generated by Google users were so great that they overwhelmed the intermediary networks — resulting in numerous services (but predominantly Google) unreachable.

You might be surprised to learn that an error by an ISP somewhere in the world could result in Google and other services going offline. This blog post explains how that can happen and what the Internet community is doing to try to fix this fragility.

When traffic is routed outside of regular and optimal routing paths, this is known as a “route leak”. An explanation of how they happen requires a little bit more context.

Every network and network provider on the Internet has their own Autonomous System (AS) number. This number is unique and indicates the part of the Internet that that organization controls. Of note for the following explanation Google’s primary AS Number is 15169. That's Google's corner of the Internet and where Google traffic should end up... by the fastest path.

A Typical view of how Google/AS15169’s routes are propagated to Tier-1 Networks.

As seen above, Google is directly connected to most of the Tier-1 networks (the largest networks link large swathes of the Internet). When everything is working as it should be, Google’s AS Path, the route packets take from network to network to reach their destination, is actually very simple. For example, in the diagram above, if you were a customer of Cogent and you were trying to get to Google, the AS Path that you would see is “174 6453 15169”. That string of numbers is like a sequence of waypoints: start on AS 174 (Cogent), go to Tata (AS 6453), then go to Google (AS 15169). So, Cogent subscribers reach Google via Tata, a huge Tier-1 provider.

During the incident, MainOne misconfigured their routing as reflected in the AS Path : “20485 4809 37282 15169”. As a result of this misconfiguration, any networks that MainOne peered with (i.e. were directly connected to) potentially had their routes leaked through this erroneous path. For example, the Cogent customer in the paragraph above (AS 174) wouldn’t have gone via Tata (AS 6453) as they should have. Instead, they were routed first through TransTelecom (a Russian Carrier, AS 20485), then to China Telecom CN2 (a cross border Chinese carrier, AS 4809), then on to MainOne (the Nigerian ISP that misconfigured everything, AS 37282), and only then were they finally handed off to Google (AS 15169). In other words,  a user in London could have had their traffic go from Russia to China to Nigeria — and only then got to Google.

The root cause of this was MainOne misconfiguring their routing. As mentioned earlier, incidents like this actually happen quite frequently. The impact of this misconfiguration should have been limited to MainOne and its customers.

However, what took this from relatively isolated and turned it into a much broader one is because CN2 — China Telecom’s premium cross-border carrier — was not filtering the routing that MainOne provided to them. In other words, MainOne told CN2 that it had authority to route Google’s IP addresses. Most networks verify this, and if it is incorrect, filter it out. CN2 did not — it simply trusted MainOne. As a result of this, MainOne’s misconfiguration propagated to a substantially larger network. Compounding this, it is likely that the Russian network TransTelecom behaved similarly towards CN2 as CN2 had behaved towards MainOne — they trusted without any verification of the routing paths that CN2 gave to them.

This demonstrates how much trust is involved in the underlying connections that make up the Internet. It's a network of networks (an internet!) that works by cooperation between different entities (countries and companies).

This is how a routing mistake made in Nigeria then propagated through China and then through Russia. Given the amount of traffic involved, the networks were overwhelmed and Google was unreachable.

It is worth explicitly stating: the fact that Google traffic was routed through Russia and China before going getting to Nigeria and only then hitting the correct destination made it appear to some people that the misconfiguration was nefarious. We do not believe this to be the case. Instead, this incident reflects a mistake that was not caught by appropriate network filtering. There was too much trust and not enough verification across a number of networks: this is a systemic problem that makes the Internet more vulnerable to mistakes than it should be.

Along with Cloudflare’s internal systems, BGPMon.net and ThousandEyes detected the incident. BGPMon has several methods to detect abnormalities; in this case, they were able to detect that the providers in the paths to reach Google were irregular.

Cloudflare’s systems immediately detected this and took auto-remediation action.

Thankfully our systems detected it and mitigated it! pic.twitter.com/qFiDkrn2Kd

— Jerome Fleury (@Jerome_UZ) November 13, 2018

Some more information about Cloudflare’s Auto Remediation system: https://blog.cloudflare.com/the-internet-is-hostile-building-a-more-resilient-network/

Cloudflare is committed to working with others to drive more secure and stable routing. We recently wrote about RPKI and how we’ll start enforcing “Strict” RPKI validation, and will continue to strive for secure internet routing.  We hope that all networks providing transit services can ensure proper filtering of their customers, end hijacks and route-leaks and implement best practices like BCP-38.

Virginia has a very important place in Internet history, as well as the history of Cloudflare’s network.

Northern Virginia, in the area around Ashburn VA, has for a long time been core to Internet infrastructure. In the early 1990’s, MAE-East (Metropolitan-Area-Exchange East) , an Internet Exchange Point (IXP) was established. MAE-East and West were some of the earliest IXPs. Internet Exchange Points are crucial interconnection points for ISPs and other Internet Networks to interconnect and exchange traffic. Eco-systems have grown around these through new data center offerings and new Internet platforms. Like many pieces of the Internet, MAE-East had a humble beginning, though not many humble beginnings grew to handle around 50% of Internet traffic exchange.

Cloudflare’s second Data Center, and one that still plays a critical component in our Global Network was Ashburn, Virginia. Similarly across many organizations, the Northern Virginia area has become a Data Center mecca. Many of the largest Clouds have a substantial amount of their footprint in Northern Virginia. Although MAE-East no longer exists, other Internet Exchange Points have come and grown in its place.

Cloudflare’s network has grown beyond what was traditional Interconnection points, like Ashburn/Northern VA, to a new Edge of the network. Cloudflare will continue to grow its Edge closer to every end-user, so today we’re announcing our Richmond and Norfolk data centers. These two data centers will cover much more of Virginia and neighboring regions.

Photo by chuttersnap / Unsplash

At the end of 2017, Xinhua reported that there will be 200 Million IPv6 users inside Mainland China by the end of this year. Halfway into the year, we’re seeing a rapid growth in IPv6 users and traffic originating from Mainland China.

IPv6 is often referred to the next generation of IP addressing. The reality is, IPv6 is what is needed for addressing today. Taking the largest mobile network in China today, China Mobile has over 900 Million mobile subscribers and over 670 Million 4G/LTE subscribers. To be able to provide service to their users, they need to provide an IP address to each subscriber’s device. This means close to a billion IP addresses would be required, which is far more than what is available in IPv4, especially as the available IP address pools have been exhausted.

To solve the addressability of clients, many networks, especially mobile networks, will use Carrier Grade NAT (CGN). This allows thousands, possibly up to hundreds of thousands, of devices to be shared behind a single internet IP address. The CGN equipment can be very expensive to scale and further, given the scale of the networks, they might need to layer CGNs behind other CGNs. This increases costs per subscriber, can reduce performance and makes scaling very challenging. A further solution, NAT64, allows IPv6 addresses to be given to subscribers, but then translated to IPv4 addresses similar to other NATs. This allows networks and ISPs to begin deploying IPv6 to subscribers, a first step in transition to IPv6.

Announcements IPv6 address blocks from China Mobile. Source: Hurricane Electric

On June 7, China Mobile started to announce IPv6 address blocks to the Internet at large. At the same time, Cloudflare started seeing traffic being exchanged with China Mobile users over IPv6 connections.

IPv4 to IPv6 percentage of traffic as seen from Cloudflare to AS9808 China Mobile’s Guangdong network.

Throughout the past 45 days, we’ve seen more and more IPv6 address blocks being announced to the internet, along with very aggressive usage. Interestingly this all started on-or-around June 8th 2018 (seven years to the day from World IPv6 Day)

It’s natural to see traffic graphs like this go up; then down after a while. This could indicate there’s some testing still going on with the deployment. We fully expect that the traffic percentage will climb back up once this is fully rolled out.

It’s fantastic to see the IPv6 enablement! We congratulate China Mobile on their successful enablement going forward.

Cebu City, the second most populous metro area, but oldest city in the Philippines is the home of Cloudflare’s newest Data Center.

Located centrally in the Philippines, Cebu has had a long standing tradition of trade and business activity, the word itself “Cebu” meaning trade. It’s central location brings excellent coverage to central and southern Philippines, while our existing location in Manila, serving the Manila Metro and northern Philippines.

Photo by Zany Jadraque / Unsplash

Cebu’s history covers hundreds of years, with rich local culture and international influence dating back from the first Spanish visitors to modern trade and shipping. One of the more popular dishes is Lechon.

Cebu is known for its white sand beaches. In between making millions of websites and applications faster and more secure for the Philippine internet users, we hope our servers can get some excellent R&R on the famous beaches.

This map reflects the network as of the publish date of this blog post. For the most up to date directory of locations please refer to our Network Map on the Cloudflare site.

Cloudflare's 127th data center is now live in Macau, helping make over 7 million Internet facing applications safer and faster. This is our 44th data center in Asia.

Cloudflare 將在澳門啟用全球第127個數據中心, 幫助超過 7,000,000 客戶的互聯網資產運行得更快、更安全。澳門也是我們在亞洲的第44個數據中心。

O 127º centro de dados da Cloudflare agora está em funcionamento em Macau, ajudando a tornar mais de 7 milhões de aplicações voltadas para a Internet de forma mais segura e rápida. Estamos felizes em compartilhar que este é o nosso 44º centro de dados na Ásia.

_CC BY-NC-ND 2.0 image by kidchen915_

Blending Chinese and Portuguese culture, just last year, Macau welcomed over 30 million visitors. Visit Macau to experience its unique and extravagant entertainment scene, see scenic spots such as the Ruins of St Paul, Senado Square, attempt the world's highest bungy jump from Macau Tower, or enjoy the foodie paradise Macau delivers!

有著與眾不同的中國及葡萄牙文化融合景觀，澳門至去年為止已經吸引了三千萬遊客來一睹她的風采。你可以拜訪著名的娛樂景觀，像是聖保祿大教堂遺址，議事亭廣場，挑戰澳門的美食，亦可以選擇從全世界最高的高空彈跳地點ㄧ澳門旅遊塔上一躍而下。

Combinando cultura chinesa e portuguesa, o ano passado, Macau recebeu mais de 30 milhões de visitantes. Recomendamos que visite Macau para experimentar a cena de entretenimento única e extravagante, explore os pontos cénica, como ruínas de São Paulo, Praça do Senado, e ainda, tente o "bungee jumping" mais alto do mundo deste da Torre de Macau, ou aproveite o paraíso da gastronomia que Macau oferece!

While Macau and it's bigger sister across the Pearl River, Hong Kong share many similarities, one big difference is Egg Tarts. Egg tarts are suggested to have been introduced into Hong Kong in the 1940's while Macau favors the portuguese style which dates back hundreds of years. Both have been adapted for local flavors and both are wonderful! Which do you like best?

雖然澳門以及香港有著許多相似之處，他們對於蛋塔的喜好卻有相當大的不同。話說蛋塔大約於1940年代左右才被引入香港，而澳門當地人偏好的葡式蛋塔卻已有百年歷史了。兩地都將他們對於蛋塔的喜好昇華成不可或缺的美食。您喜歡哪種多一點呢？

Enquanto Macau e a irmã a sua irmã do outro lado do rio das pérolas, Hong Kong, ambas compartilham muitas semelhanças, uma grande diferença seriam algumas especialidades culinárias. Os deliciosos pastéis de nata, foram introduzidos em Hong Kong na década de 1940, enquanto Macau favorece o estilo português que remonta a centenas de anos. Ambos foram adaptados para sabores locais e ambos são maravilhosos! Qual é o seu favorito (pastel de nata)?

Next week, we'll announce even more cities around the globe added to the Cloudflare global network.

澳門只是個起點！從下週起，我們將會逐漸宣佈更多城市加入 Cloudflare 的全球網路。

Na próxima semana, anunciaremos ainda mais localidades ao redor do mundo a serem adicionadas à rede global Anycast da Cloudflare.

Most of the island was hit with power outages, shortages and rolling blackouts, with street lights not functioning, nor power in many of Taipei’s shopping malls, and much other infrastructure.

Blackouts of this scale are very rare. Usually, during an outage of this scale, it would be expected that Internet traffic would greatly drop, as houses and businesses lose power and are unable to connect to the Internet. I’ve experienced this in the past, working at consumer ISPs. As households and businesses lose power, so do their modems or routers which connect them to the Internet.

However, during yesterday's outage, something different happened. I'd like to share some insights from yesterday's outage.

Photo: Taipei 101 Dark during the Blackout -Source: David Chang/EPA

Most Telecom and Data Center facilities are built with redundancy in mind and have backup power generation. Our Data Center partner, Chief, was able to switch to backup power generation without any service interruption, allowing our service to operate without interruption.

The lack of interruption was also reflected by many users still accessing the Internet. From our statistics, the number of requests didn’t drop, as illustrated by the graph following. At the beginning of the power outage, there was actually a spike in requests, as more people likely look on the Internet as to more details of what's happening. The graph below shows a timeline of requests per second, seen in our Taipei data center, with a red line marking the beginning of the power outage.

Breaking down traffic between Mobile and Desktop clients, approximately 10% of clients shifted from Desktop to Mobile devices at the beginning of the outage. The graph also shows a spike daily around lunch time, as many clients shift to their mobile phones during lunch.

The shift to to mobile devices did however cause a drop in bandwidth used, by approximately 25%. The following graph showing our bandwidth usage to HiNet, the largest ISP, demonstrates this sharp drop.

Power was fully restored around 21:40pm, (13:40 UTC), however many users were able to regain access during power-rationing and Internet usage grew to reach its usual night-time peaks.

This power outage taught us that Internet usage does not necessarily decrease during a power outage in 2017. Number of requests can actually increase, but bandwidth usage drops, reflecting a shift to usage of mobile devices.

Whilst the entire city lived in darkness, the Internet shined bright!

At around 6:24pm PST / 02:24 UTC (5 Nov. 2012 PST / 6 Nov. 2012 UTC), CloudFlare employees noticed that Google's services were offline. We use Google Apps for things like email so when we can't reach their servers the office notices quickly. I'm on the Network Engineering team so I jumped online to figure out if the problem was local to us or global.

I quickly realised that we were unable to resolve all of Googles services — or even reach 8.8.8.8, Googles public DNS server — so I started troubleshooting DNS.

$ dig +trace google.com

Here's the response I got when I tried to reach any of Google.com's name servers:

google.com. 172800 IN NS ns2.google.com.google.com. 172800 IN NS ns1.google.com.google.com. 172800 IN NS ns3.google.com.google.com. 172800 IN NS ns4.google.com.;; Received 164 bytes from 192.12.94.30#53(e.gtld-servers.net) in 152 ms;; connection timed out; no servers could be reached

The fact that no servers could be reached means something was wrong. Specifically, it meant that from our office network we were unable to reach any of Googles DNS servers.

I started to look at the network layer, see if that's where the problems lay.

PING 216.239.32.10 (216.239.32.10): 56 data bytesRequest timeout for icmp_seq 092 bytes from 1-1-15.edge2-eqx-sin.moratelindo.co.id (202.43.176.217): Time to live exceeded

That was curious. Normally, we shouldn't be seeing an Indonesian ISP (Moratel) in the path to Google. I jumped on one of CloudFlare's routers to check what was going on. Meanwhile, others reports from around the globe on Twitter suggested we weren't the only ones seeing the problem.

To understand what went wrong you need to understand a bit about how networking on the Internet works. The Internet is a collection of networks, known as "Autonomous Systems" (AS). Each network has a unique number to identify it known as AS number. CloudFlare's AS number is 13335, Google's is 15169. The networks are connected together by what is known as Border Gateway Protocol (BGP). BGP is the glue of the Internet — announcing what IP addresses belong to each network and establishing the routes from one AS to another. An Internet "route" is exactly what it sounds like: a path from the IP address on one AS to an IP address onanother AS.

BGP is largely a trust-based system. Networks trust each other to say which IP addresses and other networks are behind them. When you send a packet or make a request across the network, your ISP connects to its upstream providers or peers and finds the shortest path from your ISP to the destination network.

Unfortunately, if a network starts to send out an announcement of a particular IP address or network behind it, when in fact it is not, if that network is trusted by its upstreams and peers then packets can end up misrouted. That is what was happening here.

I looked at the BGP Routes for a Google IP Address. The route traversed Moratel (23947), an Indonesian ISP. Given that I'm looking at the routing from California and Google is operating Data Centre's not far from our office, packets should never be routed via Indonesia. The most likely cause was that Moratel was announcing a network that wasn't actually behind them.

The BGP Route I saw at the time was:

tom@edge01.sfo01> show route 216.239.34.10 inet.0: 422168 destinations, 422168 routes (422154 active, 0 holddown, 14 hidden)+ = Active Route, - = Last Active, * = Both216.239.34.0/24 *[BGP/170] 00:15:47, MED 18, localpref 100 AS path: 4436 3491 23947 15169 I > to 69.22.153.1 via ge-1/0/9.0

Looking at other routes, for example to Google's Public DNS, it was also stuck routing down the same (incorrect) path:

tom@edge01.sfo01> show route 8.8.8.8 inet.0: 422196 destinations, 422196 routes (422182 active, 0 holddown, 14 hidden)+ = Active Route, - = Last Active, * = Both8.8.8.0/24 *[BGP/170] 00:27:02, MED 18, localpref 100 AS path: 4436 3491 23947 15169 I > to 69.22.153.1 via ge-1/0/9.0

(Image Credit: The Simpsons)

Situations like this are referred to in the industry as "route leakage", as the route has "leaked" past normal paths. This isn't an unprecedented event. Google previously suffered a similar outage when Pakistan was allegedly trying to censor a video on YouTube and the National ISP of Pakistan null routed the service's IP addresses. Unfortunately, they leaked the null route externally. Pakistan Telecom's upstream provider, PCCW, trusted what Pakistan Telecom's was sending them and the routes spread across the Internet. The effect was YouTube was knocked offline for around 2 hours.

The case today was similar. Someone at Moratel likely "fat fingered" an Internet route. PCCW, who was Moratel's upstream provider, trusted the routes Moratel was sending to them. And, quickly, the bad routes spread. It is unlikely this was malicious, but rather a misconfiguaration or an error evidencing some of the failings in the BGP Trust model.

The solution was to get Moratel to stop announcing the routes they shouldn't be. A large part of being a network engineer, especially working at a large network like CloudFlare's, is having relationships with other network engineers around the world. When I figured out the problem, I contacted a colleague at Moratel to let him know what was going on. He was able to fix the problem at around 2:50 UTC / 6:50pm PST. Around 3 minutes later, routing returned to normal and Google's services came back online.

Looking at peering maps, I'd estimate the outage impacted around 3–5% of the Internet's population. The heaviest impact will have been felt in Hong Kong, where PCCW is the incumbent provider. If you were in the area and unable to reach Google's services around that time, now you know why.

This all is a reminder about how the Internet is a system built on trust. Today's incident shows that, even if you're as big as Google, factors outside of your direct control can impact the ability of your customers to get to your site so it's important to have a network engineering team that is watching routes and managing your connectivity around the clock. CloudFlare works every day to ensure our customers get the optimal possible routes. We look out for all the websites on our network to ensure that their traffic is always delivered as fast as possible. Just another day in our ongoing efforts to #savetheweb.

Moratel says the issue was caused by an unexpected hardware failure, causing this abnormal condition. This was not a malicious attempt. Moratel immediately shutdown the BGP peering with Google after contact was made while the hardware failure was being looked into.

Thanks for reading all the way to the end. If you enjoyed this post, take a second to learn more about CloudFlare or nominate us for the 2012 Crunchie Award for Best TechnicalInnovation.

CloudFlare is now connected to HKIX (Hong Kong Internet Exchange). HKIX is the largest Internet Exchange in Asia transferring around 200Gbit to its 160 ISP, Carrier and Content networks. HKIX is fundamental to the local Hong Kong ISP Market, with every provider in Hong Kong connected, as well as excellent regional coverage with networks from Thailand to Japan to Australia.

So, what does a connection to HKIX mean for CloudFlare users? Faster performance for your Hong Kong web surfers. Being a part of the HKIX allows us to deliver traffic to all Hong Kong web surfers within Hong Kong. This will improve web browsing performance by keeping the traffic local and the latency low.

Connecting to HKIX is one of the steps CloudFlare is working on to bring the Internet closer to you. Watch for more news to come over 2012!