The issue was not caused, directly or indirectly, by a cyberattack or malicious activity of any kind. This issue was caused by a change that Cloudflare made to how our network manages IP addresses onboarded through the BYOIP pipeline. This change caused Cloudflare to unintentionally withdraw customer prefixes.

For some BYOIP customers, this resulted in their services and applications being unreachable from the Internet, causing timeouts and failures to connect across their Cloudflare deployments that used BYOIP. The website for Cloudflare’s recursive DNS resolver (1.1.1.1) saw 403 errors as well. The total duration of the incident was 6 hours and 7 minutes with most of that time spent restoring prefix configurations to their state prior to the change.

Cloudflare engineers reverted the change and prefixes stopped being withdrawn when we began to observe failures. However, before engineers were able to revert the change, ~1,100 BYOIP prefixes were withdrawn from the Cloudflare network. Some customers were able to restore their own service by using the Cloudflare dashboard to re-advertise their IP addresses. We resolved the incident when we restored all prefix configurations.

We are sorry for the impact to our customers. We let you down today. This post is an in-depth recounting of exactly what happened and which systems and processes failed. We will also outline the steps we are taking to prevent outages like this from happening again.

This graph shows the amount of prefixes advertised by Cloudflare during the incident to a BGP neighbor, which correlates to impact as prefixes that weren’t advertised were unreachable on the Internet:

Out of the total 6,500 prefixes advertised to this peer, 4,306 of those were BYOIP prefixes. These BYOIP prefixes are advertised to every peer and represent all the BYOIP prefixes we advertise globally. 

During the incident, 1,100 prefixes out of the total 6,500 were withdrawn from 17:56 to 18:46 UTC. Out of the 4,306 total BYOIP prefixes, 25% of BYOIP prefixes were unintentionally withdrawn. We were able to detect impact on one.one.one.one and revert the impacting change before more prefixes were impacted. At 19:19 UTC, we published guidance to customers that they would be able to self-remediate this incident by going to the Cloudflare dashboard and re-advertising their prefixes.

Cloudflare was able to revert many of the advertisement changes around 20:20 UTC, which caused 800 prefixes to be restored. There were still ~300 prefixes that were unable to be remediated through the dashboard because the service configurations for those prefixes were removed from the edge due to a software bug. These prefixes were manually restored by Cloudflare engineers at 23:03 UTC. 

This incident did not impact all BYOIP customers because the configuration change was applied iteratively and not instantaneously across all BYOIP customers. Once the configuration change was revealed to be causing impact, the change was reverted before all customers were affected. 

The impacted BYOIP customers first experienced a behavior called BGP Path Hunting. In this state, end user connections traverse networks trying to find a route to the destination IP. This behavior will persist until the connection that was opened times out and fails. Until the prefix is advertised somewhere, customers will continue to see this failure mode. This loop-until-failure scenario affected any product that uses BYOIP for advertisement to the Internet. Additionally, visitors to one.one.one.one, the website for Cloudflare’s recursive DNS resolver, were met with HTTP 403 errors and an “Edge IP Restricted” error message. DNS resolution over the 1.1.1.1 Public Resolver, including DNS over HTTPS, was not affected. A full breakdown of the services impacted is below.

There was also a set of customers who were unable to restore service by toggling the prefixes on the Cloudflare dashboard. As engineers began reannouncing prefixes to restore service for these customers, these customers may have seen increased latency and failures despite their IP addresses being advertised. This was because the addressing settings for some users were removed from edge servers due an issue in our own software, and the state had to be propagated back to the edge. 

We’re going to get into what exactly broke in our addressing system, but to do that we need to cover a quick primer on the Addressing API, which is the underlying source of truth for customer IP addresses at Cloudflare.

The Addressing API is an authoritative dataset of the addresses present on the Cloudflare network. Any change to that dataset is immediately reflected in Cloudflare's global network. While we are in the process of improving how these systems roll out changes as a part of Code Orange: Fail Small, today customers can configure their IP addresses by interacting with public-facing APIs which configure a set of databases that trigger operational workflows propagating the changes to Cloudflare’s edge. This means that changes to the Addressing API are immediately propagated to the Cloudflare edge.

Advertising and configuring IP addresses on Cloudflare involves several steps:

• Customers signal to Cloudflare about advertisement/withdrawal of IP addresses via the Addressing API or BGP Control

• The Addressing API instructs the machines to change the prefix advertisements

• BGP will be updated on the routers once enough machines have received the notification to update the prefix

• Finally, customers can configure Cloudflare products to use BYOIP addresses via service bindings which will assign products to these ranges

The Addressing API allows us to automate most of the processes surrounding how we advertise or withdraw addresses, but some processes still require manual actions. These manual processes are risky because of their close proximity to Production. As a part of Code Orange: Fail Small, one of the goals of remediation was to remove manual actions taken in the Addressing API and replace them with safe workflows.

The specific piece of configuration that broke was a modification attempting to automate the customer action of removing prefixes from Cloudflare’s BYOIP service, a regular customer request that is done manually today. Removing this manual process was part of our Code Orange: Fail Small work to push all changes toward safe, automated, health-mediated deployment. Since the list of related objects of BYOIP prefixes can be large, this was implemented as part of a regularly running sub-task that checks for BYOIP prefixes that should be removed, and then removes them. Unfortunately, this regular cleanup sub-task queried the API with a bug.

Here is the API query from the cleanup sub-task:

 resp, err := d.doRequest(ctx, http.MethodGet, `/v1/prefixes?pending_delete`, nil)

And here is the relevant part of the API implementation:

	if v := req.URL.Query().Get("pending_delete"); v != "" {
		// ignore other behavior and fetch pending objects from the ip_prefixes_deleted table
		prefixes, err := c.RO().IPPrefixes().FetchPrefixesPendingDeletion(ctx)
		if err != nil {
			api.RenderError(ctx, w, ErrInternalError)
			return
		}

		api.Render(ctx, w, http.StatusOK, renderIPPrefixAPIResponse(prefixes, nil))
		return
	}

Because the client is passing pending_delete with no value, the result of Query().Get(“pending_delete”) here will be an empty string (“”), so the API server interprets this as a request for all BYOIP prefixes instead of just those prefixes that were supposed to be removed. The system interpreted this as all returned prefixes being queued for deletion. The new sub-task then began systematically deleting all BYOIP prefixes and all of their related dependent objects including service bindings, until the impact was noticed, and an engineer identified the sub-task and shut it down.

Our staging environment contains data that matches Production as closely as possible, but was not sufficient in this case and the mock data we relied on to simulate what would occur was insufficient.

In addition, while we have tests for this functionality, coverage for this scenario in our testing process and environment was incomplete. Initial testing and code review focused on the BYOIP self-service API journey and were completed successfully. While our engineers successfully tested the exact process a customer would have followed, testing did not cover a scenario where the task-runner service would independently execute changes to user data without explicit input.

Affected BYOIP prefixes were not all impacted in the same way, necessitating more intensive data recovery steps. As a part of Code Orange: Fail Small, we are building a system where operational state snapshots can be safely rolled out through health-mediated deployments. In the event something does roll out that causes unexpected behavior, it can be very quickly rolled back to a known-good state. However, that system is not in Production today.

BYOIP prefixes were in different states of impact during this incident, and each of these different states required different actions:

• Most impacted customers only had their prefixes withdrawn. Customers in this configuration could go into the dashboard and toggle their advertisements, which would restore service. 

• Some customers had their prefixes withdrawn and some bindings removed. These customers were in a partial state of recovery where they could toggle some prefixes but not others.

• Some customers had their prefixes withdrawn and all service bindings removed. They could not toggle their prefixes in the dashboard because there was no service (Magic Transit, Spectrum, CDN) bound to them. These customers took the longest to mitigate, as a global configuration update had to be initiated to reapply the service bindings for all these customers to every single machine on Cloudflare’s edge.

The change we were making when this incident occurred is part of the Code Orange: Fail Small initiative, which is aimed at improving the resiliency of code and configuration at Cloudflare. As a brief primer of the Code Orange: Fail Small initiatives, the work can be divided into three buckets:

• Require controlled rollouts for any configuration change that is propagated to the network, just like we do today for software binary releases.

• Change our internal “break glass” procedures and remove any circular dependencies so that we, and our customers, can act fast and access all systems without issue during an incident.

• Review, improve, and test failure modes of all systems handling network traffic to ensure they exhibit well-defined behavior under all conditions, including unexpected error states.

The change that we attempted to deploy falls under the first bucket. By moving risky, manual changes to safe, automated configuration updates that are deployed in a health-mediated manner, we aim to improve the reliability of the service.

Critical work was already ongoing to enhance the Addressing API's configuration change support through staged test mediation and better correctness checks. This work was ongoing in parallel with the deployed change. Although preventative measures weren't fully deployed before the outage, teams were actively working on these systems when the incident occurred. Following our Code Orange: Fail Small promise to require controlled rollouts of any change into Production, our engineering teams have been reaching deep into all layers of our stack to identify and fix all problematic findings. While this outage wasn't itself global, the blast radius and impact were unacceptably large, further reinforcing Code Orange: Fail Small as a priority until we have re-established confidence in all changes to our network being as gradual as possible. Now let’s talk more specifically about improvements to these systems.

One of the issues in this incident is that the pending_delete flag was interpreted as a string, making it difficult for both client and server to rationalize the value of the flag. We will improve the API schema to ensure better standardization, which will make it much easier for testing and systems to validate whether an API call is properly formed or not. This work is part of the third Code Orange workstream, which aims to create well-defined behavior under all conditions.

Today, customers make changes to the addressing schema that are persisted in an authoritative database, and that database is the same one used for operational actions. This makes manual rollback processes more challenging because engineers need to utilize database snapshots instead of rationalizing between desired and actual states. We will redesign the rollback mechanism and database configuration to ensure that we have an easy way to roll back changes quickly and also to introduce layers between customer configuration and Production.

We will snap shot the data that we read from the database and are applying to Production, and apply those snapshots in the same way that we deploy all our other Production changes, mediated by health metrics that can automatically stop the deployment if things are going wrong. This means that the next time we have a problem where the database gets changed into a bad state, we can near-instantly revert individual customers (or all customers) to a version that was working.

While this will temporarily block our customers from being able to make direct updates via our API in the event of an outage, it will mean that we can continue serving their traffic while we work to fix the database, instead of being down for that time. This work aligns with the first and second Code Orange workstreams, which involves fast rollback and also safe, health-mediated deployment of configuration.

We will improve our monitoring to detect when changes are happening too fast or too broadly, such as withdrawing or deleting BGP prefixes quickly, and disable the deployment of snapshots when this happens. This will form a type of circuit breaker to stop any out-of-control process that is manipulating the database from having a large blast radius, like we saw in this incident.

We also have some ongoing work to directly monitor that the services run by our customers are behaving correctly, and those signals can also be used to trip the circuit breaker and stop potentially dangerous changes from being applied until we have had time to investigate. This work aligns with the first Code Orange workstream, which involves safe deployment of changes.

Below is the timeline of events inclusive of deployment of the change and remediation steps:

We deeply apologize for this incident today and how it affected the service we provide our customers, and also the Internet at large. We aim to provide a network that is resilient to change, and we did not deliver on our promise to you. We are actively making these improvements to ensure improved stability moving forward and to prevent this problem from happening again.

This post is intended as a summary overview of observed and confirmed disruptions and is not an exhaustive or complete list of issues that have occurred during the quarter. These anomalies are detected through significant deviations from expected traffic patterns observed across our network. Check out the Cloudflare Radar Outage Center for a full list of verified anomalies and confirmed outages. 

The Internet was shut down in Tanzania on October 29 as violent protests took place during the country’s presidential election. Traffic initially fell around 12:30 local time (09:30 UTC), dropping more than 90% lower than the previous week. The disruption lasted approximately 26 hours, with traffic beginning to return around 14:30 local time (11:30 UTC) on October 30. However, that restoration proved to be quite brief, with a significant decrease in traffic occurring around 16:15 local time (13:15 UTC), approximately two hours after it returned. This second near-complete outage lasted until November 3, when traffic aggressively returned after 17:00 local time (14:00 UTC). Nominal drops in announced IPv4 and IPv6 address space were also observed during the shutdown, but there was never a complete loss of announcements, which would have signified a total disconnection of the country from the Internet. (Autonomous systems announce IP address space to other Internet providers, letting them know what blocks of IP addresses they are responsible for.)

Tanzania’s president later expressed sympathy for the members of the diplomatic community and foreigners residing in the country regarding the impact of the Internet shutdown. Internet and social media services were also restricted in 2020 ahead of the country’s general elections.

Digicel Haiti is unfortunately no stranger to Internet disruptions caused by cable cuts, and the network experienced two more such incidents during the fourth quarter. On October 16, traffic from Digicel Haiti (AS27653) began to fall at 14:30 local time (18:30 UTC), reaching near zero at 16:00 local time (20:00 UTC). A translated X post from the company’s Director General noted: “We advise our clientele that @DigicelHT is experiencing 2 cuts on its international fiber optic infrastructure.” Traffic began to recover after 17:00 local time (21:00 UTC), and reached expected levels within the following hour. At 17:33 local time (21:34 UTC), the Director General posted that “the first fiber on the international infrastructure has been repaired” and service had been restored. 

On November 25, another translated X post from the provider’s Director General stated that its “international optical fiber infrastructure on National Road 1” had been cut. We observed traffic dropping on Digicel’s network approximately an hour earlier, with a complete outage observed between 02:00 - 08:00 local time (07:00 - 13:00 UTC). A follow-on X post at 08:22 local time (13:22 UTC) stated that all services had been restored.

At 17:30 local time (12:30 UTC) on October 20, Internet traffic for Cybernet/StormFiber (AS9541) dropped sharply, falling to a level approximately 50% the same time a week prior. At the same time, the network’s announced IPv4 address space dropped by over a third. The cause of these shifts was damage to the PEACE submarine cable, which suffered a cut in the Red Sea near Sudan. 

PEACE is one of several submarine cable systems (including IMEWE and SEA-ME-WE-4) that carry international Internet traffic for Pakistani providers. The provider pledged to fully restore service by October 27, but traffic and announced IPv4 address space had recovered to near expected levels by around 02:00 local time on October 21 (21:00 UTC on October 20).

Unusual traffic patterns observed across multiple Internet providers in Cameroon on October 23 were reportedly caused by problems on the WACS (West Africa Cable System) submarine cable, which connects countries along the west coast of Africa to Portugal. 

A (translated) published report stated that MTN informed subscribers that “following an incident on the WACS fiber optic cable, Internet service is temporarily disrupted” and Orange Cameroun informed subscribers that “due to an incident on the international access fiber, Internet service is disrupted.” An X post from Camtel stated “Cameroon Telecommunications (CAMTEL) wishes to inform the public that a technical incident involving WACS cable equipment in Batoke (LIMBE) occurred in the early hours of 23 October 2025, causing Internet connectivity disruptions throughout the country.” 

Traffic across the impacted providers originally fell just at around  05:00 local time (04:00 UTC) before recovering to expected levels around 22:00 local time (21:00 UTC). Traffic across these networks was quite volatile during the day, dropping 90-99% at times. It isn’t clear what caused the visible spikiness in the traffic pattern—possibly attempts to shift Internet traffic to other submarine cable systems that connect to Cameroon. Announced IP address space from MTN Cameroon and Orange Cameroon dropped during this period as well, although Camtel’s announced IP address space did not change.

Connectivity in the Central African Republic and Republic of Congo was also reportedly impacted by the WACS issues.

On December 9, we saw traffic from Claro Dominicana (AS6400), an Internet provider in the Dominican Republic, drop sharply around 12:15 local time (16:15 UTC). Traffic levels fell again around 14:15 local time (18:15 UTC), bottoming out 77% lower than the previous week before quickly returning to expected levels. The connectivity disruption was likely caused by two fiber optic outages, as an X post from the provider during the outage noted that they were “causing intermittency and slowness in some services.” A subsequent post on X from Claro stated that technicians had restored Internet services nationwide by repairing the severed fiber optic cables.

According to a (translated) X post from the Empresa de Transmisión Eléctrica Dominicana (ETED), a transmission line outage caused an interruption in electrical service in the Dominican Republic on November 11. This power outage impacted Internet traffic from the country, resulting in a nearly 50% drop in traffic compared to the prior week, starting at 13:15 local time (17:15 UTC). Traffic levels remained lower until approximately 02:00 local time (06:00 UTC) on December 12, with a later (translated) X post from ETED noting “At 2:20 a.m. we have completed the recovery of the national electrical system, supplying 96% of the demand…”

A subsequent technical report found that “the blackout began at the 138 kV San Pedro de Macorís I substation, where a live line was manually disconnected, triggering a high-intensity short circuit. Protection systems responded immediately, but the fault caused several nearby lines to disconnect, separating 575 MW of generation in the eastern region from the rest of the grid. The imbalance caused major power plants to trip automatically as part of their built-in safety mechanisms.”

On December 9, a major power outage impacted multiple regions across Kenya. Kenya Power explained that the outage “was triggered by an incident on the regional Kenya-Uganda interconnected power network, which caused a disturbance on the Kenyan side of the system” and claimed that “[p]ower was restored to most of the affected areas within approximately 30 minutes.” However, impacts to Internet connectivity lasted for nearly four hours, between 19:15 - 23:00 local time (16:15 - 20:00 UTC). The power outage caused traffic to drop as much as 18% at a national level, with the traffic shifts most visible in Nakuru County and Kaimbu County.

Russian drone strikes on the Odesa region in Ukraine on December 12 damaged warehouses and energy infrastructure, with the latter causing power outages in parts of the region. Those outages disrupted Internet connectivity, resulting in traffic dropping by as much as 57% as compared to the prior week. After the initial drop at midnight on December 13 (22:00 UTC on December 12), traffic gradually recovered over the following several days, returning to expected levels around 14:30 local time (12:30 UTC) on December 16.

Hurricane Melissa made landfall on Jamaica on October 28 and left a trail of damage and destruction in its path. Associated power outages and infrastructure damage impacted Internet connectivity, causing traffic to initially drop by approximately half, starting around 06:15 local time (11:15 UTC), ultimately reaching as much as 70% lower than the previous week. Internet traffic from Jamaica remained well below pre-hurricane levels for several days, and ultimately started to make greater progress towards expected levels during the morning of November 4. It can often take weeks or months for Internet traffic from a country to return to “normal” levels following storms that cause massive and widespread damage – while power may be largely restored within several days, damage to physical infrastructure takes significantly longer to address.

On November 26, Cyclone Senyar caused catastrophic floods and landslides in Sri Lanka and Indonesia, killing over 1,000 people and damaging telecommunications and power infrastructure across these countries. The infrastructure damage resulted in disruptions to Internet connectivity, and resultant lower traffic levels, across multiple regions.

In Sri Lanka, regions outside the main Western Province were the most affected, and several provinces saw traffic drop between 80% and 95% as compared to the prior week, including North Western, Southern, Uva, Eastern, Northern, North Central, and Sabaragamuwa.

In Indonesia, Aceh and the Sumatra regions saw the biggest Internet disruptions. In Aceh, traffic initially dropped over 75% as compared to the previous week. In Sumatra, North Sumatra was the most affected, with an early 30% drop as compared to the previous week, before starting to recover more actively the following week.

On October 3, subscribers to Indonesian Internet provider Smartfren (AS18004) experienced a service disruption. The issues were acknowledged by the provider in an X post, which stated (in translation), “Currently, telephone, SMS and data services are experiencing problems in several areas.” Traffic from the provider fell as much as 84%, starting around 09:00 local time (02:00 UTC). The disruption lasted for approximately eight hours, as traffic returned to expected levels around 17:00 local time (10:00 UTC). Smartfren did not provide any additional information on what caused the service problems.

Major British Internet provider Vodafone UK (AS5378 & AS25135) experienced a brief service outage on October 23. At 15:00 local time (14:00 UTC), traffic on both Vodafone ASNs dropped to zero. Announced IPv4 address space from AS5378 fell by 75%, while announced IPv4 address space from AS25135 disappeared entirely. Both Internet traffic and address space recovered two hours later, returning to expected levels around 17:00 local time (16:00 UTC). Vodafone did not provide any information on their social media channels about the cause of the outage, and their network status checker page was also unavailable during the outage.

According to a published report, a DNS resolution issue disrupted Internet services for customers of Italian provider Fastweb (AS12874) on October 22, causing observed traffic volumes to drop by over 75%. Fastweb acknowledged the issue, which impacted wired Internet customers between 09:30 - 13:00 local time (08:30 - 12:00 UTC).

Although not an Internet outage caused by connectivity failure, the impact of DNS resolution issues on Internet traffic is very similar. When a provider’s DNS resolver is experiencing problems, switching to a service like Cloudflare’s 1.1.1.1 public DNS resolver will often restore connectivity.

On December 7, a concurrent drop in traffic was observed across SBIN (AS28683), MTN Benin (AS37424), and Etisalat Benin (AS37136). Between 18:30 - 19:30 local time (17:30 - 18:30 UTC), traffic dropped as much as 80% as compared to the prior week at a country level, nearly 100% at Etisalat and MTN, and over 80% at SBIN.

While an attempted coup had taken place earlier in the day, it is unclear whether the observed Internet disruption was related in any way. From a routing perspective, all three impacted networks share Cogent (AS174) as an upstream provider, so a localized issue at Cogent may have contributed to the brief outage.  

According to a reported announcement from Israeli provider Cellcom (AS1680), on December 18, there was “a malfunction affecting Internet connectivity that is impacting some of our customers.” This malfunction dropped traffic nearly 70% as compared to the prior week, and occurred between 09:30 - 11:00 local time (07:30 - 09:00 UTC). The “malfunction” may have been a DNS failure, according to a published report.

Closing out 2025, on December 30, a major technical failure at Israeli provider Partner Communications (AS12400) disrupted mobile, TV, and Internet services across the country. Internet traffic from Partner fell by two-thirds as compared to the previous week between 14:00 - 15:00 local time (12:00 - 13:00 UTC). During the outage, queries to Cloudflare’s 1.1.1.1 public DNS resolver spiked, suggesting that the problem may have been related to Partner’s DNS infrastructure. However, the provider did not publicly confirm what caused the outage.

During the fourth quarter, we launched a new Cloud Observatory page on Radar that tracks availability and performance issues at a region level across hyperscaler cloud platforms, including Amazon Web Services, Microsoft Azure, Google Cloud Platform, and Oracle Cloud Infrastructure.

On October 20, the Amazon Web Services us-east-1 region in Northern Virginia experienced “increased error rates and latencies” that affected multiple services within the region. The issues impacted not only customers with public-facing Web sites and applications that rely on infrastructure within the region, but also Cloudflare customers that have origin resources hosted in us-east-1.

We began to see the impact of the problems around 06:30 UTC, as the share of error (5xx-class) responses began to climb, reaching as high as 17% around 08:00 UTC. The number of failures encountered when attempting to connect to origins in us-east-1 climbed as well, peaking around 12:00 UTC.

The impact could also be clearly seen in key network performance metrics, which remained elevated throughout the incident, returning to normal levels just before the end of the incident, around 23:00 UTC. Both TCP and TLS handshake durations got progressively worse throughout the incident—these metrics measure the amount of time needed for Cloudflare to establish TCP and TLS connections respectively with customer origin servers in us-east-1. In addition, the amount of time elapsed before Cloudflare received response headers from the origin increased significantly during the first several hours of the incident, before gradually returning to expected levels.

On October 29, Microsoft Azure experienced an incident impacting Azure Front Door, its content delivery network service. According to Azure's report on the incident, “A specific sequence of customer configuration changes, performed across two different control plane build versions, resulted in incompatible customer configuration metadata being generated. These customer configuration changes themselves were valid and non-malicious – however they produced metadata that, when deployed to edge site servers, exposed a latent bug in the data plane. This incompatibility triggered a crash during asynchronous processing within the data plane service.”

The incident report marked the start time at 15:41 UTC, although we observed the volume of failed connection attempts to Azure-hosted origins begin to climb about 45 minutes prior. The TCP and TLS handshake metrics also became more volatile during the incident period, with TCP handshakes taking over 50% longer at times, and TLS handshakes taking nearly 200% longer at peak. The impacted metrics began to improve after 20:00 UTC, and according to Microsoft, the incident ended at 00:05 UTC on October 30.

In addition to the outages discussed above, Cloudflare also experienced two disruptions during the fourth quarter. While these were not Internet outages in the classic sense, they did prevent users from accessing Web sites and applications delivered and protected by Cloudflare when they occurred.

The first incident took place on November 18, and was caused by a software failure triggered by a change to one of our database systems' permissions, which caused the database to output multiple entries into a “feature file” used by our Bot Management system. Additional details, including a root cause analysis and timeline, can be found in the associated blog post.

The second incident occurred on December 5, and impacted a subset of customers, accounting for approximately 28% of all HTTP traffic served by Cloudflare. It was triggered by changes being made to our request body parsing logic while attempting to detect and mitigate a newly disclosed industry-wide React Server Components vulnerability. A post-mortem blog post contains additional details, including a root cause analysis and timeline.

For more information about the work underway at Cloudflare to prevent outages like these from happening again, check out our blog post detailing “Code Orange: Fail Small.”

The disruptions observed in the fourth quarter underscore the importance of real-time data in maintaining global connectivity. Whether it’s a government-ordered shutdown or a minor technical issue, transparency allows the technical community to respond faster and more effectively. We will continue to track these shifts on Cloudflare Radar, providing the insights needed to navigate the complexities of modern networking. We share our observations on the Cloudflare Radar Outage Center, via social media, and in posts on blog.cloudflare.com. Follow us on social media at @CloudflareRadar (X), noc.social/@cloudflareradar (Mastodon), and radar.cloudflare.com (Bluesky), or contact us via email.

As a reminder, while these blog posts feature graphs from Radar and the Radar Data Explorer, the underlying data is available from our API. You can use the API to retrieve data to do your own local monitoring or analysis, or you can use the Radar MCP server to incorporate Radar data into your AI tools.

On November 18, 2025, Cloudflare’s network experienced significant failures to deliver network traffic for approximately two hours and ten minutes. Nearly three weeks later, on December 5, 2025, our network again failed to serve traffic for 28% of applications behind our network for about 25 minutes.

We published detailed post-mortem blog posts following both incidents, but we know that we have more to do to earn back your trust. Today we are sharing details about the work underway at Cloudflare to prevent outages like these from happening again.

We are calling the plan “Code Orange: Fail Small”, which reflects our goal of making our network more resilient to errors or mistakes that could lead to a major outage. A “Code Orange” means the work on this project is prioritized above all else. For context, we declared a “Code Orange” at Cloudflare once before, following another major incident that required top priority from everyone across the company. We feel the recent events require the same focus.  Code Orange is our way to enable that to happen, allowing teams to work cross-functionally as necessary to get the job done while pausing any other work.

The Code Orange work is organized into three main areas:

• Require controlled rollouts for any configuration change that is propagated to the network, just like we do today for software binary releases.

• Review, improve, and test failure modes of all systems handling network traffic to ensure they exhibit well-defined behavior under all conditions, including unexpected error states.

• Change our internal “break glass”* procedures, and remove any circular dependencies so that we, and our customers, can act fast and access all systems without issue during an incident.

These projects will deliver iterative improvements as they proceed, rather than one “big bang” change at their conclusion. Every individual update will contribute to more resiliency at Cloudflare. By the end, we expect Cloudflare’s network to be much more resilient, including for issues such as those that triggered the global incidents we experienced in the last two months.

We understand that these incidents are painful for our customers and the Internet as a whole. We’re deeply embarrassed by them, which is why this work is the first priority for everyone here at Cloudflare.

^{* Break glass procedures at Cloudflare allow certain individuals to elevate their privilege under certain circumstances to perform urgent actions to resolve high severity scenarios.}

In the first incident, users visiting a customer site on Cloudflare saw error pages that indicated Cloudflare could not deliver a response to their request. In the second, they saw blank pages.

Both outages followed a similar pattern. In the moments leading up to each incident we instantaneously deployed a configuration change in our data centers in hundreds of cities around the world.

The November change was an automatic update to our Bot Management classifier. We run various artificial intelligence models that learn from the traffic flowing through our network to build detections that identify bots. We constantly update those systems to stay ahead of bad actors trying to evade our security protection to reach customer sites.

During the December incident, while trying to protect our customers from a vulnerability in the popular open source framework React, we deployed a change to a security tool used by our security analysts to improve our signatures. Similar to the urgency of new bot management updates, we needed to get ahead of the attackers who wanted to exploit the vulnerability. That change triggered the start of the incident.

This pattern exposed a serious gap in how we deploy configuration changes at Cloudflare, versus how we release software updates. When we release software version updates, we do so in a controlled and monitored fashion. For each new binary release, the deployment must successfully complete multiple gates before it can serve worldwide traffic. We deploy first to employee traffic, before carefully rolling out the change to increasing percentages of customers worldwide, starting with free users. If we detect an anomaly at any stage, we can revert the release without any human intervention.

We have not applied that methodology to configuration changes. Unlike releasing the core software that powers our network, when we make configuration changes, we are modifying the values of how that software behaves and we can do so instantly. We give this power to our customers too: If you make a change to a setting in Cloudflare, it will propagate globally in seconds.

While that speed has advantages, it also comes with risks that we need to address. The past two incidents have demonstrated that we need to treat any change that is applied to how we serve traffic in our network with the same level of tested caution that we apply to changes to the software itself.

Our ability to deploy configuration changes globally within seconds was the core commonality across the two incidents. In both events, a wrong configuration took down our network in seconds.

Introducing controlled rollouts of our configuration, just as we already do for software releases, is the most important workstream of our Code Orange plan.

Configuration changes at Cloudflare propagate to the network very quickly. When a user creates a new DNS record, or creates a new security rule, it reaches 90% of servers on the network within seconds. This is powered by a software component that we internally call Quicksilver.

Quicksilver is also used for any configuration change required by our own teams. The speed is a feature: we can react and globally update our network behavior very quickly. However, in both incidents this caused a breaking change to propagate to the entire network in seconds rather than passing through gates to test it.

While the ability to deploy changes to our network on a near-instant basis is useful in many cases, it is rarely necessary. Work is underway to treat configuration the same way that we treat code by introducing controlled deployments within Quicksilver to any configuration change.

We release software updates to our network multiple times per day through what we call our Health Mediated Deployment (HMD) system. In this framework, every team at Cloudflare that owns a service (a piece of software deployed into our network) must define the metrics that indicate a deployment has succeeded or failed, the rollout plan, and the steps to take if it does not succeed.

Different services will have slightly different variables. Some might need longer wait times before proceeding to more data centers, while others might have lower tolerances for error rates even if it causes false positive signals.

Once deployed, our HMD toolkit begins to carefully progress against that plan while monitoring each step before proceeding. If any step fails, the rollback will automatically begin and the team can be paged if needed.

By the end of Code Orange, configuration updates will follow this same process. We expect this to allow us to quickly catch the kinds of issues that occurred in these past two incidents long before they become widespread problems.

While we are optimistic that better control over configuration changes will catch more problems before they become incidents, we know that mistakes can and will occur. During both incidents, errors in one part of our network became problems in most of our technology stack, including the control plane that customers rely on to configure how they use Cloudflare.

We need to think about careful, graduated rollouts not just in terms of geographic progression (spreading to more of our data centers) or in terms of population progression (spreading to employees and customer types). We also need to plan for safer deployments that contain failures from service progression (spreading from one product like our Bot Management service to an unrelated one like our dashboard).

To that end, we are in the process of reviewing the interface contracts between every critical product and service that comprise our network to ensure that we a) assume failure will occur between each interface and b) handle that failure in the absolute most reasonable way possible. 

To go back to our Bot Management service failure, there were at least two key interfaces where, if we had assumed failure was going to happen, we could have handled it gracefully to the point that it was unlikely any customer would have been impacted. The first was in the interface that read the corrupted config file. Instead of panicking, there should have been a sane set of validated defaults which would have allowed traffic to pass through our network, while we would have, at worst, lost the realtime fine-tuning that feeds into our bot detection machine-learning models. The second interface was between the core software that runs our network and the Bot Management module itself. In the event that our bot management module failed (as it did), we should not have dropped traffic by default. Instead, we could have come up with, yet again, a more sane default of allowing the traffic to pass with a passable classification.

During the incidents, it took us too long to resolve the problem. In both cases, this was worsened by our security systems preventing team members from accessing the tools they needed to fix the problem, and in some cases, circular dependencies slowed us down as some internal systems also became unavailable.

As a security company, all our tools are behind authentication layers with fine-grained access controls to ensure customer data is safe and to prevent unauthorized access. This is the right thing to do, but at the same time, our current processes and systems slowed us down when speed was a top priority.

Circular dependencies also affected our customer experience. For example, during the November 18 incident, Turnstile, our no CAPTCHA bot solution, became unavailable. As we use Turnstile on the login flow to the Cloudflare dashboard, customers who did not have active sessions, or API service tokens, were not able to log in to Cloudflare in the moment of most need to make critical changes.

Our team will be reviewing and improving all of the break glass procedures and technology to ensure that, when necessary, we can access the right tools as fast as possible while maintaining our security requirements. This includes reviewing and removing circular dependencies, or being able to “bypass” them quickly in the event there is an incident. We will also increase the frequency of our training exercises, so that processes are well understood by all teams prior to any potential disaster scenario in the future. 

While we haven’t captured in this post all the work being undertaken internally, the workstreams detailed above describe the top priorities the teams are being asked to focus on. Each of these workstreams maps to a detailed plan touching nearly every product and engineering team at Cloudflare. We have a lot of work to do.

By the end of Q1, and largely before then, we will:

• Ensure all production systems are covered by Health Mediated Deployments (HMD) for configuration management.

• Update our systems to adhere to proper failure modes as appropriate for each product set.

• Ensure we have processes in place so the right people have the right access to provide proper remediation during an emergency.

Some of these goals will be evergreen. We will always need to better handle circular dependencies as we launch new software and our break glass procedures will need to update to reflect how our security technology changes over time.

We failed our users and the Internet as a whole in these past two incidents. We have work to do to make it right. We plan to share updates as this work proceeds and appreciate the questions and feedback we have received from our customers and partners.

Our view is unique, due to Cloudflare’s global network, which has a presence in 330 cities in over 125 countries/regions, handling over 81 million HTTP requests per second on average, with more than 129 million HTTP requests per second at peak on behalf of millions of customer Web properties, in addition to responding to approximately 67 million (authoritative + resolver) DNS queries per second. Cloudflare Radar uses the data generated by these Web and DNS services, combined with other complementary data sets, to provide near-real time insights into traffic, bots, security, connectivity, and DNS patterns and trends that we observe across the Internet. 

Our Radar Year in Review takes that observability and, instead of a real-time view, offers a look back at 2025: incorporating interactive charts, graphs, and maps that allow you to explore and compare selected trends and measurements year-over-year and across geographies, as well as share and embed Year in Review graphs. 

The 2025 Year In Review is organized into six sections: Traffic, AI, Adoption & Usage, Connectivity, Security, and Email Security, with data spanning the period from January 1 to December 2, 2025. To ensure consistency, we kept underlying methodologies unchanged from previous years’ calculations. We also incorporated several new data sets this year, including multiple AI-related metrics, global speed test activity, and hyper-volumetric DDOS size progression. Trends for over 200 countries/regions are available on the microsite; smaller or less-populated locations are excluded due to insufficient data. Some metrics are only shown worldwide and are not displayed if a country/region is selected. 

In this post, we highlight key findings and interesting observations from the major Year In Review microsite sections, and we have again published a companion Most Popular Internet Services blog post that specifically explores trends seen across top Internet Services.

We encourage you to visit the 2025 Year in Review microsite to explore the datasets and metrics in more detail, including those for your country/region to see how they have changed since 2024, and how they compare to other areas of interest.

We hope you’ll find the Year in Review to be an insightful and powerful tool — to explore the disruptions, advances, and metrics that defined the Internet in 2025. 

Let’s dig in.

• Global Internet traffic grew 19% in 2025, with significant growth starting in August. ➜

• The top 10 most popular Internet services saw a few year-over-year shifts, while a number of new entrants landed on category lists. ➜

• Starlink traffic doubled in 2025, including traffic from over 20 new countries/regions. ➜

• Googlebot was again responsible for the highest volume of request traffic to Cloudflare in 2025 as it crawled millions of Cloudflare customer sites for search indexing and AI training. ➜

• The share of human-generated Web traffic that is post-quantum encrypted has grown to 52%. ➜

• Googlebot was responsible for more than a quarter of Verified Bot traffic. ➜

• Crawl volume from dual-purpose Googlebot dwarfed other AI bots and crawlers. ➜

• AI “user action” crawling increased by over 15x in 2025. ➜

• While other AI bots accounted for 4.2% of HTML request traffic, Googlebot alone accounted for 4.5%. ➜

• Anthropic had the highest crawl-to-refer ratio among the leading AI and search platforms. ➜

• AI crawlers were the most frequently fully disallowed user agents found in robots.txt files. ➜

• On Workers AI, Meta’s llama-3-8b-instruct model was the most popular model, and text generation was the most popular task type. ➜

• iOS devices generated 35% of mobile device traffic globally — and more than half of device traffic in many countries. ➜

• The shares of global Web requests using HTTP/3 and HTTP/2 both increased slightly in 2025. ➜

• JavaScript-based libraries and frameworks remained integral tools for building Web sites. ➜

• One-fifth of automated API requests were made by Go-based clients. ➜

• Google remains the top search engine, with Yandex, Bing, and DuckDuckGo distant followers. ➜

• Chrome remains the top browser across platforms and operating systems – except on iOS, where Safari has the largest share. ➜

• Almost half of the 174 major Internet outages observed around the world in 2025 were due to government-directed regional and national shutdowns of Internet connectivity. ➜

• Globally, less than a third of dual-stack requests were made over IPv6, while in India, over two-thirds were. ➜

• European countries had some of the highest download speeds, all above 200 Mbps. Spain remained consistently among the top locations across measured Internet quality metrics. ➜

• London and Los Angeles were hotspots for Cloudflare speed test activity in 2025. ➜

• More than half of request traffic comes from mobile devices in 117 countries/regions. ➜

• 6% of global traffic over Cloudflare’s network was mitigated by our systems — either as potentially malicious or for customer-defined reasons. ➜

• 40% of global bot traffic came from the United States, with Amazon Web Services and Google Cloud originating a quarter of global bot traffic. ➜

• Organizations in the "People and Society” sector were the most targeted during 2025. ➜

• Routing security, measured as the shares of RPKI valid routes and covered IP address space, saw continued improvement throughout 2025. ➜

• Hyper-volumetric DDoS attack sizes grew significantly throughout the year. ➜

• More than 5% of email messages analyzed by Cloudflare were found to be malicious. ➜

• Deceptive links, identity deception, and brand impersonation were the most common types of threats found in malicious email messages. ➜

• Nearly all of the email messages from the .christmas and .lol Top Level Domains were found to be either spam or malicious. ➜

To determine the traffic trends over time for the Year in Review, we use the average daily traffic volume (excluding bot traffic) over the second full calendar week (January 12-18) of 2025 as our baseline. (The second calendar week is used to allow time for people to get back into their “normal” school and work routines after the winter holidays and New Year’s Day.) The percent change shown in the traffic trends chart is calculated relative to the baseline value — it does not represent absolute traffic volume for a country/region. The trend line represents a seven-day trailing average, which is used to smooth the sharp changes seen with data at a daily granularity. 

Traffic growth in 2025 appeared to occur in several phases. Traffic was, on average, somewhat flat through mid-April, generally within a couple of percent of the baseline value. However, it then saw growth through May to approximately 5% above baseline, staying in the +4-7% range through mid-August. It was at that time that growth accelerated, climbing steadily through September, October, and November, peaking at 19% growth for the year. Aided by a late-November increase, 2025’s rate of growth is about 10% higher than the 17% growth observed in 2024. In past years, we have also observed traffic growth accelerating in the back half of the year, although in 2022-2024, that acceleration started in July. It’s not clear why this year’s growth was seemingly delayed by several weeks.

^{Internet traffic trends in 2025, worldwide}

Botswana saw the highest peak growth, reaching 298% above baseline on November 8, and ending the period 295% over baseline. (More on what accounts for that growth in the Starlink section below.) Botswana and Sudan were the only countries/regions to see traffic more than double over the course of the year, although some others experienced peak increases over 100% at some point during the year.

^{Internet traffic trends in 2025, Botswana}

The impact of extended Internet disruptions are clearly visible within the graphs as well. For example, on October 29, the Tanzanian government imposed an Internet shutdown there in response to election day protests. That shutdown lasted just a day, but another one followed from October 30 until November 3. Although traffic in the country had increased more than 40% above baseline ahead of the shutdowns, the disruption ultimately dropped traffic more than 70% below baseline — a rapid reversal. Traffic recovered quickly after connectivity was restored. A similar pattern was observed in Jamaica, where Internet traffic spiked ahead of the arrival of Hurricane Melissa on October 28, and then dropped significantly after the storm caused power outages and infrastructure damage on the island. Traffic began to rebound after the storm’s passing, returning to a level just above baseline by early December.

^{Internet traffic trends in 2025, Tanzania}

^{Internet traffic trends in 2025, Jamaica}

For the Year in Review, we look at the 11-month year-to-date period. In addition to an “overall” ranked list, we also rank services across nine categories, based on analysis of anonymized query data of traffic to our 1.1.1.1 public DNS resolver from millions of users around the world. For the purposes of these rankings, domains that belong to a single Internet service are grouped together.

Google and Facebook once again held the top two spots among the top 10. Although the other members of the top 10 list remained consistent with 2024’s rankings, there was some movement in the middle. Microsoft, Instagram, and YouTube all moved higher; Amazon Web Services (AWS) dropped one spot lower, while TikTok fell four spots.

^{Top Internet services in 2025, worldwide}

Among Generative AI services, ChatGPT/OpenAI remained at the top of the list. But there was movement elsewhere, highlighting the dynamic nature of the industry. Services that moved up the rankings include Perplexity, Claude/Anthropic, and GitHub Copilot. New entries in the top 10 for 2025 include Google Gemini, Windsurf AI, Grok/xAI, and DeepSeek.

^{Top Generative AI services in 2025, worldwide}

Other categories saw movement within their lists as well – Shopee (“the leading e-commerce online shopping platform in Southeast Asia and Taiwan”) is a new entrant to the E-Commerce list, and HBO Max joined the Video Streaming ranking. These categorical rankings, as well as trends seen by specific services, are explored in more detail in a separate blog post.

In addition, this year we are also providing top Internet services insights at a country/region level for the Overall, Generative AI, Social Media, and Messaging categories. (In 2024, we only shared Overall insights.)

SpaceX Starlink’s satellite-based Internet service continues to be a popular option for bringing connectivity to unserved or underserved areas, as well as to users on planes and boats. We analyzed aggregate request traffic volumes associated with Starlink's primary autonomous system (AS14593) to track the growth in usage of the service throughout 2025. The request volume shown on the trend line in the chart represents a seven-day trailing average. 

Globally, traffic from Starlink continued to see consistent growth throughout 2025, with total request volume up 2.3x across the year. We tend to see rapid traffic growth when Starlink service becomes available in a country/region, and that trend continues in 2025. 

^{Starlink traffic growth in 2025, worldwide}

That’s exactly what we saw in the more than 20 new countries/regions where @Starlink announced availability: within days, Starlink traffic in those places increased rapidly. These included Armenia, Niger, Sri Lanka, and Sint Maarten.

We also saw Starlink traffic from a number of locations that are not currently marked for service availability. However, there are IPv4 and/or IPv6 prefixes associated with these countries in Starlink’s published geofeed. Given the ability for Starlink users to roam with their service (and equipment), this traffic likely comes from roaming users in those areas.

^{Starlink traffic growth in 2025, Niger}

Of countries/regions where service was active before 2025, Benin, Timor-Leste, and Botswana had some of the largest traffic growth, at 51x, 19x, and 16x respectively. Starlink service availability in Benin was first announced in November 2023, Timor-Leste in December 2024, and Botswana in August 2024.

^{Starlink traffic growth in 2025, Botswana}

Similar services, such as Amazon Leo, Eutelsat Konnect, and China’s Qianfan, continue to grow their satellite constellations and move towards commercial availability. We hope to review traffic growth across these services in the future as well.

To look at the aggregate request traffic Cloudflare saw in 2025 from the entire IPv4 Internet, we can use a Hilbert curve, which allows us to visualize a sequence of IPv4 addresses in a two-dimensional pattern that keeps nearby IP addresses close to each other, making them useful for surveying the Internet's IPv4 address space. Within the visualization, we aggregate IPv4 addresses into /20 prefixes, meaning that at the highest zoom level, each square represents traffic from 4,096 IPv4 addresses. This level of aggregation keeps the amount of data used for the visualization manageable. See the 2024 Year in Review blog post for additional details about the visualization.

For the third year in a row, the IP address block that had the maximum request volume to Cloudflare during 2025 was Google’s 66.249.64.0/20 –  one of several used by the Googlebot web crawler to retrieve content for search indexing and AI training. That a Googlebot IP address block ranked again as the top request traffic source is unsurprising, given the number of web properties on Cloudflare’s network and Googlebot’s aggressive crawling activity. The Googlebot prefix accounted for nearly 4x as much IPv4 request traffic as the next largest traffic source, 146.20.240.0/20, which is part of a larger block of IPv4 address space announced by Rackspace Hosting. As a cloud and hosting provider, Rackspace supports many different types of customers and applications, so the driver of the observed traffic to Cloudflare isn’t known.

^{Zoomed Hilbert curve view showing the address block that generated the highest volume of requests in 2025}

This year, we’ve added the ability to search for an autonomous system (ASN) to the visualization, allowing you to see how broadly a network provider’s IP address holdings are distributed across the IPv4 universe. 

One example is AS16509 (AMAZON-02, used with AWS), which shows the results of Amazon’s acquisitions of large amounts of IPv4 address space over the years. Another example is AS7018 (ATT-INTERNET4, AT&T), which is one of the largest announcers of IPv4 address space in the United States. Much of the traffic we see from this ASN comes from 12.0.0.0/8, a block of over 16 million IPv4 addresses that has been owned by AT&T since 1983.

^{Hilbert curve showing the IPv4 address blocks from AS7018 that sent traffic to Cloudflare in 2025}

“Post-quantum” refers to a set of cryptographic techniques designed to protect encrypted data from “harvest now, decrypt later” attacks by adversaries that have the ability to capture and store current data for future decryption by sufficiently advanced quantum computers. The Cloudflare Research team has been working on post-quantum cryptography since 2017, and regularly publishes updates on the state of the post-quantum Internet.

After seeing significant growth in 2024, the global share of post-quantum encrypted traffic nearly doubled throughout 2025, from 29% at the start of the year to 52% in early December. 

^{Post-quantum encrypted TLS 1.3 traffic growth in 2025, worldwide}

Twenty-eight countries/regions saw their share of post-quantum encrypted traffic more than double throughout the year, including significant growth in Puerto Rico and Kuwait. Kuwait’s share nearly tripled, from 13% to 37%, and Puerto Rico’s share grew from 20% to 49%. 

Those three were among others that saw significant share growth in mid-September, concurrent with Apple releasing operating system updates, in which “TLS-protected connections will automatically advertise support for hybrid, quantum-secure key exchange in TLS 1.3”. In Kuwait and Puerto Rico, over half of request traffic is from mobile devices, and approximately half comes from iOS devices in both locations as well, so it is not surprising that this software update resulted in a significant increase in post-quantum traffic share

^{Post-quantum encrypted TLS 1.3 traffic growth in 2025, Puerto Rico}

To that end, the share of post-quantum encrypted traffic from Apple iOS devices grew significantly in September after iOS 26 was officially released. Just four days after release, the global share of requests with post-quantum support from iOS devices grew from just under 2% to 11%. By early December, more than 25% of requests from iOS devices used post-quantum encryption.

The new Bots Directory on Cloudflare Radar provides a wealth of information about Verified Bots and Signed Agents, including their operators, categories, and associated user agents, links to documentation, and traffic trends. Verified Bots must conform to a set of requirements as well as being verified through either Web Bot Auth or IP validation. A signed agent is controlled by an end user and a verified signature-agent from their Web Bot Auth implementation, and must conform to a separate set of requirements.

Googlebot is used to crawl Web site content for search indexing and AI training, and it was far and away the most active bot seen by Cloudflare throughout 2025. It was most active between mid-February and mid-July, peaking in mid-April, and was responsible for over 28% of traffic from Verified Bots. Other Google-operated bots that were responsible for notable amounts of traffic included Google AdsBot (used to monitor Web sites where Google ads are served), Google Image Proxy (used to retrieve and cache images embedded in email messages), and GoogleOther (used by various product teams for fetching publicly accessible content from sites).

OpenAI’s GPTBot, which crawls content for AI training, was the next most active bot, originating about 7.5% of Verified Bot traffic, with fairly volatile crawling activity during the first half of the year. Microsoft’s Bingbot crawls Web site content for search indexing and AI training and generated 6% of Verified Bot traffic throughout the year, showing relatively stable activity. 

^{Verified Bot traffic trends in 2025, worldwide}

Search engine crawlers and AI crawlers are the two most active Verified Bot categories, with traffic patterns mapping closely to the leading bots in those categories, including GoogleBot and OpenAI’s GPTBot. Search engine crawlers were responsible for 40% of Verified Bot traffic, with AI crawlers generating half as much (20%). Search engine optimization bots were also quite active, driving over 13% of requests from Verified Bots.

^{Verified Bot traffic trends by category in 2025, worldwide}

In September, a Cloudflare blog post laid out a proposal for responsible AI bot principles, one of which was “AI bots should have one distinct purpose and declare it.” In the AI bots best practices overview on Radar, we note that several bot operators have dual-purpose crawlers, including Google and Microsoft.

Because Googlebot crawls for both search engine indexing and AI training, we have included it in this year’s AI crawler overview. In 2025, its crawl volume dwarfed that of other leading AI bots. Request traffic began to increase in mid-February, peaking in late April, and then slowly declined through late July. After that, it grew gradually into the end of the year. Bingbot also has a similar dual purpose, although its crawl volume is a fraction of Googlebot’s. Bingbot’s crawl activity trended generally upwards across the year.

^{AI crawler traffic trends in 2025, worldwide}

OpenAI’s GPTBot is used to crawl content that may be used in training OpenAI's generative AI foundation models. Its crawling activity was quite volatile across the year, reaching its highest levels in June, but it ended November slightly above the crawl levels seen at the beginning of the year. 

Crawl volume for OpenAI’s ChatGPT-User, which visits Web pages when users ask ChatGPT or a CustomGPT questions, saw sustained growth over the course of the year, with a weekly usage pattern becoming more evident starting in mid-February, suggesting increasing usage at schools and in the workplace. Peak request volumes were as much as 16x higher than at the beginning of the year. A drop in activity was also evident in the June to August timeframe, when many students were out of school and many professionals took vacation time. 

OAI-SearchBot, which is used to link to and surface websites in search results in ChatGPT's search features, saw crawling activity grow gradually through August, then several traffic spikes in August and September, before starting to grow more aggressively heading into October, with peak request volume during a late October spike approximately 5x higher than the beginning of the year.

^{OpenAI crawler traffic trends in 2025, worldwide}

Crawling by Anthropic’s ClaudeBot effectively doubled through the first half of the year, but gradually declined during the second half, returning to a level approximately 10% higher than the start of the year. Perplexity’s PerplexityBot crawling traffic grew slowly through January and February, but saw a big jump in activity from mid-March into April. After that, growth was more gradual through October, before seeing a significant increase again in November, winding up about 3.5x higher than where it started the year.

^{ClaudeBot traffic trends in 2025, worldwide}

^{PerplexityBot traffic trends in 2025, worldwide}

ByteDance’s Bytespider, one of 2024’s top AI crawlers, saw crawling volume below several other training bots, and its activity dropped across the year, continuing the decline observed last year.

Most AI bot crawling is done for one of three purposes: training, which gathers Web site content for AI model training; search, which indexes Web site content for search functionality available on AI platforms; and user action, which visits Web sites in response to user questions posed to a chatbot. Note that search crawling may also include crawling for Retrieval-Augmented Generation (RAG), which enables a content owner to bring their own data into LLM generation without retraining or fine-tuning a model. (A fourth “undeclared” purpose captures traffic from AI bots whose crawling purpose is unclear or unknown.)

Crawling for model training is responsible for the overwhelming majority of AI crawler traffic, reaching as much as 7-8x search crawling and 32x user action crawling at peak. The training traffic figure is heavily influenced by OpenAI’s GPTBot, and as such, it followed a very similar pattern through the year.

Crawling for search was strongest through mid-March, when it dropped by approximately 40%. It returned to more gradual growth after that, though it ended the surveyed time period just under 10% lower than the start of the year.

User action crawling started 2025 with the lowest crawl volume of the three defined purposes, but more than doubled through January and February. It again doubled in early March, and from there, it continued to grow throughout the year, up over 21x from January through early December. This growth maps very closely to the traffic trends seen for OpenAI’s ChatGPT-User bot.

^{User action crawler traffic trends in 2025, worldwide}

AI bots have frequently been in the news during 2025 as content owners raise concerns about the amount of traffic that they are generating, especially as much of it does not translate into end users being referred back to the source Web sites. To better understand the impact of AI bot crawling activity, as compared to non-AI bots and human Web usage, we analyzed request traffic for HTML content across Cloudflare’s customer base and classified it as coming from a human, an AI bot, or another “non-AI” type of bot. (Note that because we are focusing on just HTML content here, the bot and human shares of traffic will differ from that shown on Radar, which analyzes request traffic for all content types.) Because Googlebot crawls so actively, and is dual-purpose, we have broken its share out separately in this analysis.

Throughout 2025, we found that traffic from AI bots accounted for an average of 4.2% of HTML requests. The share varied widely throughout the year, dropping as low as 2.4% in early April, and reaching as high as 6.4% in late June.

To that end, non-AI bots started 2025 responsible for half of requests to HTML pages, seven percentage points above human-generated traffic. This gap grew as wide as 25 percentage points during the first few days of June. However, these traffic shares began to draw closer together starting in mid June, and starting on September 11, entered a period where the human generated share of HTML traffic sometimes exceeded that of non-AI bots. As of December 2, human traffic generated 47% of HTML requests, and non-AI bots generated 44%.

Googlebot is a particularly voracious crawler, and this year it originated 4.5% of HTML requests, a share slightly larger than AI bots in aggregate. Starting the year at just under 2.5%, its share ramped quickly over the next four months, peaking at 11% in late April. It subsequently fell back towards its starting point over the next several months, and then grew again during the second half of the year, ending with a 5% share. This share shift largely mirrors Googlebot’s crawling activity as discussed above.

^{HTML traffic shares by bot type in 2025, worldwide}

We launched the crawl-to-refer ratio metric on Radar on July 1 to track how often a given AI or search platform sends traffic to a site relative to how often it crawls that site. A high ratio means a whole lot of AI crawling without sending actual humans to a Web site.

It can be a volatile metric, with the values shifting day-by-day as crawl activity and referral traffic change. This metric compares total number of requests from relevant user agents associated with a given search or AI platform where the response was of Content-type: text/html by the total number of requests for HTML content where the Referer header contained a hostname associated with a given search or AI platform. 

Anthropic had the highest crawl-to-refer ratios this year, reaching as much as 500,000:1, although they were quite erratic from January through May. Both the magnitude and erratic nature of the metric was likely due to sparse referral traffic over that time period. After that, the ratios became more consistent, but remained higher than others, ranging from ~25,000:1 to ~100,000:1.

OpenAI’s ratios over time were quite spiky, and reached as much as 3,700:1 in March. These shifts may be due to the stabilization of GPTBot crawling activity, coupled with increased usage of ChatGPT search functionality, which includes links back to source Web sites within its responses. Users following those links would increase Referer counts, potentially lowering the ratio. (Assuming that crawl traffic wasn’t increasing at a similar or greater rate.)

Perplexity had the lowest crawl-to-refer ratios of the major AI platforms, starting the year below 100:1 before spiking in late March above 700:1, concurrent with a spike of crawl traffic seen from PerplexityBot.  Settling back down after the spike, peak ratio values generally remained below 400:1, and below 200:1 from September onwards.

Among search platforms, Microsoft’s ratio unexpectedly exhibited a cyclical weekly pattern, reaching its lowest levels on Thursdays, and peaking on Sundays. Peak ratio values were generally in the 50:1 to 70:1 range across the year. Starting the year just over 3:1, Google’s crawl-to-refer ratio increased steadily through April, reaching as high as 30:1. After peaking, it fell somewhat erratically through mid-July, dropping back to 3:1, although it has been slowly increasing through the latter half of 2025. DuckDuckGo’s ratio remained below 1:1 for the first three calendar quarters of 2025, but experienced a sudden jump to 1.5:1 in mid-October and stayed elevated for the remainder of the period.

^{AI & search platform crawl-to-refer ratios in 2025, worldwide}

The robots.txt file, formally defined in RFC 9309 as the Robots Exclusion Protocol, is a text file that content owners can use to signal to Web crawlers which parts of a Web site the crawlers are allowed to access, using directives to explicitly allow or disallow search and AI crawlers from their whole site, or just parts of it. The directives within the file are effectively a “keep out” sign and don’t provide any formal access control. Having said that, Cloudflare’s managed robots.txt feature automatically updates a site’s existing robots.txt or creates a robots.txt file on the site that includes directives asking popular AI bot operators to not use the content for AI model training. In addition, our AI Crawl Control capabilities can track violations of a site’s robots.txt directives, and give the site owner the ability to block requests from the offending user agent.

On Cloudflare Radar, we provide insight into the number of robots.txt files found among our top 10,000 domains and the full/partial disposition of the allow and disallow directives found within the files for selected crawler user agents. (In this context, “full” refers to directives that apply to the whole site, and “partial” refers to directives that apply to specified paths or file types.) Within the Year in Review microsite, we show how the disposition of these directives changed over the course of 2025.

The user agents with the highest number of fully disallowed directives are those associated with AI crawlers, including GPTBot, ClaudeBot, and CCBot. The directives for Googlebot and Bingbot crawlers, used for both search indexing and AI training, leaned heavily towards partial disallow, likely focused on cordoning off login endpoints and other non-content areas of a site. For these two bots, directives applying to the whole site remained a small fraction of the total number of disallow directives observed through the year. 

^{Robots.txt disallow directives by user agent}

The number of explicit allow directives found across the discovered robots.txt files was a fraction of the observed disallow directives, likely because allow is the default policy, absent any specific directive. Googlebot had the largest number of explicit allow directives, although over half of them were partial allows. Allow directives targeting AI crawlers were found across fewer domains, with directives targeting OpenAI’s crawlers leaning more towards explicit full allows. 

Google-Extended is a user agent token that web publishers can use to manage whether content that Google crawls from their sites may be used for training Gemini models or providing site content from the Google Search index to Gemini, and the number of allow directives targeting it tripled during the year — most partially allowed access at the start of the year, while the end of the year saw a larger number of directives that explicitly allowed full site access than those that allowed access to just some of the site’s content. 

^{Robots.txt allow directives by user agent}

The AI model landscape is rapidly evolving, with providers regularly releasing more powerful models, capable of tasks like text and image generation, speech recognition, and image classification. Cloudflare collaborates with AI model providers to ensure that Workers AI supports these models as soon as possible following their release, and we recently acquired Replicate to greatly expand our catalog of supported models. In February 2025, we introduced visibility on Radar into the popularity of publicly available supported models as well as the types of tasks that these models perform, based on customer account share. 

Throughout the year, Meta’s llama-3-8b-instruct model was dominant, with an account share (36.3%) more than three times larger than the next most popular models, OpenAI’s whisper (10.1%) and Stability AI’s stable-diffusion-xl-base-1.0 (9.8%). Both Meta and BAAI (Beijing Academy of Artificial Intelligence) had multiple models among the top 10, and the top 10 models had an account share of 89%, with the balance spread across a long tail of other models.

^{Most popular models on Workers AI in 2025, worldwide}

Task popularity was driven in large part by the top models, with text generation, text-to-image, and automatic speech recognition topping the list. Text generation was used by 48.2% of Workers AI customer accounts, nearly four times more than the text-to-image share of 12.3% and automatic speech recognition’s 11.0% share. 

^{Most popular tasks on Workers AI in 2025, worldwide}

In addition to the year-to-date analysis presented above, below we present point-in-time analyses of what is being crawled. Note that these insights are not included in the Year in Review microsite.

Within the AI section of Year in Review, we are looking at traffic from AI bots and crawlers globally, without regard for the geography associated with the account that owns the content being crawled. If we drill down a level geographically, using data from October 2025, and look at which bots generate the most crawling traffic for sites owned by customers with a billing address in a given geographic region, we find that Googlebot accounts for between 35% and 55% of crawler traffic in each region.

OpenAI’s GPTBot or Microsoft’s Bingbot are second most active, with crawling shares of 13-14%. In the developed economies across North America, Europe, and Oceania, Bingbot maintains a solid lead over AI crawlers. But for sites based in fast-growing markets across South America and Asia, GPTBot holds a slimmer lead over Bingbot.

In analyzing AI crawler activity by customer industry during October 2025, we found that Retail and Computer Software consistently attracted the most AI crawler traffic, together representing just over 40% of all activity.

Others in the top 10 accounted for much smaller shares of crawling activity. These top 10 industries accounted for just under 70% of crawling, with the balance spread across a long tail of other industries.

^{Industry share of AI crawling activity, October 2025}

The two leading mobile device operating systems globally are Apple’s iOS and Google’s Android. By analyzing information in the User-Agent header included with each Web request, we can calculate the distribution of traffic by client operating system throughout the year. Android devices generate the majority of mobile device traffic globally, due to the wide distribution of price points, form factors, and capabilities of such devices.

Globally, the share of traffic from iOS grew slightly year-over-year, up two percentage points to 35% in 2025. Looking at the top countries for iOS traffic share, Monaco had the highest share, at 70%, and iOS drove 50% or more of mobile device traffic in a total of 30 countries/regions, including Denmark (65%), Japan (57%), and Puerto Rico (52%).

^{Distribution of mobile device traffic by operating system in 2025, worldwide}

For countries/regions with higher Android usage, the shares were significantly larger. Twenty-seven had Android adoption above 90% in 2025, with Papua New Guinea the highest at 97%. Sudan, Malawi, Bangladesh, and Ethiopia also registered an Android share of 95% or more. Android was responsible for 50% or more of mobile device traffic in 175 countries/regions, with the Bahamas’ 51% share placing it at the bottom of that list. 

^{Distribution of iOS and Android usage in 2025}

HTTP (HyperText Transfer Protocol) is the protocol that makes the Web work. Over the last 30+ years, it has gone through several major revisions. The first standardized version, HTTP/1.0, was adopted in 1996, HTTP/1.1 in 1999, and HTTP/2 in 2015. HTTP/3, standardized in 2022, marked a significant update, running on top of a new transport protocol known as QUIC. Using QUIC as its underlying transport allows HTTP/3 to establish connections more quickly, as well as deliver improved performance by mitigating the effects of packet loss and network changes. Because it also provides encryption by default, using HTTP/3 mitigates the risk of attacks. 

Globally in 2025, 50% of requests to Cloudflare were made over HTTP/2, HTTP/1.x accounted for 29%, and the remaining 21% were made via HTTP/3. These shares are largely unchanged from 2024 — HTTP/2 and HTTP/3 gained just fractions of a percentage point this year.

^{Distribution of traffic by HTTP version in 2025, worldwide}

Geographically, usage of HTTP/3 appears to be both increasing and spreading. Last year, we noted that we had found eight countries/regions sending more than a third of their requests over HTTP/3. In 2025, 15 countries/regions sent more than a third of requests over HTTP/3, with Georgia’s 38% adoption just exceeding 2024’s top adoption rate of 37% in Réunion. (Looking at historical data, Georgia started the year around 46% HTTP/3 adoption, but dropped through the first half of the year before leveling off.) Armenia had the largest increase in HTTP/3 adoption year-over-year, jumping from 25% to 37%. 

Seven countries/regions saw overall HTTP/3 usage levels below 10% due to high levels of bot-originated HTTP/1.x traffic. These include Hong Kong, Dominica, Singapore, Ireland, Iran, Seychelles, and Gibraltar. 

To deliver a modern Web site, developers must capably integrate a growing collection of libraries and frameworks with third-party tools and platforms. All of these components must work together to ensure a performant, feature-rich, problem-free user experience. As in past years, we used Cloudflare Radar’s URL Scanner to scan Web sites associated with the top 5,000 domains to identify the most popular technologies and services used across eleven categories. 

jQuery is self-described as a fast, small, and feature-rich JavaScript library, and our scan found it on 8x as many sites as Slick, a JavaScript library used to display image carousels. React remained the top JavaScript framework used for building Web interfaces, found on twice as many scanned sites as Vue.js. PHP, node.js, and Java remained the most popular programming languages/technologies, holding a commanding lead over other languages, including Ruby, Python, Perl, and C.

^{Top Web site technologies, JavaScript libraries category in 2025}

WordPress remained the most popular content management system (CMS), though its share of scanned sites dropped to 47%, with the difference distributed across gains seen by multiple challengers. HubSpot and Marketo remained the top marketing automation platforms, with a combined share 10% higher YoY. Among A/B testing tools, VWO’s share grew by eight percentage points year-over-year, extending its lead over Optimizely, while Google Optimize, which was sunsetted in September 2023, saw its share fall from 14% to 4%.

Application programming interfaces (APIs) are the foundation of modern dynamic Web sites and both Web-based and native applications. These sites and applications rely heavily on automated API calls to provide customized information. Analyzing the Web traffic protected and delivered by Cloudflare, we can identify requests being made to API endpoints. By applying heuristics to these API-related requests determined to not be coming from a person using a browser or native mobile application, we can identify the top languages used to build API clients.

In 2025, 20% of automated API requests were made by Go-based clients, representing significant growth from Go’s 12% share in 2024. Python’s share also increased year-over-year, growing from 9.6% to 17%. Java jumped to third place, reaching an 11.2% share, up from 7.4% in 2024. Node.js, last year’s second-most popular language, saw its share fall to just 8.3% in 2025, pushing it down to fourth place, while .NET remained at the bottom of the top five, dropping to just 2.3%.

^{Most popular automated API client languages in 2025}

Cloudflare is in a unique position to measure search engine market share because we protect websites and applications for millions of customers. To that end, since the fourth quarter of 2021, we have been publishing quarterly reports on this data. We use the HTTP referer header to identify the search engine sending traffic to customer sites and applications, and present the market share data as an overall aggregate, as well as broken out by device type and operating system. (Device type and operating system insights are based on the User-Agent and Client Hints HTTP request headers.)

Globally, Google referred the most traffic to sites protected and delivered by Cloudflare, with a nearly 90% share in 2025. The other search engines in the top 5 include Bing (3.1%), Yandex (2.0%), Baidu (1.4%), and DuckDuckGo (1.2%). Looking at trends across the year, Yandex dropped from a 2.5% share in May to a 1.5% share in July, while Baidu grew from 0.9% in April to 1.6% in June.

^{Overall search engine market share in 2025, worldwide}

Yandex users are primarily based in Russia, where the domestic platform holds a 65% market share, almost double that of Google at 34%. In the Czech Republic, users prefer Google (84%), but local search engine Seznam’s 7.7% share is a strong showing compared to the second place search engines in other countries. 

^{Overall search engine market share in 2025, Czech Republic}

For traffic from “desktop” systems aggregated globally, Google’s market share drops to about 80%, while Bing’s jumps to nearly 11%. This is likely driven by the continued market dominance of Windows-based systems: On Windows, Google refers just 76% of traffic, while Bing refers about 14%. For traffic from mobile devices, Google holds almost 93% of market share, with the same share seen for traffic from both Android and iOS devices.

^{Overall search engine market share in 2025, Windows-based systems}

For additional details, including search engines aggregated under “Other”, please refer to the quarterly Search Engine Referral Reports on Cloudflare Radar.

Cloudflare is also in a unique position to measure browser market share, and we have been publishing quarterly reports on the topic for several years. To identify the browser and associated operating system making content requests, we use information from the User-Agent and Client Hints HTTP headers. We present browser market share data as an overall aggregate, as well as broken out by device type and operating system. Note that the shares of browsers available on both desktop and mobile devices, such as Google Chrome or Apple Safari, are presented in aggregate.

Globally, two-thirds of request traffic to Cloudflare came from Chrome in 2025, similar to its share last year. Safari, available exclusively on Apple devices, was the second most-popular browser, with a 15.4% market share. They were followed by Microsoft Edge (7.4%), Mozilla Firefox (3.7%) and Samsung Internet (2.3%). 

^{Overall browser market share in 2025, worldwide}

In Russia, Chrome remains the most popular with a 44% share, but the domestic Yandex Browser comes in a strong second with a 33% market share, as compared to the sub-10% shares for Safari, Edge, and Opera. Interestingly, the Yandex Browser actually beat Chrome by a percentage point (39% to 38%) in June before giving up significant market share to Chrome as the year progressed.

^{Overall browser market share in 2025, Russia}

As the default browser on iOS, Safari is far and away the most popular on such devices, with a 79% market share, four times Chrome’s 19% share. Less than 1% of requests come from DuckDuckGo, Firefox, and QQ Browser (developed in China by Tencent). In contrast, on Android, 85% of requests are from Chrome, while vendor-provided Samsung Internet is a distant second with a 6.6% share. Huawei Browser, another vendor-provided browser, is third at just 1%. And despite being the default browser on Windows, Edge’s 19% share pales in comparison to Chrome, which leads with a 69% share on that operating system.

^{Overall browser market share in 2025, iOS devices}

For additional details, including browsers aggregated under “Other”, please refer to the quarterly Browser Market Share Reports on Cloudflare Radar.

Internet outages continue to be an ever-present threat, and the potential impact of these outages continues to grow, as they can lead to economic losses, disrupted educational and government services, and limited communications. During 2025, we covered significant Internet disruptions and their associated causes in our quarterly summary posts (Q1, Q2, Q3) as well standalone posts covering major outages in Portugal & Spain and Afghanistan. The Cloudflare Radar Outage Center tracks these Internet outages, and uses Cloudflare traffic data for insights into their scope and duration.

Nearly half of the observed outages this year were related to Internet shutdowns intended to prevent cheating on academic exams. Countries including Iraq, Syria, and Sudan again implemented regular multi-hour shutdowns over the course of several weeks during exam periods. Other government-directed shutdowns in Libya and Tanzania were implemented in response to protests and civil unrest, while in Afghanistan, the Taliban ordered the shutdown of fiber optic Internet connectivity in multiple provinces as part of a drive to “prevent immorality.”

Cable cuts, affecting both submarine and domestic fiber optic infrastructure, were also a leading cause of Internet disruptions in 2025. These cuts resulted in network providers in countries/regions including the United States, South Africa, Haiti, Pakistan, and Hong Kong experiencing service disruptions lasting from several hours to several days. Other notable outages include one caused by a fire in a telecom building in Cairo, Egypt, which disrupted Internet connectivity across multiple service providers for several days, and another in Jamaica, where damage caused by Hurricane Melissa resulted in lower Internet traffic from the island for over a week.

Within the timeline on the Year in Review microsite, hovering over a dot will display information about that outage, and clicking on it will link to additional insights.

^{Over 170 major Internet outages were observed around the world during 2025}

Available IPv4 address space has been largely exhausted for a decade or more, though solutions like Network Address Translation have enabled network providers to stretch limited IPv4 resources. This has served in part to slow the adoption of IPv6, designed in the mid-1990s as a successor protocol to IPv4, and offers an expanded address space intended to better support the expected growth in the number of Internet-connected devices.

For nearly 15 years, Cloudflare has been a vocal and active advocate for IPv6 as well, launching solutions including Automatic IPv6 Gateway in 2011, which enabled free IPv6 support for all of our customers and IPv6 support by default for all of our customers in 2014. Simplistically, server-side support is only half of what is needed to drive IPv6 adoption, because end user connections need to support it as well. By aggregating and analyzing the IP version used for requests made to Cloudflare across the year, we can get insight into the distribution of traffic across IPv6 and IPv4.

Globally, 29% of IPv6-capable (“dual-stack”) requests for content were made over IPv6, up a percentage point from 28% in 2024. India again topped the list with an IPv6 adoption rate of 67%, followed by just three other countries/regions (Malaysia, Saudi Arabia, and Uruguay) that also made more than half of such requests over IPv6, the same as last year. Some of the largest gains were seen in Belize, which grew from 4.3% to 24% year-over-year, and Qatar, which saw its adoption nearly double to 33% in 2025. Unfortunately, some countries/regions still lag the leaders, with 94 seeing adoption rates below 10%, including Russia (8.6%), Ireland (6.5%), and Hong Kong (3.0%). Even further behind are the 20 countries/regions with adoption rates below 1%, including Tanzania (0.9%), Syria (0.3%), and Gibraltar (0.1%).

^{Distribution of traffic by IP version in 2025, worldwide}

^{Top five countries for IPv6 adoption in 2025}

Over the past decade or so, we have turned to Internet speed tests for many purposes: keeping our service providers honest, troubleshooting a problematic connection, or showing off a particularly high download speed on social media. In fact, we’ve become conditioned to focus on download speeds as the primary measure of a connection’s quality. While it is absolutely an important metric, for increasingly popular use cases — like videoconferencing, live-streaming, and online gaming — strong upload speeds and low latency are also critical. However, even when Internet providers offer service tiers that include high symmetric speeds and lower latency, consumer adoption is often mixed due to cost, availability, or other issues.

Tests on speed.cloudflare.com measure both download and upload speeds, as well as loaded and unloaded latency. By aggregating the results of tests taken around the world during 2025, we can get a country/region perspective on average values for these connection quality metrics, as well as insight into the distribution of the measurements.

Europe was well-represented among those with the highest average download speeds in 2025. Spain, Hungary, Portugal, Denmark, Romania, and France were all in the top 10, with both Spain and Hungary averaging download speeds above 300 Mbps. Spain’s average grew by 25 Mbps from 2024, while Hungary’s jumped 46 Mbps. Meanwhile, Asian countries had many of the highest average upload speeds, with South Korea, Macau, Singapore, and Japan reaching the top 10, all seeing averages in excess of 130 Mbps.

But it was Spain that topped the list for the upload metric as well at 206 Mbps, up 13 Mbps from 2024. The country’s strong showing across both speed metrics is potentially attributable to “UNICO-Broadband,” a “call for projects by telecommunications operators aiming at the deployment of high-speed broadband infrastructure capable of providing services at symmetric speeds of at least 300 Mbps, scalable at 1 Gbps,” which aimed to cover 100 % of the population in 2025.

^{Countries/regions with the highest download speeds in 2025, worldwide}

As noted above, low latency connections are needed to provide users with good gaming and videoconferencing/streaming experiences. The latency metric can be broken down into loaded and idle latency. The former measures latency on a loaded connection, where bandwidth is actively being consumed, while the latter measures latency on an “idle” connection, when there is no other network traffic present. (These definitions are from the speed test application’s perspective.) 

In 2025, a number of European countries were among those with both the lowest idle and loaded latencies. For average idle latency, Iceland measured the lowest at 13 ms, just 2 ms better than Moldova. In addition to these two, Portugal, Spain, and Hungary also ranked among the top 10, all with average idle latencies below 20 ms. Moldova topped the list of countries/regions with the lowest average loaded latency, at 73 ms. Hungary, Spain, Belgium, Portugal, Slovakia, and Slovenia were also part of the top 10, all with average loaded latencies below 100 ms.

^{Measured idle/loaded latency, Moldova}

As we discussed above, the speed test at speed.cloudflare.com measures a user’s connection speeds and latency. We reviewed the aggregate findings from those tests, highlighting the countries/regions with the best results. However, we also wondered about test activity around the world -– where are users most concerned about their connection quality, and how frequently do they perform tests? A new animated Year in Review visualization illustrates speed test activity, aggregated weekly.

Data is aggregated at a regional level and the associated activity is plotted on the map, with circles sized based on the number of tests taken each week. Note that locations with fewer than 100 speed tests per week are not plotted. Looking at test volume across the year, the greater London and Los Angeles areas were most active, as were Tokyo and Hong Kong and several U.S. cities.

Animating the graph to see changes across the year, a number of week-over-week surges in test volume are visible. These include in the Nairobi, Kenya, area during the seven-day period ending June 10; in the Tehran, Iran, area the period ending July 29; across multiple areas in Russia the period ending August 5; and in the Karnataka, India, area the period ending October 28. It isn’t clear what drove these increases in test volume — the Cloudflare Radar Outage Center does not show any observed Internet outages impacting those areas around those times, so it is unlikely to be subscribers testing the restoration of connectivity.

^{Cloudflare speed test activity by location in 2025}

For better or worse, over the last quarter-century, mobile devices have become an indispensable part of everyday life. Adoption varies around the world — statistics from the World Bank show multiple countries/regions with mobile phone ownership above 90%, while in several others, ownership rates are below 10%, as of October 2025. In some countries/regions, mobile devices primarily connect to the Internet via Wi-Fi, while other countries/regions are “mobile first,” where 4G/5G services are the primary means of Internet access.

Information contained within the User-Agent header included with each request to Cloudflare enables us to categorize it as coming from a mobile, desktop, or other type of device. Aggregating this categorization globally across 2025 found that 43% of requests were from mobile devices, up from 41% in 2024. The balance came from “classic” laptop and desktop type devices. Similar to an observation made last year, these traffic shares were in line with those measured in Year in Review reports dating back to 2022, suggesting that mobile device usage has achieved a “steady state.”

In 117 countries/regions, more than half of requests came from mobile devices, led by Sudan and Malawi at 75% and 74% respectively. Five other African countries/regions — Eswatini (Swaziland), Yemen, Botswana, Mozambique, and Somalia — also had mobile request shares above 70% in 2025, in line with strong mobile phone ownership in the region. Among countries/regions with low mobile device traffic share, Gibraltar was the only one below 10% (at 5.1%), with just six others originating less than a quarter of requests from mobile devices. This is fewer than in 2024, when a dozen countries/regions had a mobile share below 25%.

^{Distribution of traffic by device type in 2025, worldwide}

^{Global distribution of traffic by device type in 2025}

Cloudflare automatically mitigates attack traffic targeting customer websites and applications using DDoS mitigation techniques or Web Application Firewall (WAF) Managed Rules, protecting them from a variety of threats posed by malicious actors. We also enable customers to mitigate traffic, even if it isn’t malicious, using techniques like rate-limiting requests or blocking all traffic from a given location. The need to do so may be driven by regulatory or business requirements. We looked at the overall share of traffic to Cloudflare’s network throughout 2025 that was mitigated for any reason, as well as the share that was blocked as a DDoS attack or by WAF Managed Rules.

This year, 6.2% of global traffic was mitigated, down a quarter of a percentage point from 2024. 3.3% of traffic was mitigated as a DDoS attack, or by managed rules, up one-tenth of a percentage point year over year. General mitigations were applied to more than 10% of the traffic coming from over 30 countries/regions, while 14 countries/regions had DDoS/WAF mitigations applied to more than 10% of originated traffic. Both counts were down in comparison to 2024. 

Equatorial Guinea had the largest shares of mitigated traffic with 40% generally mitigated and 29% with DDoS/WAF mitigations applied. These shares grew over the last year, from 26% (general) and 19% (DDoS/WAF). In contrast, Dominica had the smallest shares of mitigated traffic, with just 0.7% of traffic mitigated, with DDoS/WAF mitigations applied to just 0.1%.

The large increase in mitigated traffic seen during July in the graph below is due to a very large DDoS attack campaign that primarily targeted a single Cloudflare customer domain.

^{Mitigated traffic trends in 2025, worldwide}

A bot is a software application programmed to do certain tasks, and Cloudflare uses advanced heuristics to differentiate between bot traffic and human traffic, scoring each request on the likelihood that it originates from a bot or a human user. By monitoring traffic suspected to be from bots, site and application owners can spot and, if necessary, block potentially malicious activity. However, not all bots are malicious — bots can also be helpful, and Cloudflare maintains a directory of verified bots that includes those used for things like search engine indexing, security scanning, and site/application monitoring. Regardless of intent, we analyzed where bot traffic was originating from in 2025, using the IP address of a request to identify the network (autonomous system) and country/region associated with the bot making the request. 

Globally, the top 10 countries/regions accounted for 71% of observed bot traffic. Forty percent originated from the United States, far ahead of Germany’s 6.5% share. The US share was up over five percentage points from 2024, while Germany’s share was down a fraction of a percentage point. The remaining countries in the top 10 all contributed bot traffic shares below 5% in 2025.

^{Global bot traffic distribution by source country/region in 2025}

Looking at bot traffic by network, we found that cloud platforms remained among the leading sources. This is due to a number of factors, including the ease of using automated tools to quickly provision compute resources, their relatively low cost, their broadly distributed geographic footprints, and the platforms’ high-bandwidth Internet connectivity. 

Two autonomous systems associated with Amazon Web Services accounted for a total of 14.4% of observed bot traffic, and two associated with Google Cloud were responsible for a combined 9.7% of bot traffic. They were followed by Microsoft Azure, which originated 5.5% of bot traffic. The shares from all three platforms were up as compared to 2024. These cloud platforms have a strong regional data center presence in many of the countries/regions in the top 10. Elsewhere, around the world, local telecommunications providers frequently accounted for the largest shares of automated bot traffic observed in those countries/regions.

^{Global bot traffic distribution by source network in 2025}

Attackers are constantly shifting their tactics and targets, mixing things up in an attempt to evade detection, or based on the damage they intend to cause. They may try to cause financial harm to businesses by targeting ecommerce sites during a busy shopping period, make a political statement by attacking government-related or civil society sites, or attempt to knock opponents offline by attacking a game server. To identify vertical-targeted attack activity during 2025, we analyzed mitigated traffic for customers that had an associated industry and vertical within their customer record. Mitigated traffic was aggregated weekly by source country/region across 17 target verticals.

Organizations in the "People and Society” vertical were the most targeted across the year, with 4.4% of global mitigated traffic targeting the vertical. Customers classified as “People and Society” include religious institutions, nonprofit organizations, civic & social organizations, and libraries. The vertical started out the year with under 2% of mitigated traffic, but saw the share jump to 10% the week of March 5, and increase to over 17% by the end of the month. Other attack surges targeting these sites occurred in late April (to 19.1%) and early July (to 23.2%). Many of these types of organizations are protected by Cloudflare’s Project Galileo, and this blog post details the attacks and threats they experienced in 2024 and 2025.

Gambling/Games, the most-targeted vertical last year, saw its share of mitigated attacks drop by more than half year-over-year, to just 2.6%. While one might expect to see attacks targeting gambling sites peak around major sporting events like the Super Bowl and March Madness, such a trend was not evident, as attack share peaked at 6.5% the week of March 5 — a month after the Super Bowl, and a couple of weeks before the start of March Madness.

^{Global mitigated traffic share by vertical in 2025, summary view}

Border Gateway Protocol (BGP) is the Internet’s core routing protocol, enabling traffic to flow between source and destination by communicating routes between networks. However, because it relies on trust between connected networks, incorrect information shared between peers (intentionally or not) can send traffic to the wrong place — potentially to systems under control of an attacker. To address this, Resource Public Key Infrastructure (RPKI) was developed as a cryptographic method of signing records that associate a BGP route announcement with the correct originating autonomous system (AS) number to ensure that the information being shared originally came from a network that is allowed to do so. Cloudflare has been a vocal advocate for routing security, including as a founding participant in the MANRS CDN and Cloud Programme and by providing a public tool that enables users to test whether their Internet provider has implemented BGP safely. 

We analyzed data available on Cloudflare Radar’s Routing page to determine the share of RPKI valid routes and how that share changed throughout 2025, as well as determining the share of IP address space covered by valid routes. The latter metric is noteworthy because a route announcement covering a large amount of IP address space (millions of IPv4 addresses) has a greater potential impact than an announcement covering a small block of IP address space (hundreds of IPv4 addresses).

We started 2025 with 50% valid IPv4 routes, growing to 53.9% by December 2. The share of valid IPv6 routes increased to 60.1%, up 4.7 percentage points. Looking at the global share of IP address space covered by valid routes, IPv4 increased to 48.5%, a three percentage point increase. The share of IPv6 address space covered by valid routes fell slightly to 61.6%. Although the year-over-year changes for these metrics are slowing, we have made significant progress over the last five years. Since the start of 2020, the share of RPKI valid IPv4 routes and IPv4 address space have both grown by approximately 3x.

^{Shares of global RPKI valid routing entries by IP version in 2025}

^{Shares of globally announced IP address space covered by RPKI valid routes in 2025}

Barbados saw the biggest growth in the share of valid IPv4 routes, growing from 2.2% to 20.8%. Looking at valid IPv6 routes, Mali saw the most significant share growth in 2025, from 10.0% to 58.3%. 

Barbados also experienced the biggest increase in the share of IPv4 space covered by valid routes, jumping from just 2.0% to 18.6%. For IPv6 address space, both Tajikistan and Dominica went from having effectively no space covered by valid routes at the start of the year, to 5.5% and 3.5% respectively. 

In our quarterly DDoS Report series (Q1, Q2, Q3), we have highlighted the increasing frequency and size of hyper-volumetric network layer attacks targeting Cloudflare customers and Cloudflare’s infrastructure. We define a “hyper-volumetric network layer attack” as one that operates at Layer 3/4 and that peaks at more than one terabit per second (1 Tbps) or more than one billion packets per second (1 Bpps). These reports provide a quarterly perspective, but we also wanted to show a view of activity across the year to understand when attackers are most active, and how attack sizes have grown over time. 

Looking at hyper-volumetric attack activity in 2025 from a Tbps perspective, July saw the largest number of such attacks, at over 500, while February saw the fewest, at just over 150. Attack intensity remained generally below 5 Tbps, although a 10 Tbps attack blocked at the end of August was a harbinger of things to come. This attack was the first of a campaign of >10 Tbps attacks that took place during the first week of September, ahead of a series of >20 Tbps attacks during the last week of the month. In early October, multiple increasingly larger hyper-volumetric attacks were observed, with the largest for the month peaking at 29.7 Tbps. However, that record was soon eclipsed, as an early November attack reached 31.4 Tbps.

From a Bpps perspective, hyper-volumetric attack activity was much lower, with November experiencing the most (over 140), while just three were seen in February and June. Attack intensity across the year generally remained below 4 Bpps through late August, though a succession of increasingly larger attacks were seen over the next several months, peaking in October. Although the intensity of most of the 110+ attacks blocked in October was below 5 Bpps, a 14 Bpps attack seen during the month was the largest hyper-volumetric attack by packets per second blocked during the year, besting five other successive record-setting attacks that occurred in September.

^{Peak DDoS attack sizes in 2025}

Recent statistics suggest that email remains the top communication channel for external business contact, despite the growing enterprise use of collaboration/messaging apps. Given its broad enterprise usage, attackers still find it to be an attractive entry point into corporate networks. Generative AI tools make it easier to craft highly targeted malicious emails that convincingly impersonate trusted brands or legitimate senders (like corporate executives) but contain deceptive links, dangerous attachments, or other types of threats. Cloudflare Email Security protects customers from email-based attacks, including those carried out through targeted malicious email messages. 

In 2025, an average of 5.6% of emails analyzed by Cloudflare were found to be malicious. The share of messages processed by Cloudflare Email Security that were found to be malicious generally ranged between 4% and 6% throughout most of the year. Our data shows a jump in malicious email share starting in October, likely due to an improved classification system implemented by Cloudflare Email Security.  

^{Global malicious email share trends in 2025}

Deceptive links were the top malicious email threat category in 2025, found in 52% of messages, up from 43% in 2024. Since the display text for a hyperlink in HTML can be arbitrarily set, attackers can make a URL appear as if it links to a benign site when, in fact, it is actually linking to a malicious resource that can be used to steal login credentials or download malware. The share of processed emails containing deceptive links was as high as 70% in late April, and again in mid-November.

Identity deception occurs when an attacker sends an email claiming to be someone else. They may do this using domains that look similar, are spoofed, or use display name tricks to appear to be coming from a trusted domain. Brand impersonation is a form of identity deception where an attacker sends a phishing message that impersonates a recognizable company or brand. Brand impersonation may also use display name spoofing or domain impersonation. Identity deception (38%) and brand impersonation (32%) were growing threats in 2025, up from 35% and 23% respectively in 2024. Both saw an increase in mid-November.

^{Email threat category trends in 2025, worldwide}

In addition to providing traffic, geographic distribution, and digital certificate insights for Top Level Domains (TLDs) like .com or .us, Cloudflare Radar also provides insights into the “most abused” TLDs – those with domains that we have found are originating the largest shares of malicious and spam email among messages analyzed by Cloudflare Email Security. The analysis is based on the sending domain’s TLD, found in the From: header of an email message. For example, if a message came from sender@example.com, then example.com is the sending domain, and .com is the associated TLD. For the Year in Review analysis, we only included TLDs from which we saw an average minimum of 30 messages per hour.

Based on messages analyzed throughout 2025, we found that .christmas and .lol were the most abused TLDs, with 99.8% and 99.6% of messages from these TLDs respectively characterized as either spam or malicious. Sorting the list of TLDs by malicious email share, .cfd and .sbs both had more than 90% of analyzed emails categorized as malicious. The .best TLD was the worst in terms of spam email share, with 69% of email messages characterized as spam.

^{TLDs originating the largest total shares of malicious and spam email in 2025}

Although the Internet and the Web continue to evolve and change over time, it appears that some of the key metrics have become fairly stable. However, we expect that others, such as those metrics tracking AI trends, will shift over the coming years as that space evolves at a rapid pace. 

We encourage you to visit the Cloudflare Radar 2025 Year In Review microsite and explore the trends for your country/region, and consider how they impact your organization as you plan for 2026. You can also get near real-time insight into many of these metrics and trends on Cloudflare Radar. And as noted above, for insights into the top Internet services across multiple industry categories and countries/regions, we encourage you to read the companion Year in Review blog post.

If you have any questions, you can contact the Cloudflare Radar team at radar@cloudflare.com or on social media at @CloudflareRadar (X), https://noc.social/@cloudflareradar (Mastodon), and radar.cloudflare.com (Bluesky).

As the saying goes, it takes a village to make our annual Year in Review happen, from aggregating and analyzing the data, to creating the microsite, to developing associated content. I’d like to acknowledge those team members that contributed to this year’s effort, with thanks going out to: Jorge Pacheco, Sabina Zejnilovic, Carlos Azevedo, Mingwei Zhang, Sofia Cardita (data analysis); André Páscoa, Nuno Pereira (frontend development); João Tomé (Most Popular Internet Services); David Fidalgo, Janet Villarreal, and the internationalization team (translations); Jackie Dutton, Kari Linder, Guille Lasarte (Communications); Laurel Wamsley (blog editing); and Paula Tavares (Engineering Management), as well as other colleagues across Cloudflare for their support and assistance.

^{Note: This post was updated to clarify the relationship of the internal WAF tool with the incident on Dec. 5.}

On December 5, 2025, at 08:47 UTC (all times in this blog are UTC), a portion of Cloudflare’s network began experiencing significant failures. The incident was resolved at 09:12 (~25 minutes total impact), when all services were fully restored.

A subset of customers were impacted, accounting for approximately 28% of all HTTP traffic served by Cloudflare. Several factors needed to combine for an individual customer to be affected as described below.

The issue was not caused, directly or indirectly, by a cyber attack on Cloudflare’s systems or malicious activity of any kind. Instead, it was triggered by changes being made to our body parsing logic while attempting to detect and mitigate an industry-wide vulnerability disclosed this week in React Server Components.

Any outage of our systems is unacceptable, and we know we have let the Internet down again following the incident on November 18. We will be publishing details next week about the work we are doing to stop these types of incidents from occurring.

The graph below shows HTTP 500 errors served by our network during the incident timeframe (red line at the bottom), compared to unaffected total Cloudflare traffic (green line at the top).

Cloudflare's Web Application Firewall (WAF) provides customers with protection against malicious payloads, allowing them to be detected and blocked. To do this, Cloudflare’s proxy buffers HTTP request body content in memory for analysis. Before today, the buffer size was set to 128KB.

As part of our ongoing work to protect customers who use React against a critical vulnerability, CVE-2025-55182, we started rolling out an increase to our buffer size to 1MB, the default limit allowed by Next.js applications, to make sure as many customers as possible were protected.

This first change was being rolled out using our gradual deployment system. During rollout, we noticed that our internal WAF testing tool did not support the increased buffer size. As this internal test tool was not needed at that time and had no effect on customer traffic, we made a second change to turn it off.

This second change of turning off our WAF testing tool was implemented using our global configuration system. This system does not perform gradual rollouts, but rather propagates changes within seconds to the entire fleet of servers in our network and is under review following the outage we experienced on November 18. 

Unfortunately, in our FL1 version of our proxy, under certain circumstances, the second change of turning off our WAF rule testing tool caused an error state that resulted in 500 HTTP error codes to be served from our network.

As soon as the change propagated to our network, code execution in our FL1 proxy reached a bug in our rules module which led to the following Lua exception:

[lua] Failed to run module rulesets callback late_routing: /usr/local/nginx-fl/lua/modules/init.lua:314: attempt to index field 'execute' (a nil value)

resulting in HTTP code 500 errors being issued.

The issue was identified shortly after the change was applied, and was reverted at 09:12, after which all traffic was served correctly.

Customers that have their web assets served by our older FL1 proxy AND had the Cloudflare Managed Ruleset deployed were impacted. All requests for websites in this state returned an HTTP 500 error, with the small exception of some test endpoints such as /cdn-cgi/trace.

Customers that did not have the configuration above applied were not impacted. Customer traffic served by our China network was also not impacted.

Cloudflare’s rulesets system consists of sets of rules which are evaluated for each request entering our system. A rule consists of a filter, which selects some traffic, and an action which applies an effect to that traffic. Typical actions are “block”, “log”, or “skip”. Another type of action is “execute”, which is used to trigger evaluation of another ruleset.

Our internal logging system uses this feature to evaluate new rules before we make them available to the public. A top level ruleset will execute another ruleset containing test rules. It was these test rules that we were attempting to disable.

We have a killswitch subsystem as part of the rulesets system which is intended to allow a rule which is misbehaving to be disabled quickly. This killswitch system receives information from our global configuration system mentioned in the prior sections. We have used this killswitch system on a number of occasions in the past to mitigate incidents and have a well-defined Standard Operating Procedure, which was followed in this incident.

However, we have never before applied a killswitch to a rule with an action of “execute”. When the killswitch was applied, the code correctly skipped the evaluation of the execute action, and didn’t evaluate the sub-ruleset pointed to by it. However, an error was then encountered while processing the overall results of evaluating the ruleset:

if rule_result.action == "execute" then
  rule_result.execute.results = ruleset_results[tonumber(rule_result.execute.results_index)]
end

This code expects that, if the ruleset has action=”execute”, the “rule_result.execute” object will exist. However, because the rule had been skipped, the rule_result.execute object did not exist, and Lua returned an error due to attempting to look up a value in a nil value.

This is a straightforward error in the code, which had existed undetected for many years. This type of code error is prevented by languages with strong type systems. In our replacement for this code in our new FL2 proxy, which is written in Rust, the error did not occur.

We made an unrelated change that caused a similar, longer availability incident two weeks ago on November 18, 2025. In both cases, a deployment to help mitigate a security issue for our customers propagated to our entire network and led to errors for nearly all of our customer base.

We have spoken directly with hundreds of customers following that incident and shared our plans to make changes to prevent single updates from causing widespread impact like this. We believe these changes would have helped prevent the impact of today’s incident but, unfortunately, we have not finished deploying them yet.

We know it is disappointing that this work has not been completed yet. It remains our first priority across the organization. In particular, the projects outlined below should help contain the impact of these kinds of changes:

• Enhanced Rollouts & Versioning: Similar to how we slowly deploy software with strict health validation, data used for rapid threat response and general configuration needs to have the same safety and blast mitigation features. This includes health validation and quick rollback capabilities among other things.

• Streamlined break glass capabilities: Ensure that critical operations can still be achieved in the face of additional types of failures. This applies to internal services as well as all standard methods of interaction with the Cloudflare control plane used by all Cloudflare customers.

• "Fail-Open" Error Handling: As part of the resilience effort, we are replacing the incorrectly applied hard-fail logic across all critical Cloudflare data-plane components. If a configuration file is corrupt or out-of-range (e.g., exceeding feature caps), the system will log the error and default to a known-good state or pass traffic without scoring, rather than dropping requests. Some services will likely give the customer the option to fail open or closed in certain scenarios. This will include drift-prevention capabilities to ensure this is enforced continuously.

Before the end of next week we will publish a detailed breakdown of all the resiliency projects underway, including the ones listed above. While that work is underway, we are locking down all changes to our network in order to ensure we have better mitigation and rollback systems before we begin again.

These kinds of incidents, and how closely they are clustered together, are not acceptable for a network like ours. On behalf of the team at Cloudflare we want to apologize for the impact and pain this has caused again to our customers and the Internet as a whole.

The issue was not caused, directly or indirectly, by a cyber attack or malicious activity of any kind. Instead, it was triggered by a change to one of our database systems' permissions which caused the database to output multiple entries into a “feature file” used by our Bot Management system. That feature file, in turn, doubled in size. The larger-than-expected feature file was then propagated to all the machines that make up our network.

The software running on these machines to route traffic across our network reads this feature file to keep our Bot Management system up to date with ever changing threats. The software had a limit on the size of the feature file that was below its doubled size. That caused the software to fail.

After we initially wrongly suspected the symptoms we were seeing were caused by a hyper-scale DDoS attack, we correctly identified the core issue and were able to stop the propagation of the larger-than-expected feature file and replace it with an earlier version of the file. Core traffic was largely flowing as normal by 14:30. We worked over the next few hours to mitigate increased load on various parts of our network as traffic rushed back online. As of 17:06 all systems at Cloudflare were functioning as normal.

We are sorry for the impact to our customers and to the Internet in general. Given Cloudflare's importance in the Internet ecosystem any outage of any of our systems is unacceptable. That there was a period of time where our network was not able to route traffic is deeply painful to every member of our team. We know we let you down today.

This post is an in-depth recount of exactly what happened and what systems and processes failed. It is also the beginning, though not the end, of what we plan to do in order to make sure an outage like this will not happen again.

The chart below shows the volume of 5xx error HTTP status codes served by the Cloudflare network. Normally this should be very low, and it was right up until the start of the outage.

The volume prior to 11:20 is the expected baseline of 5xx errors observed across our network. The spike, and subsequent fluctuations, show our system failing due to loading the incorrect feature file. What’s notable is that our system would then recover for a period. This was very unusual behavior for an internal error.

The explanation was that the file was being generated every five minutes by a query running on a ClickHouse database cluster, which was being gradually updated to improve permissions management. Bad data was only generated if the query ran on a part of the cluster which had been updated. As a result, every five minutes there was a chance of either a good or a bad set of configuration files being generated and rapidly propagated across the network.

This fluctuation made it unclear what was happening as the entire system would recover and then fail again as sometimes good, sometimes bad configuration files were distributed to our network. Initially, this led us to believe this might be caused by an attack. Eventually, every ClickHouse node was generating the bad configuration file and the fluctuation stabilized in the failing state.

Errors continued until the underlying issue was identified and resolved starting at 14:30. We solved the problem by stopping the generation and propagation of the bad feature file and manually inserting a known good file into the feature file distribution queue. And then forcing a restart of our core proxy.

The remaining long tail in the chart above is our team restarting remaining services that had entered a bad state, with 5xx error code volume returning to normal at 17:06.

The following services were impacted:

As well as returning HTTP 5xx errors, we observed significant increases in latency of responses from our CDN during the impact period. This was due to large amounts of CPU being consumed by our debugging and observability systems, which automatically enhance uncaught errors with additional debugging information.

Every request to Cloudflare takes a well-defined path through our network. It could be from a browser loading a webpage, a mobile app calling an API, or automated traffic from another service. These requests first terminate at our HTTP and TLS layer, then flow into our core proxy system (which we call FL for “Frontline”), and finally through Pingora, which performs cache lookups or fetches data from the origin if needed.

We previously shared more detail about how the core proxy works here. 

As a request transits the core proxy, we run the various security and performance products available in our network. The proxy applies each customer’s unique configuration and settings, from enforcing WAF rules and DDoS protection to routing traffic to the Developer Platform and R2. It accomplishes this through a set of domain-specific modules that apply the configuration and policy rules to traffic transiting our proxy.

One of those modules, Bot Management, was the source of today’s outage. 

Cloudflare’s Bot Management includes, among other systems, a machine learning model that we use to generate bot scores for every request traversing our network. Our customers use bot scores to control which bots are allowed to access their sites — or not.

The model takes as input a “feature” configuration file. A feature, in this context, is an individual trait used by the machine learning model to make a prediction about whether the request was automated or not. The feature configuration file is a collection of individual features.

This feature file is refreshed every few minutes and published to our entire network and allows us to react to variations in traffic flows across the Internet. It allows us to react to new types of bots and new bot attacks. So it’s critical that it is rolled out frequently and rapidly as bad actors change their tactics quickly.

A change in our underlying ClickHouse query behaviour (explained below) that generates this file caused it to have a large number of duplicate “feature” rows. This changed the size of the previously fixed-size feature configuration file, causing the bots module to trigger an error.

As a result, HTTP 5xx error codes were returned by the core proxy system that handles traffic processing for our customers, for any traffic that depended on the bots module. This also affected Workers KV and Access, which rely on the core proxy.

Unrelated to this incident, we were and are currently migrating our customer traffic to a new version of our proxy service, internally known as FL2. Both versions were affected by the issue, although the impact observed was different.

Customers deployed on the new FL2 proxy engine, observed HTTP 5xx errors. Customers on our old proxy engine, known as FL, did not see errors, but bot scores were not generated correctly, resulting in all traffic receiving a bot score of zero. Customers that had rules deployed to block bots would have seen large numbers of false positives. Customers who were not using our bot score in their rules did not see any impact.

Throwing us off and making us believe this might have been an attack was another apparent symptom we observed: Cloudflare’s status page went down. The status page is hosted completely off Cloudflare’s infrastructure with no dependencies on Cloudflare. While it turned out to be a coincidence, it led some of the team diagnosing the issue to believe that an attacker may be targeting both our systems as well as our status page. Visitors to the status page at that time were greeted by an error message:

In the internal incident chat room, we were concerned that this might be the continuation of the recent spate of high volume Aisuru DDoS attacks:

I mentioned above that a change in the underlying query behaviour resulted in the feature file containing a large number of duplicate rows. The database system in question uses ClickHouse’s software.

For context, it’s helpful to know how ClickHouse distributed queries work. A ClickHouse cluster consists of many shards. To query data from all shards, we have so-called distributed tables (powered by the table engine Distributed) in a database called default. The Distributed engine queries underlying tables in a database r0. The underlying tables are where data is stored on each shard of a ClickHouse cluster.

Queries to the distributed tables run through a shared system account. As part of efforts to improve our distributed queries security and reliability, there’s work being done to make them run under the initial user accounts instead.

Before today, ClickHouse users would only see the tables in the default database when querying table metadata from ClickHouse system tables such as system.tables or system.columns.

Since users already have implicit access to underlying tables in r0, we made a change at 11:05 to make this access explicit, so that users can see the metadata of these tables as well. By making sure that all distributed subqueries can run under the initial user, query limits and access grants can be evaluated in a more fine-grained manner, avoiding one bad subquery from a user affecting others.

The change explained above resulted in all users accessing accurate metadata about tables they have access to. Unfortunately, there were assumptions made in the past, that the list of columns returned by a query like this would only include the “default” database:

SELECT name, type FROM system.columns WHERE table = 'http_requests_features' order by name;

Note how the query does not filter for the database name. With us gradually rolling out the explicit grants to users of a given ClickHouse cluster, after the change at 11:05 the query above started returning “duplicates” of columns because those were for underlying tables stored in the r0 database.

This, unfortunately, was the type of query that was performed by the Bot Management feature file generation logic to construct each input “feature” for the file mentioned at the beginning of this section. 

The query above would return a table of columns like the one displayed (simplified example):

However, as part of the additional permissions that were granted to the user, the response now contained all the metadata of the r0 schema effectively more than doubling the rows in the response ultimately affecting the number of rows (i.e. features) in the final file output. 

Each module running on our proxy service has a number of limits in place to avoid unbounded memory consumption and to preallocate memory as a performance optimization. In this specific instance, the Bot Management system has a limit on the number of machine learning features that can be used at runtime. Currently that limit is set to 200, well above our current use of ~60 features. Again, the limit exists because for performance reasons we preallocate memory for the features.

When the bad file with more than 200 features was propagated to our servers, this limit was hit — resulting in the system panicking. The FL2 Rust code that makes the check and was the source of the unhandled error is shown below:

This resulted in the following panic which in turn resulted in a 5xx error:

thread fl2_worker_thread panicked: called Result::unwrap() on an Err value

Other systems that rely on our core proxy were impacted during the incident. This included Workers KV and Cloudflare Access. The team was able to reduce the impact to these systems at 13:04, when a patch was made to Workers KV to bypass the core proxy. Subsequently, all downstream systems that rely on Workers KV (such as Access itself) observed a reduced error rate. 

The Cloudflare Dashboard was also impacted due to both Workers KV being used internally and Cloudflare Turnstile being deployed as part of our login flow.

Turnstile was impacted by this outage, resulting in customers who did not have an active dashboard session being unable to log in. This showed up as reduced availability during two time periods: from 11:30 to 13:10, and between 14:40 and 15:30, as seen in the graph below.

The first period, from 11:30 to 13:10, was due to the impact to Workers KV, which some control plane and dashboard functions rely upon. This was restored at 13:10, when Workers KV bypassed the core proxy system. The second period of impact to the dashboard occurred after restoring the feature configuration data. A backlog of login attempts began to overwhelm the dashboard. This backlog, in combination with retry attempts, resulted in elevated latency, reducing dashboard availability. Scaling control plane concurrency restored availability at approximately 15:30.

Now that our systems are back online and functioning normally, work has already begun on how we will harden them against failures like this in the future. In particular we are:

• Hardening ingestion of Cloudflare-generated configuration files in the same way we would for user-generated input

• Enabling more global kill switches for features

• Eliminating the ability for core dumps or other error reports to overwhelm system resources

• Reviewing failure modes for error conditions across all core proxy modules

Today was Cloudflare's worst outage since 2019. We've had outages that have made our dashboard unavailable. Some that have caused newer features to not be available for a period of time. But in the last 6+ years we've not had another outage that has caused the majority of core traffic to stop flowing through our network.

An outage like today is unacceptable. We've architected our systems to be highly resilient to failure to ensure traffic will always continue to flow. When we've had outages in the past it's always led to us building new, more resilient systems.

On behalf of the entire team at Cloudflare, I would like to apologize for the pain we caused the Internet today.

As we have noted in the past, this post is intended as a summary overview of observed and confirmed disruptions, and is not an exhaustive or complete list of issues that have occurred during the quarter. A larger list of detected traffic anomalies is available in the Cloudflare Radar Outage Center. These anomalies are detected through significant deviations from expected traffic patterns observed across our network. Note that both bytes-based and request-based traffic graphs are used within the post to illustrate the impact of the observed disruptions — the choice of metric to include was generally made based on which better illustrated the impact of the disruption.

Regular drops in traffic from Sudan were observed between 12:00-15:00 UTC (14:00-17:00 local time) each day from July 7-10. Partial outages were observed at Sudatel (AS15706), and near-complete outages at SDN Mobitel (AS36998) and MTN Sudan (AS36972). Similar drops were also seen in traffic to our 1.1.1.1 DNS resolver from these impacted ASNs.

We have observed Sudan implementing government-directed Internet shutdowns in the past (2021, 2022), and given that the timing aligns with the last four days of postponed 2024 secondary school certificate examinations, in addition to fitting the pattern of short-duration disruptions repeating across multiple days, we believe that these drops in traffic were exam-related shutdowns as well. 

In our second quarter post, we covered the cellular connectivity-focused exam-related Internet shutdowns that Syria chose to implement this year in an effort to limit their impact. During the second quarter, the shutdowns associated with the “Basic Education Certificate” took place on June 21, 24, and 29 between 05:15 - 06:00 UTC (08:15 - 09:00 local time). Exams and associated shutdowns for the “Secondary Education Certificate” were scheduled to take place between July 12 and August 3, and during that period, we observed six additional Internet disruptions in Syria on July 12, 17, 21, 28, 31, and August 3, as shown in the graph below.

At the end of the exam period, the Syrian Ministry of Education posted a Telegram message that was presumably intended to justify the shutdowns, and the focus on cellular connectivity. Translated, it said in part:

“As part of its efforts to ensure the integrity of the examination process, and in coordination with relevant authorities, the Ministry of Education was able to uncover organized exam cheating networks in three examination centers in Lattakia Governorate. These networks used advanced electronic technologies and devices in their attempt to manipulate the exam process.

The network was seized in cooperation with the Lattakia Education Directorate, following close monitoring and detection of suspicious attempts. It was found that members of the network used small earphones, wireless communication devices, and mobile phones equipped with advanced transmission and reception technologies, which contradict educational values and violate the integrity of the examination process and the principle of justice.”

A slightly more unusual government directed shutdown took place in Venezuela on August 18 when Venezuelan provider SuperCable (AS22313) ceased service. An X post from Venezuelan industry watcher VE sin Filtro published a notification from CONATEL, the National Commission of Telecommunications in Venezuela, that notified SuperCable that as of March 14, 2025, its authority to operate in the country had been revoked, and established a 60 day transition period so that users could find another provider. Another X post from VE sin Filtro shared an email that SuperCable subscribers received from the company announcing the end of the service and, and noted that half an hour after the email was sent, subscribers were left without Internet connectivity. Traffic began to fall at 15:00 UTC (11:00 local time), and was gone after 15:30 UTC (11:30 local time). Connectivity remained shut down through the end of the quarter.

Interestingly, we did not see a corresponding full loss of announced IP address space when traffic disappeared. However, such full losses did occur between August 19-21, and again briefly on September 16. The number of announced /24s (blocks of 256 IPv4 addresses) fell from 95 to 63 on September 25, and remained at that level through the end of the quarter.

Similar to Syria, we covered the latest rounds of exam-related Internet shutdowns in Iraq in our second quarter blog post. In that post, we noted that the shutdowns in the main part of the country ran until July 3 for preparatory school exams, and through July 6 in the Kurdistan region. These can be seen in the graph below.

The Kurdistan Regional Government in Iraq ordered Internet services to be suspended on August 23 between 03:30 and 04:45 UTC (6:30-7:45 local time), and again every Saturday, Monday, and Wednesday until September 8 to prevent cheating on the second round of grade 12 exams. Similar to last quarter, KNET (AS206206), Newroz Telecom (AS21277), IQ Online (AS48492), and KorekTel (AS59625) were impacted by the ordered shutdowns.

In the main part of the country, starting on August 26, the latest round of Internet shutdowns for high school exams began, scheduled through September 13, taking place between 03:00-05:00 UTC (06:00-08:00 local time). Networks impacted by these shutdowns included Earthlink (AS199739), Asiacell (AS51684), Zainas (AS59588), Halasat (AS58322), and HulumTele (AS203214).

In mid-September, the Taliban ordered the shutdown of fiber optic Internet connectivity in multiple provinces across Afghanistan, as part of a drive to “prevent immorality”. It was the first such ban issued since the Taliban took full control of the country in August 2021. As many as 15 provinces experienced shutdowns, and these regional shutdowns blocked Afghani students from attending online classes, impacted commerce and banking, and limited access to government agencies and institutions such as passport and registration offices, customs offices.

Less than two weeks later, just after 11:30 UTC (16:00 local time) on Monday, September 29, 2025, subscribers of wired Internet providers in Afghanistan experienced a brief service interruption, lasting until just before 12:00 UTC (16:30 local time). Mobile providers Afghan Wireless (AS38472) and Etisalat (AS131284) remained available during that period. However, just after 12:30 UTC (17:00 local time), the Internet was completely shut down, taking the country completely offline.

These shutdowns are reviewed in more detail in our September 30 blog post, Nationwide Internet shutdown in Afghanistan extends localized disruptions. Connectivity was restored around 11:45 UTC (16:15 local time) on October 1.

On July 7, a post on X from Claro alerted subscribers to a service disruption caused by damage to two fiber optic cables. According to a subsequent post, one was damaged by work being done by CORAAVEGA (La Vega Water And Sewerage Corporation) and the other by work being done by the Dominican Electric Transmission Company. As a result of the damage, traffic from Claro (AS6400) began to drop just before 16:00 UTC (12:00 local time), falling just over two-thirds compared to the prior week. Claro’s technicians were able to quickly locate the faults and repair them, with traffic recovering around 18:00 UTC (14:00 local time).

Between 12:45-15:45 UTC (13:45-16:45 local time) on July 19, users in Angola experienced an Internet disruption, with Unitel Angola (AS37119) experiencing as much as a 95% drop in traffic as compared to the previous week, and Connectis (AS327932) suffering a complete outage. According to an X post from Unitel Angola, it “was caused by a disruption at our partner Angola Cables, resulting from public road works that affected the national fiber optic interconnections.”

However, the timing of the disruption coincided with protests over the rise in diesel fuel prices, and local non-governmental organizations disputed Unitel Angola’s explanation, claiming that it was actually due to a government-directed Internet shutdown. Multiple Angolan network providers experienced a drop in announced IP address space during the period the Internet disruption occurred, and analysis of routing information for these networks finds that they share Angola Cables (AS37468) as an upstream provider, lending some credence to the explanation from Unitel Angola.

Digicel Haiti (AS27653) is no stranger to Internet disruptions caused by damage to both terrestrial and submarine cables, experiencing such problems during the first and second quarters of 2025, as well as first, second, and third quarters of 2024. The most recent such disruption occurred on August 26, when they experienced two different cuts on their fiber optic infrastructure, according to an X post from the company’s Director General. Traffic dropped by approximately 80% during the disruption, which lasted from 19:30-23:00 UTC (15:30-19:00 UTC).

Telegeography’s Submarine Cable Map shows that the Red Sea has a high density of submarine cables that carry data between Europe, Africa, and Asia. Cuts to these cables can significantly impact connectivity, ranging from increased latency on international connections to complete outages. The impacts may only affect a single country, or they may disrupt multiple countries connected to a damaged cable. On September 6, Pakistan Telecom (AS17557) posted a message on X that stated “We would like to inform that submarine cable cuts have occurred in Saudi waters near Jeddah, impacting partial bandwidth capacity on SMW4 and IMEWE systems. As a result, internet users in Pakistan may experience some service degradation during peak hours.” (Initial reporting that the cable cuts occurred near Jeddah were apparently incorrect, as the damage occurred in Yemeni waters.)

Looking at the impact in Pakistan, we observed traffic drop by 25-30% in Sindh and Punjab between 12:00-20:00 UTC (17:00 - 01:00 local time).

In the United Arab Emirates, Etisalat alerted customers via a post on X that they “may experience slowness in data services due to an interruption in the international submarine cables.” Between 11:00-22:00 UTC (15:00-02:00 local time) on September 6, traffic from AS8966 (Etisalat) dropped as much as 28%.

Also in the UAE, service provider du (AS15802) told their customers via a post on X that “You may experience some slowness in our data services due to an International submarine cable cut.” This slowness is visible in Radar’s Internet quality metrics for the network between 11:00-22:00 UTC (15:00-02:00 local time) on September 6, with median bandwidth dropping by more than half, from 25 Mbps to as low as 9.8 Mbps, and median latency doubling from 30 ms to over 60 ms.

The graphs below provide another view of the impact of the cable cuts, based on Cloudflare network probes between New Delhi (del-c) to London (lhr-a) and Bombay (bom-c) to Frankfurt (fra-a). For the former pair of data centers, mean latency grew by approximately 20%, and for the latter pair, by approximately 30%, starting around 23:00 UTC on September 5. (The stable latency line at the bottom of both graphs represents probes going over the Cloudflare backbone, which was not impacted by the cable cuts.)

Fiber optic cables are frequently damaged by errant ship anchors (submarine) or construction equipment (terrestrial), but on September 26, a stray bullet damaged a cable in the Dallas, Texas area, disrupting Internet connectivity for Spectrum (AS11427) customers. Spectrum acknowledged the service interruption in a post on X, followed by another post four and a half hours later stating that the issue had been resolved. Although neither post cited the bullet as the cause of the disruption, news reports attributed the claim to a Spectrum spokesperson. Overall, the disruption was fairly nominal, lasting for just two hours between 18:00-20:00 UTC (13:00-15:00 local time), with traffic dropping less than 25% as compared to the prior week.

“Major cable breaks” disrupted Internet connectivity for customers of Telkom (AS37457) in South Africa on September 27. Although Telkom acknowledged the initial service disruption and its subsequent resolution in posts on X, it didn’t provide any information about the cause in these posts. However, it apparently later issued a statement, stating “Telkom confirms that mobile voice and data services, which were disrupted earlier on Saturday due to major cable breaks, have now been fully restored nationwide.” The disruption lasted six hours, from 08:00-14:00 UTC (10:00-16:00 local time), with traffic dropping as much as 50% as compared to the previous week.

A reported power outage at one of Airtel Tanzania's data centers on July 1 resulted in a multi-hour disruption in connectivity for its mobile customers. The service interruption occurred between 11:30-18:00 UTC (14:30-21:00 local time), with traffic dropping on Airtel Tanzania (AS37133) by as much as 40% as compared to the previous week.

According to the Industry and Trade Ministry in the Czech Republic, a fallen power cable caused a widespread power outage on July 4. This power outage impacted Internet connectivity within the country, with traffic dropping by as much as 32%. Traffic fell just after the power outage began at 10:00 UTC (12:00 local time), and although it was “nearly fully resolved” by 16:00 UTC (18:00 local time), traffic did not return to expected levels until closer to 20:00 UTC (22:00 local time). This trailing traffic recovery aligns with a published report that noted “While ČEPS, the national transmission system operator, restored full grid functionality by mid-afternoon, tens of thousands remained without electricity into the evening.”

On St. Vincent and the Grenadines, the St Vincent Electricity Services Limited (VINLEC) stated in a Facebook post that a “system failure” caused a power outage that affected customers on mainland St. Vincent. According to VINLEC, the system failed at approximately 11:30 local time on August 16 (03:30 UTC on August 17), and power was restored to all customers just after 04:00 local time on August 17 (08:00 UTC). During the four-hour power outage, which also disrupted Internet connectivity, traffic dropped by as much as 80% below expected levels.

In Curaçao, a series of Facebook posts from Aqualectra, the island’s water and power company, confirmed that there was a power outage, and provided updates on the progress towards restoration. The impact of the power outage to Internet connectivity was visible in traffic disruptions across several Internet service providers, including Flow (AS52233) and UTS (AS11081). The observed disruptions lasted for most of the day, with traffic dropping around 06:45 UTC (02:45 local time) and recovering to expected levels around 23:45 UTC (19:45 local time). During the disruption, the country's traffic dropped by over 80% as compared to the previous week, with Flow experiencing a near complete outage.

Wide-scale power outages occur all too frequently in Cuba, and when power is lost, Internet connectivity follows. We have covered many such events in this series of blog posts over the last several years, and the latest occurred on September 10. That morning, an X post from the Unión Eléctrica de Cuba reported the collapse of the national electric power system at 09:14 local time (13:14 UTC) following the unexpected shutdown of the Antonio Guiteras Thermoelectric Power Plant (CTE). The island’s Internet traffic dropped by nearly 60% (as compared to expected levels) almost immediately, and remained lower than normal for over a day, returning to expected levels around 17:15 UTC on September 11 (13:15 local time) when the Ministerio de Energía y Minas de Cuba posted on X that the national electric system had been restored.

A contractor cutting through three high voltage cables caused a nationwide power outage in Gibraltar on September 16, according to a Facebook post from the Gibraltar government. This power outage resulted in a disruption to Internet traffic between 11:15-18:30 UTC (13:15-20:30 local time), falling as low as 80% below the previous week.

A magnitude 8.8 earthquake struck the Kamchatka Peninsula in Russia at 23:24 UTC on July 29 (11:24 local time on July 30), and was powerful enough to trigger tsunami warnings for Japan, Alaska, Hawaii, Guam, and other Russian regions. The graphs below show that there was an immediate impact to Internet traffic across several networks in the region, including Rostelecom (AS12389) and InterkamService (AS42742), where traffic dropped by 75% or more. While traffic started to recover almost immediately across both providers, traffic on Rostelecom approached expected levels much more quickly than on InterkamService.

A cyberattack targeting Houthi-controlled YemenNet (AS30873) on August 11 briefly disrupted connectivity across the network in Yemen. A significant drop in traffic occurred at around 14:15 UTC (17:15 local time), recovering by 15:00 UTC (18:00 local time). This observed drop in traffic aligns with the reported timing and duration of the attack, which was focused on YemenNet’s ADSL infrastructure.

The attack also apparently impacted YemenNet’s routing, as announced IPv4 address space began to decline as the attack commenced. Although the attack ended within an hour after it started, announced address space remained depressed for approximately an additional hour, reaching as low as 510 /24s (blocks of 256 IPv4 addresses) being announced, down from a “steady state” of 870 /24s.

A fire at the Ramses Central Exchange in Cairo, Egypt on July 7 disrupted telecommunications services for a number of providers with infrastructure in the facility. The fire broke out in a Telecom Egypt equipment room, and impacted connectivity across multiple providers, including Etisalat (AS36992), Mobinil (AS37069), Orange Egypt (AS24863), and Vodafone Egypt (AS24835). Internet traffic across these providers initially dropped at 14:30 UTC (17:30 local time). Recovery to expected levels varied across the providers, with Etisalat recovering by July 9, Vodafone and Mobinil by July 10, and Orange Egypt on July 11.

On July 10, Telecom Egypt announced that services affected by the fire had been restored, after operations were transferred to alternative exchanges.

Global satellite Internet service provider Starlink (AS14593) acknowledged a July 24 network outage through a post on X. The Vice President of Network Engineering at SpaceX explained, in a subsequent X post, that “The outage was due to failure of key internal software services that operate the core network.”

Traffic initially dropped around 19:15 UTC, and the disruption lasted approximately 2.5 hours. The impact of the Starlink outage was particularly noticeable in countries including Yemen and Sudan, where traffic dropped by approximately 50%, as well as in Zimbabwe, South Sudan, and Chad.

At around 16:30 UTC on August 19 (00:30 local time on August 20), we observed an anomalous 25% drop in China’s Internet traffic. Our analysis of related metrics found that this disruption caused a drop in the share of IPv4 traffic, as well as a spike in the share of HTTP traffic (meaning that HTTPS traffic share had fallen), as shown in the graphs below.

Further analysis also found the share of TCP connections terminated in the Post SYN stage doubled during the observed outage, from 39% to 78%, as shown below. The cause of these unusual observations was ultimately uncovered by a Great Firewall Report blog post, which stated, in part: “Between approximately 00:34 and 01:48 (Beijing Time, UTC+8) on August 20, 2025, the Great Firewall of China (GFW) exhibited anomalous behavior by unconditionally injecting forged TCP RST+ACK packets to disrupt all connections on TCP port 443. This incident caused massive disruption of the Internet connections between China and the rest of the world. … The responsible device does not match the fingerprints of any known GFW devices, suggesting that the incident was caused by either a new GFW device or a known device operating in a novel or misconfigured state.” This explanation is consistent with the anomalies visible in the Radar graphs.

Subscribers of Nayatel (AS23674) experienced an approximately 90 minute disruption to Internet connectivity on September 24, due to a reported outage at an upstream provider. Traffic dropped as much as 57% between around 09:15-10:45 UTC (14:15-15:45 local). Transworld (AS38193) is one of several upstream providers to Nayatel, and a more significant drop in traffic is visible for that network, lasting from around 09:15-12:15 UTC (14:15-17:15 local time). The Nayatel disruption was likely less significant than the one seen at Transworld because Transworld is upstream of only a portion of the prefixes originated by Nayatel — traffic from other Nayatel prefixes was carried by other providers that remained available.

Several weeks after experiencing a full Internet shutdown, Iran again experienced a sudden drop in Internet traffic around 21:00 UTC on July 5 (00:30 local time on July 6), with traffic falling 80% as compared to the prior week. While most of the “unknown” disruptions covered in this series of posts are observed but have no associated acknowledgement or explanation, this disruption had multiple competing explanations.

A published report noted “IRNA, Iran’s official news agency, cited the state-run Telecommunications Infrastructure Company, reporting a national-level disruption in international connectivity that affected most internet service providers Saturday night. Yet government officials have not publicly addressed the cause.” However, posts from civil society groups that follow Internet connectivity in Iran (net4people, FilterWatch) suggested that the disruption was again due to an intentional shutdown. And a post thread on X referenced, and disputed, a claim that the disruption was due to a DDoS attack. Unfortunately, no definitive root cause for this disruption could be found.

Customers of Claro Colombia experienced an Internet disruption that lasted just over 30 minutes on August 6, with traffic falling two-thirds or more as compared to the prior week between 16:45 - 17:20 UTC. The disruption affected multiple ASNs owned by Claro, including AS10620, AS14080, and AS26611. (The Telmex Colombia and Comcel names shown in the graphs below are historical – Telmex and Comcel merged in 2012 and have operated under the Claro brand since then.) Claro did not acknowledge the disruption on social media, nor did it provide any explanation for it.

A near-complete outage at Pakistani backbone provider PTCL (AS17557) caused traffic from the network provider to drop 90% at 16:10 UTC (21:10 local time) on August 19. PTCL acknowledged the issue in a post on X, noting “We are currently facing data connectivity challenges on our PTCL and Ufone services.” Although they published a subsequent post several hours later after service was restored, they did not provide any additional information about the cause of the outage. However, one published report claimed “The disruption was primarily caused by a technical fault in PTCL’s fiber optic infrastructure.” while another report claimed “According to industry sources, the internet disruption in Pakistan may be connected to a technical fault in the fiber optic backbone or issues with main internet providers responsible for international online traffic.

Interestingly, traffic from PTCL to Cloudflare’s 1.1.1.1 DNS resolver spiked as the outage began, and the share of requests made over UDP grew from 94% to 99%. In addition, routing data shows that there was also a small drop in announced IPv4 address space coincident with the outage. However, these additional observations do not necessarily confirm a “technical fault in PTCL’s fiber optic infrastructure” as the ultimate cause of the disruption.

To their credit, South African provider RSAWEB (AS37053) quickly acknowledged an issue with their FTTx and Enterprise connectivity on September 10, but neither their initial post nor subsequent updates provided any information on the cause of the problem. Whatever the cause, it resulted in a near-complete loss of Internet traffic from RSAWEB between 15:00 and 16:30 UTC (17:00 - 18:30 local time).

After experiencing a brief disruption in July due to a software failure, Starlink (AS14593) suffered another short disruption between 04:00-05:00 UTC on September 15. Although Starlink generally acknowledges disruptions to their global network on their X account, and often providing a root cause, in this case they apparently published an acknowledgement on X, but deleted it after the issue was resolved. In addition to the drop in traffic, we observed a concurrent drop in announced IPv4 address space and spike in BGP announcements (likely withdrawals), suggesting that the disruption may have been caused by a network-related issue.

The recent launch of regional traffic insights on Radar brings yet another perspective to our ability to investigate observed Internet traffic anomalies. We can now drill down at regional and network levels, as well as exploring the impact across DNS traffic, connection bandwidth and latency, TCP connection tampering, and announced IP address space, helping us understand the impact of such events. And while these blog posts feature graphs from Radar and the Radar Data Explorer, the underlying data is available from our rich API. You can use the API to retrieve data to do your own local monitoring or analysis, or the Radar MCP server to incorporate Radar data into your AI tools.

The Cloudflare Radar team is constantly monitoring for Internet disruptions, sharing our observations on the Cloudflare Radar Outage Center, via social media, and in posts on blog.cloudflare.com. Follow us on social media at @CloudflareRadar (X), noc.social/@cloudflareradar (Mastodon), and radar.cloudflare.com (Bluesky), or contact us via email.

However, just after 12:30 UTC (17:00 local time), the Internet was completely shut down, with Afghani news outlet TOLOnews initially reporting in a post on X that “Sources have confirmed to TOLOnews that today (Monday), afternoon, fiber-optic Internet will be shut down across the country.” This shutdown is likely an extension of the regional shutdowns of fiber optic connections that took place earlier in September, and it will reportedly remain in force “until further notice”. (The earlier regional shutdowns are discussed in more detail below.)

While Monday’s first shutdown was only partial, with mobile connectivity apparently remaining available, the graphs below show that the second event took the country completely offline, with web and DNS traffic dropping to zero at a national level, as seen in the graphs below.

While the shutdown will impact subscribers to fixed and mobile Internet services, it also “threatens to paralyze critical services including banking, customs operations and emergency communications” across the country. The X post from TOLOnews also noted that television and radio networks would face disruptions.

HTTP request traffic is traffic coming from web browsers, applications, and automated tools, and is a clear signal of the availability of Internet connectivity. The graph below shows this request volume dropping sharply as the shutdown was implemented.

^{HTTP request traffic from Afghanistan, September 29, 2025}

Cloudflare sends bytes back in response to those HTTP requests (“HTTP bytes”), as well as sending bytes back in response to traffic associated with other services, such as our 1.1.1.1 DNS resolver, authoritative DNS, WARP, etc. (“total bytes”). Cloudflare stopped receiving client traffic from the services when the shutdown began, causing the bytes transferred in response to drop to zero.

^{Internet traffic from Afghanistan, September 29, 2025}

1.1.1.1 is Cloudflare’s privacy-focused DNS resolver, and processes DNS lookup requests from clients. As connectivity was cut, traffic to the service disappeared.

^{DNS query traffic to Cloudflare’s 1.1.1.1 resolver from Afghanistan, September 29, 2025}

At a regional level, it appears that traffic from Kabul fell slightly later than traffic from the other regions, trailing them by approximately a half hour.

^{HTTP request traffic from the top five provinces in Afghanistan, September 29, 2025}

The delay in traffic loss seen in Kabul may be associated with a more gradual loss of traffic seen at AS38742 (Afghan Wireless), which saw traffic approach zero just after 13:00 UTC (17:30 local time). This conjecture is supported by a published report that noted “Residents across Kabul and several provincial cities reported on Monday that fiber-optic services were no longer available, with only limited mobile data functioning briefly before signal towers stopped working altogether.”

Interestingly, it appears that as of 00:00 UTC (04:30 local time) on September 30, we continue to see a very small amount of traffic from this network. (This is in contrast to other networks, whose lines disappeared from the graph around 12:30 UTC (17:00 local time)).

^{HTTP request traffic from the top 10 ASNs in Afghanistan, September 29, 2025}

Network providers announce IP address space that they are responsible for to other networks, enabling the routing of traffic to and from those IP addresses. When these announcements are withdrawn, the resources in that address space, whether clients or servers, can no longer reach, or are no longer reachable from, the rest of the Internet.

In Afghanistan, announced IPv4 address space dropped rapidly as the shutdown was implemented, falling by two-thirds from 604 to 197 announced /24s (blocks of 256 IPv4 addresses) in the first 20 minutes, and then dropping further over the next 90 minutes. Through the end of the day, several networks continued to announce a small amount of IPv4 address space: four /24s from AS38742 (Afghan Wireless), two from AS149024 (Afghan Bawar ICT Services), and one each from AS138322 (Afghan Wireless) and AS136479 (Cyber Telecom).

Afghan Wireless is a mobile connectivity provider, and Afghan Bawar and Cyber Telecom appear to offer wireless/mobile services as well. The prefixes still visible from Afghan Wireless appear to be routed through AS17557 (Pakistan Telecom), while the prefixes from the other two providers (Afghan Bawar, Cyber Telecom) appear to be routed through AS40676 (Psychz Networks), a US-based solutions provider.

^{Announced IPv4 address space from Afghanistan, September 29, 2025}

Announced IPv6 address space fell as well, though not quite as catastrophically, dropping by three-fourths almost immediately, from 262,407 /48s (blocks of over 1.2 septillion IPv6 addresses) to 65,542.

^{Announced IPv6 address space from Afghanistan, September 29, 2025}

In mid-September, the Taliban ordered the shutdown of fiber optic Internet connectivity in multiple provinces across Afghanistan, as part of a drive to “prevent immorality”. It was the first such ban issued since the Taliban took full control of the country in August 2021.

These regional shutdowns blocked Afghani students from attending online classes, impacted commerce and banking, and limited access to government agencies and institutions such as passport and registration offices, customs offices. As many as 15 provinces experienced shutdowns, and we review the observed impacts across several of them below, using the regional traffic data recently made available on Cloudflare Radar.

Balkh appeared to be one of the earliest targeted provinces, with traffic dropping midday (UTC) on September 15. While some nominal recovery occurred on September 23, traffic remained well below pre-shutdown levels.

^{Internet traffic from Balkh, Afghanistan, September 1-28, 2025}

After several days of peak traffic levels double those seen in previous weeks, traffic in Takhar fell on September 16, remaining near zero until September 21, when a small amount of connectivity was apparently restored.

^{Internet traffic from Takhar, Afghanistan, September 1-28, 2025}

In Kandahar, lower peak traffic volumes are visible between September 17 and September 21. The partial restoration of traffic is coincident with the restoration of Internet services highlighted in a published report, though it notes that “The restoration of services is limited to point-to-point connections for key government offices, including banks, customs offices, and the Directorate for National ID Cards.”

^{Internet traffic from Kandahar, Afghanistan, September 1-28, 2025}

Baghlan experienced an anomalous spike in traffic on September 16, with total traffic spiking 3x higher than peaks seen during the previous weeks. However, on September 17, traffic dropped to a fraction of pre-shutdown levels. Except for a return to near-normal levels on September 21 & 22, the disruption remained in place through the end of the month.

^{Internet traffic from Baghlan, Afghanistan, September 1-28, 2025}

Traffic in Nangarhar was disrupted between September 19-22, but quickly recovered to pre-shutdown levels once restored.

^{Internet traffic from Nangarhar, Afghanistan, September 1-28, 2025}

After experiencing an apparent issue at the start of the month, Internet traffic in Oruzgan, again fell on September 19. After an apparent complete shutdown, on September 23, a small amount of traffic was again visible.

^{Internet traffic from Oruzgan, Afghanistan, September 1-28, 2025}

Internet connectivity was also disrupted in the province of Herat, although differently. From September 22-25, partial Internet outages were implemented between 16:30-03:30 UTC (21:00-08:00 local time), with traffic volumes dropping to approximately half of those seen at the same time the prior weeks. The intent of these “Internet curfew” shutdowns is unclear, but Herat residents noted that they “severely disrupted their business and educational activities”.

^{Internet traffic from Herat, Afghanistan, September 16-29, 2025}

While Internet shutdowns remain all too common around the world, most (though not all) are comparatively short-lived, and are generally in response to a local event, such as exams, unrest/riots, elections, etc. Given the broad impact of this shutdown across all facets of daily personal, social, and professional life in Afghanistan, analysts state that it "could deepen Afghanistan’s digital isolation, further damage its struggling economy and drive more Afghans out of work at a time when humanitarian needs are already severe."

You can follow the latest state of Internet connectivity in Afghanistan on Cloudflare Radar. The Cloudflare Radar team will continue to monitor traffic from Afghanistan as well, sharing our observations on the Cloudflare Radar Outage Center, via social media, and in posts on blog.cloudflare.com. Follow us on social media at @CloudflareRadar (X), noc.social/@cloudflareradar (Mastodon), and radar.cloudflare.com (Bluesky), or contact us via email.

We had an outage in our Tenant Service API which led to a broad outage of many of our APIs and the Cloudflare Dashboard. 

The incident’s impact stemmed from several issues, but the immediate trigger was a bug in the dashboard. This bug caused repeated, unnecessary calls to the Tenant Service API. The API calls were managed by a React useEffect hook, but we mistakenly included a problematic object in its dependency array. Because this object was recreated on every state or prop change, React treated it as “always new,” causing the useEffect to re-run each time. As a result, the API call executed many times during a single dashboard render instead of just once. This behavior coincided with a service update to the Tenant Service API, compounding instability and ultimately overwhelming the service, which then failed to recover.

When the Tenant Service became overloaded, it had an impact on other APIs and the dashboard because Tenant Service is part of our API request authorization logic.  Without Tenant Service, API request authorization can not be evaluated.  When authorization evaluation fails, API requests return 5xx status codes.

We’re very sorry about the disruption.  The rest of this blog goes into depth on what happened, and what steps we are taking to prevent it from happening again.

The Cloudflare dashboard was severely impacted throughout the full duration of the incident.

The Cloudflare API was severely impacted for two periods during the incident when the Tenant API Service was down.

Our first goal in an incident is to restore service.  Often that involves fixing the underlying issue directly, but not always.  In this case we noticed increased usage across our Tenant Service, so we focused on reducing the load and increasing the available resources.  We installed a global rate limit on the Tenant Service to help regulate the load.  The Tenant Service is a GoLang process that runs on Kubernetes in a subset of our datacenters.  We increased the number of pods available as well to help improve throughput.  While we did this, we had others on the team continue to investigate why we were seeing the unusually high usage.  Ultimately, increasing the resources available to the tenant service helped with availability but was insufficient to restore normal service.

After the Tenant Service began reporting healthy again and the API largely recovered, we still observed a considerable number of errors being reported from the service. We theorized that these were responsible for the ongoing Dashboard availability issues and made a patch to the service with the expectation that it would improve the API health and restore the dashboard to a healthy state. Ultimately this change degraded service further and was quickly reverted. The second outage can be seen in the graph above.

It’s painful to have an outage like this.  That said, there were a few things that helped lessen the impact.  Our automatic alerting service quickly identified the correct people to join the call and start working on remediation.  Additionally, this was a failure in the control plane which has strict separation of concerns from the data plane.  Thus, the outage did not affect services on Cloudflare’s network.  The majority of users at Cloudflare were unaffected unless they were making configuration changes or using our dashboard.

We believe it’s important to learn from our mistakes and this incident is an opportunity to make some improvements.  Those improvements can be categorized as either ways to reduce / eliminate the impact of a similar change or as improvements to our observability tooling to better inform the team during future events.

We use Argo Rollouts for releasing, which monitors deployments for errors and automatically rolls back that service on a detected error.  We’ve been migrating our services over to Argo Rollouts but have not yet updated the Tenant Service to use it.  Had it been in place, we would have automatically rolled back the second Tenant Service update limiting the second outage.  This work had already been scheduled by the team and we’ve increased the priority of the migration.

When we restarted the Tenant Service, everyone’s dashboard began to re-authenticate with the API.  This caused the API to become unstable again causing issues with everyone’s dashboard.  This pattern is a common one often referred to as a Thundering Herd.  Once a resource or service is made available, everyone tries to use it all at once. This is common, but was amplified by the bug in our dashboard logic. The fix for this behavior has already been released via a hotfix shortly after the impact was over.  We’ll be introducing changes to the dashboard that include random delays to spread out retries and reduce contention as well. 

Finally, the Tenant Service was not allocated sufficient capacity to handle spikes in load like this. We’ve allocated substantially more resources to this service, and are improving the monitoring so that we will be proactively alerted before this service hits capacity limits.

We immediately saw an increase in our API usage but found it difficult to identify which requests were retries vs new requests.  Had we known that we were seeing a sustained large volume of new requests, it would have made it easier to identify the issue as a loop in the dashboard.  We are adding changes to how we call our APIs from our dashboard to include additional information, including if the request is a retry or new request.

We’re very sorry about the disruption.  We will continue to investigate this issue and make improvements to our systems and processes.

Customers with origins in AWS us-east-1 began experiencing impact at 16:27 UTC. The impact was substantially reduced by 19:38 UTC, with intermittent latency increases continuing until 20:18 UTC.

This was a regional problem between Cloudflare and AWS us-east-1, and global Cloudflare services were not affected. The degradation in performance was limited to traffic between Cloudflare and AWS us-east-1. The incident was a result of a surge of traffic from a single customer that overloaded Cloudflare's links with AWS us-east-1. It was a network congestion event, not an attack or a BGP hijack.

We’re very sorry for this incident. In this post, we explain what the failure was, why it occurred, and what we’re doing to make sure this doesn’t happen again.

Cloudflare helps anyone to build, connect, protect, and accelerate their websites on the Internet. Most customers host their websites on origin servers that Cloudflare does not operate. To make their sites fast and secure, they put Cloudflare in front as a reverse proxy. 

When a visitor requests a page, Cloudflare will first inspect the request. If the content is already cached on Cloudflare’s global network, or if the customer has configured Cloudflare to serve the content directly, Cloudflare will respond immediately, delivering the content without contacting the origin. If the content cannot be served from cache, we fetch it from the origin, serve it to the visitor, and cache it along the way (if it is eligible). The next time someone requests that same content, we can serve it directly from cache instead of making another round trip to the origin server. 

When Cloudflare responds to a request with the cached content, it will send the response traffic over internal Data Center Interconnect (DCI) links through a series of network equipment and eventually reach the routers that represent our network edge (our “edge routers”) as shown below:

Our internal network capacity is designed to be larger than the available traffic demand in a location to account for failures of redundant links, failover from other locations, traffic engineering within or between networks, or even traffic surges from users. The majority of Cloudflare’s network links were operating normally, but some edge router links to an AWS peering switch had insufficient capacity to handle this particular surge. 

At approximately 16:27 UTC on August, 21, 2025, a customer started sending many requests from AWS us-east-1 to Cloudflare for objects in Cloudflare’s cache. These requests generated a volume of response traffic that saturated all available direct peering connections between Cloudflare and AWS. This initial saturation became worse when AWS, in an effort to alleviate the congestion, withdrew some BGP advertisements to Cloudflare over some of the congested links. This action rerouted traffic to an additional set of peering links connected to Cloudflare via an offsite network interconnection switch, which subsequently also became saturated, leading to significant performance degradation. The impact became worse for two reasons: One of the direct peering links was operating at half-capacity due to a pre-existing failure, and the Data Center Interconnect (DCI) that connected Cloudflare’s edge routers to the offsite switch was due for a capacity upgrade. The diagram below illustrates this using approximate capacity estimates:

In response, our incident team immediately engaged with our partners at AWS to address the issue. Through close collaboration, we successfully alleviated the congestion and fully restored services for all affected customers.

When impact started, we saw a significant amount of traffic related to one customer, resulting in congestion:

This was handled by manual traffic actions both from Cloudflare and AWS. You can see some of the attempts by AWS to alleviate the congestion by looking at the number of IP prefixes AWS is advertising to Cloudflare during the duration of the outage. The lines in different colors correspond to the number of prefixes advertised per BGP session with us. The dips indicate AWS attempting to mitigate by withdrawing prefixes from the BGP sessions in an attempt to steer traffic elsewhere:

The congestion in the network caused network queues on the routers to grow significantly and begin dropping packets. Our edge routers were dropping high priority packets consistently during the outage, as seen in the chart below, which shows the queue drops for our Ashburn routers during the impact period:

The primary impact to customers as a result of this congestion would have been latency, loss (timeouts), or low throughput. We have a set of latency Service Level Objectives defined which imitate customer requests back to their origins measuring availability and latency. We can see that during the impact period, the percentage of requests whose latency fails to meet the target SLO threshold dips below an acceptable level in lock step with the packet drops during the outage:

After the congestion was alleviated, there was a brief period where both AWS and Cloudflare were attempting to normalize the prefix advertisements that had been adjusted to attempt to mitigate the congestion. That caused a long tail of latency that may have impacted some customers, which is why you see the packet drops resolve before the customer latencies are restored.

This event has underscored the need for enhanced safeguards to ensure that one customer's usage patterns cannot negatively affect the broader ecosystem. Our key takeaways are the necessity of architecting for better customer isolation to prevent any single entity from monopolizing shared resources and impacting the stability of the platform for others, and augmenting our network infrastructure to have sufficient capacity to meet demand. 

To prevent a recurrence of this issue, we are implementing a multi-phased mitigation strategy. In the short and medium term: 

• We are developing a mechanism to selectively deprioritize a customer’s traffic if it begins to congest the network to a degree that impacts others.

• We are expediting the Data Center Interconnect (DCI) upgrades which will provide network capacity significantly above what it is today.

• We are working with AWS to make sure their and our BGP traffic engineering actions do not conflict with one another in the future.

Looking further ahead, our long-term solution involves building a new, enhanced traffic management system. This system will allot network resources on a per-customer basis, creating a budget that, once exceeded, will prevent a customer's traffic from degrading the service for anyone else on the platform. This system will also allow us to automate many of the manual actions that were taken to attempt to remediate the congestion seen during this incident.

Customers accessing AWS us-east-1 through Cloudflare experienced an outage due to insufficient network congestion management during an unusual high-traffic event.

We are sorry for the disruption this incident caused for our customers. We are actively making these improvements to ensure improved stability moving forward and to prevent this problem from happening again.

Cloudflare's 1.1.1.1 Resolver service became unavailable to the Internet starting at 21:52 UTC and ending at 22:54 UTC. The majority of 1.1.1.1 users globally were affected. For many users, not being able to resolve names using the 1.1.1.1 Resolver meant that basically all Internet services were unavailable. This outage can be observed on Cloudflare Radar.

The outage occurred because of a misconfiguration of legacy systems used to maintain the infrastructure that advertises Cloudflare’s IP addresses to the Internet.

This was a global outage. During the outage, Cloudflare's 1.1.1.1 Resolver was unavailable worldwide.

We’re very sorry for this outage. The root cause was an internal configuration error and not the result of an attack or a BGP hijack. In this blog, we’re going to talk about what the failure was, why it occurred, and what we’re doing to make sure this doesn’t happen again.

Cloudflare introduced the 1.1.1.1 public DNS Resolver service in 2018. Since the announcement, 1.1.1.1 has become one of the most popular DNS Resolver IP addresses and it is free for anyone to use.

Almost all of Cloudflare's services are made available to the Internet using a routing method known as anycast, a well-known technique intended to allow traffic for popular services to be served in many different locations across the Internet, increasing capacity and performance. This is the best way to ensure we can globally manage our traffic, but also means that problems with the advertisement of this address space can result in a global outage.   

Cloudflare announces these anycast routes to the Internet in order for traffic to those addresses to be delivered to a Cloudflare data center, providing services from many different places. Most Cloudflare services are provided globally, like the 1.1.1.1 public DNS Resolver, but a subset of services are specifically constrained to particular regions. 

These services are part of our Data Localization Suite (DLS), which allows customers to configure Cloudflare in a variety of ways to meet their compliance needs across different countries and regions. One of the ways in which Cloudflare manages these different requirements is to make sure the right service's IP addresses are Internet-reachable only where they need to be, so your traffic is handled correctly worldwide. A particular service has a matching "service topology" – that is, traffic for a service should be routed only to a particular set of locations.

On June 6, during a release to prepare a service topology for a future DLS service, a configuration error was introduced: the prefixes associated with the 1.1.1.1 Resolver service were inadvertently included alongside the prefixes that were intended for the new DLS service. This configuration error sat dormant in the production network as the new DLS service was not yet in use,  but it set the stage for the outage on July 14. Since there was no immediate change to the production network there was no end-user impact, and because there was no impact, no alerts were fired.

Any traffic coming to Cloudflare via 1.1.1.1 Resolver services on these IPs was impacted. Traffic to each of these addresses were also impacted on the corresponding routes. 

1.1.1.0/24
1.0.0.0/24 
2606:4700:4700::/48
162.159.36.0/24
162.159.46.0/24
172.64.36.0/24
172.64.37.0/24
172.64.100.0/24
172.64.101.0/24
2606:4700:4700::/48
2606:54c1:13::/48
2a06:98c1:54::/48

When the impact started we observed an immediate and significant drop in queries over UDP, TCP and DNS over TLS (DoT). Most users have 1.1.1.1, 1.0.0.1, 2606:4700:4700::1111, or 2606:4700:4700::1001 configured as their DNS server. Below you can see the query rate for each of the individual protocols and how they were impacted during the incident:

It’s worth noting that DoH (DNS-over-HTTPS) traffic remained relatively stable as most DoH users use the domain cloudflare-dns.com, configured manually or through their browser, to access the public DNS resolver, rather than by IP address. DoH remained available and traffic was mostly unaffected as cloudflare-dns.com uses a different set of IP addresses. Some DNS traffic over UDP that also used different IP addresses remained mostly unaffected as well.

As the corresponding prefixes were withdrawn, no traffic sent to those addresses could reach Cloudflare. We can see this in the timeline for the BGP announcements for 1.1.1.0/24:

^{Pictured above is the timeline for BGP withdrawal and re-announcement of 1.1.1.0/24 globally}

When looking at the query rate of the withdrawn IPs it can be observed that almost no traffic arrives during the impact window. When the initial fix was applied at 22:20 UTC, a large spike in traffic can be seen before it drops off again. This spike is due to clients retrying their queries. When we started announcing the withdrawn prefixes again, queries were able to reach Cloudflare once more. It took until 22:54 UTC before routing was restored in all locations and traffic returned to mostly normal levels.

As described above, a configuration change on June 6 introduced an error in the service topology for a pre-production, DLS service. On July 14, a second change to that service was made: an offline data center location was added to the service topology for the pre-production DNS service in order to allow for some internal testing. This change triggered a refresh of the global configuration of the associated routes, and it was at this point that the impact from the earlier configuration error was felt. The service topology for the 1.1.1.1 Resolver's prefixes was reduced from all locations down to a single, offline location. The effect was to trigger the global and immediate withdrawal of all 1.1.1.1 prefixes.

As routes to 1.1.1.1 were withdrawn, the 1.1.1.1 service itself became unavailable. Alerts fired and an incident was declared.

The way that Cloudflare manages service topologies has been refined over time and currently consist of a combination of a legacy and a strategic system that are synced. Cloudflare's IP ranges are currently bound and configured across these systems that  dictate where an IP range should be announced (in terms of datacenter location) on the edge network. The legacy approach of hard-coding explicit lists of data center locations and attaching them to particular prefixes has proved error-prone, since (for example) bringing a new data center online requires many different lists to be updated and synced consistently. This model also has a significant flaw in that updates to the configuration do not follow a progressive deployment methodology: Even though this release was peer-reviewed by multiple engineers, the change didn’t go through a series of canary deployments before reaching every Cloudflare data center. Our newer approach is to describe service topologies without needing to hard-code IP addresses, which better accommodate expansions to new locations and customer scenarios while also allowing for a staged deployment model, so changes can propagate slowly with health monitoring. During the migration between these approaches, we need to maintain both systems and synchronize data between them, which looks like this:

Initial alerts were triggered for the DNS Resolver at 22:01, indicating query, proxy, and data center failures. While investigating the alerts, we noted traffic toward the Resolver prefixes had drastically dropped and was no longer being received at our edge data centers. Internally, we use BGP to control route advertisements, and we found the Resolver routes from servers were completely missing.

Once our configuration error had been exposed and Cloudflare systems had withdrawn the routes from our routing table, all of the 1.1.1.1 routes should have disappeared entirely from the global Internet routing table. However, this isn’t what happened with the prefix 1.1.1.0/24. Instead, we got reports from Cloudflare Radar that Tata Communications India (AS4755) had started advertising 1.1.1.0/24: from the perspective of the routing system, this looked exactly like a prefix hijack. This was unexpected to see while we were troubleshooting the routing problem, but to be perfectly clear: this BGP hijack was not the cause of the outage. We are following up with Tata Communications.

We reverted to the previous configuration at 22:20 UTC. Near instantly, we began readvertising the BGP prefixes which were previously withdrawn from the routers, including 1.1.1.0/24. This restored 1.1.1.1 traffic levels to roughly 77% of what they were prior to the incident. However, during the period since withdrawal, approximately 23% of the fleet of edge servers had been automatically reconfigured to remove required IP bindings as a result of the topology change. To add the configurations back, these servers needed to be reconfigured with our change management system which is not an instantaneous process by default for safety. 

The process by which the IP bindings can be restored normally takes some time, as the network in individual locations is designed to be updated over a course of multiple hours. We implement a progressive rollout, rather than on all nodes at once to ensure we don’t introduce additional impact. However, given the severity of the incident, we accelerated the rollout of the fix after verifying the changes in testing locations to restore service as quickly and safely as possible. Normal traffic levels were observed at 22:54 UTC.

We take incidents like this seriously, and we recognise the impact that this incident had. Though this specific issue has been resolved, we have identified several steps we can take to mitigate the risk of a similar problem occurring in the future. We are implementing the following plan as a result of this incident:

Staging Addressing Deployments: Legacy components do not leverage a gradual, staged deployment methodology. Cloudflare will deprecate these systems which enables modern progressive and health mediated deployment processes to provide earlier indication in a staged manner and rollback accordingly.

Deprecating Legacy Systems: We are currently in an intermediate state in which current and legacy components need to be updated concurrently, so we will be migrating addressing systems away from risky deployment methodologies like this one. We will accelerate our deprecation of the legacy systems in order to provide higher standards for documentation and test coverage.

Cloudflare's 1.1.1.1 DNS Resolver service fell victim to an internal configuration error.

We are sorry for the disruption this incident caused for our customers. We are actively making these improvements to ensure improved stability moving forward and to prevent this problem from happening again.

This outage lasted 2 hours and 28 minutes, and globally impacted all Cloudflare customers using the affected services. The cause of this outage was due to a failure in the underlying storage infrastructure used by our Workers KV service, which is a critical dependency for many Cloudflare products and relied upon for configuration, authentication and asset delivery across the affected services. Part of this infrastructure is backed by a third-party cloud provider, which experienced an outage today and directly impacted availability of our KV service.

We’re deeply sorry for this outage: this was a failure on our part, and while the proximate cause (or trigger) for this outage was a third-party vendor failure, we are ultimately responsible for our chosen dependencies and how we choose to architect around them.

This was not the result of an attack or other security event. No data was lost as a result of this incident. Cloudflare Magic Transit and Magic WAN, DNS, Cache, proxy, WAF and related services were not directly impacted by this incident.

As a rule, Cloudflare designs and builds our services on our own platform building blocks, and as such many of Cloudflare’s products are built to rely on the Workers KV service. 

The following table details the impacted services, including the user-facing impact, operation failures, and increases in error rates observed:

Workers KV is built as what we call a “coreless” service which means there should be no single point of failure as the service runs independently in each of our locations worldwide. However, Workers KV today relies on a central data store to provide a source of truth for data. A failure of that store caused a complete outage for cold reads and writes to the KV namespaces used by services across Cloudflare.

Workers KV is in the process of being transitioned to significantly more resilient infrastructure for its central store: regrettably, we had a gap in coverage which was exposed during this incident. Workers KV removed a storage provider as we worked to re-architect KV’s backend, including migrating it to Cloudflare R2, to prevent data consistency issues (caused by the original data syncing architecture), and to improve support for data residency requirements.

One of our principles is to build Cloudflare services on our own platform as much as possible, and Workers KV is no exception. Many of our internal and external services rely heavily on Workers KV, which under normal circumstances helps us deliver the most robust services possible, instead of service teams attempting to build their own storage services. In this case, the cascading impact from the failure from Workers KV exacerbated the issue and significantly broadened the blast radius. 

The incident timeline, including the initial impact, investigation, root cause, and remediation, are detailed below. 

_{Workers KV error rates to storage infrastructure. 91% of requests to KV failed during the incident window.}

_{Cloudflare Access percentage of successful requests. Cloudflare Access relies directly on Workers KV and serves as a good proxy to measure Workers KV availability over time.}

All timestamps referenced are in Coordinated Universal Time (UTC).

We’re taking immediate steps to improve the resiliency of services that depend on Workers KV and our storage infrastructure. This includes existing planned work that we are accelerating as a result of this incident.

This encompasses several workstreams, including efforts to avoid singular dependencies on storage infrastructure we do not own, improving the ability for us to recover critical services (including Access, Gateway and WARP) 

Specifically:

• (Actively in-flight): Bringing forward our work to improve the redundancy within Workers KV’s storage infrastructure, removing the dependency on any single provider. During the incident window we began work to cut over and backfill critical KV namespaces to our own infrastructure, in the event the incident continued.

• (Actively in-flight): Short-term blast radius remediations for individual products that were impacted by this incident so that each product becomes resilient to any loss of service caused by any single point of failure, including third party dependencies.

• (Actively in-flight): Implementing tooling that allows us to progressively re-enable namespaces during storage infrastructure incidents. This will allow us to ensure that key dependencies, including Access and WARP, are able to come up without risking a denial-of-service against our own infrastructure as caches are repopulated.

This list is not exhaustive: our teams continue to revisit design decisions and assess the infrastructure changes we need to make in both the near (immediate) term and long term to mitigate the incidents like this going forward.

This was a serious outage, and we understand that organizations and institutions that are large and small depend on us to protect and/or run their websites, applications, zero trust and network infrastructure.  Again we are deeply sorry for the impact and are working diligently to improve our service resiliency. 

The breadth of Cloudflare’s network and our customer base provides us with a unique perspective on Internet resilience, enabling us to observe the Internet impact of this power outage at both a local and national level, as well as at a network level, across traffic, network quality, and routing metrics.

In Portugal, Internet traffic dropped as the power grid failed, with traffic immediately dropping by half as compared to the previous week, falling to approximately 90% below the previous week within the next five hours.

Request traffic from users in Portugal to Cloudflare’s 1.1.1.1 DNS resolver also fell when the power went out, initially dropping by 40% as compared to the previous week, and falling further over the next several hours. 

At a network level, the loss of Internet traffic from local providers including NOS, Vodafone, MEO, and NOWO was swift and significant. The Cloudflare Radar graphs below show that traffic from those networks effectively evaporated over the hours after the power outage began. The autonomous systems (ASNs) shown below for these providers may carry a mix of fixed and mobile broadband traffic. However, MEO breaks out at least some of their mobile traffic onto a separate ASN, and the graph below for MEO-MOVEL (AS42863) shows that request traffic from that network more than doubled after the power went out, as subscribers turned to their mobile devices for information about what was happening. However, despite the initial spike, this mobile traffic also fell over the next several hours, dropping to approximately half of the volume seen the prior week.

In addition to looking at traffic at a national and network level, we can also look at traffic at a regional level. As noted above, the power outage did not impact every region of the country. The traffic graphs below show the changes in Internet traffic from the parts of Portugal where an impact was observed.

In Lisbon and Porto, a sharp, but limited drop in traffic was observed as the power outage began, with traffic recovering slightly almost as quickly. However, traffic gradually declined in the subsequent hours, in contrast to the other regions reviewed below.

The most significant immediate traffic drops were observed in Aveiro, Beja, Bragança, Castelo Branco, Évora, Faro, Guarda, Portalegre, Santarém, Viana do Castelo, Vila Real, and Viseu. In these areas, traffic fell and then quickly stabilized at very low volumes. In Braga and Setúbal, traffic declined more gradually after the initial drop.

The power outage also impacted the quality of connectivity at a national level in Portugal. Prior to the loss of power, median download speeds across the country were around 40 Mbps, but within several hours after the state of the outage, fell as low as 15 Mbps. As expected, latency at a country level saw an opposite impact. Prior to the loss of power, median latency was around 20 ms. However, it gradually grew to as much as 50 ms. The lower download speeds and higher latency are likely due to the congestion of the network links that remained available.

Network infrastructure in Portugal was also impacted by the power outage, with the impact seen as a drop in announced IP address space. (This means that portions of Portuguese providers’ networks are no longer visible to the rest of the Internet.) The number of announced IPv4 /24s (blocks of 256 IPv4 addresses) dropped by ~300 (around 1.2%), and the number of announced IPv6 /48s (blocks of over 1.2 octillion IPv6 addresses) dropped from 17,928,551 to 16,355,607 (around 9%). Address space began to drop further after 16:00 UTC, possibly as a result of backup power being exhausted and associated network infrastructure falling offline.

In Spain, Internet traffic dropped as the power grid failed, with traffic immediately dropping by around 60% as compared to the previous week, falling to approximately 80% below the previous week within the next five hours.

Request traffic from users in Spain to Cloudflare’s 1.1.1.1 DNS resolver also fell when the power went out, initially dropping by 54% as compared to the previous week, but quickly stabilizing. 

At a network level, traffic volumes from the top five ASNs in Spain fell rapidly once power was lost, with most declining gradually over the next several hours. In contrast, traffic from Digi Spain Telecom (AS57269) fell quickly, but then stabilized at the lower level. In comparison to the previous week, traffic from these providers fell between 75% and 93% in the hours after the power outage began.

In most of the impacted regions in Spain, traffic dropped off quickly and stabilized, or continued to fall further. However, some recovery in traffic is also evident, and can be seen in Navarre, La Rioja, Cantabria, and Basque Country. This traffic recovery is likely associated with an initial restoration of power in those regions, as an update from Red Eléctrica (operator of Spain’s national electricity grid) noted that “Electricity is now available in parts of Catalonia, Aragon, the Basque Country, Galicia, Asturias, Navarre, Castile and León, Extremadura, Andalusia, and La Rioja.”

The power outage also impacted the quality of connectivity at a national level in Spain. Prior to the loss of power, median download speeds across the country were around 35 Mbps, but within several hours after the state of the outage, fell as low as 19 Mbps. Interestingly, the median bandwidth didn’t see the clean gradual decline as it did in Portugal, instead falling and recovering twice before gradually declining.

As expected, latency at a country level saw a significant increase. Prior to the loss of power, median latency was around 22 ms, but grew to as much as 40 ms. As in Portugal, the lower download speeds and higher latency are likely due to the congestion of the network links that remained available.

Similar to Portugal, network infrastructure in Spain was also impacted by the power outage, with the impact seen as a drop in announced IP address space. By 14:30 UTC, the number of announced IPv4 /24 address blocks had fallen by around 2.4%, and continued to drop further over the following hours. The number of announced IPv6 /48 address blocks fell by over 8% during that same time span, and also continued to drop in the following hours.

Parts of Andorra and France were also reportedly impacted by the power outage, with additional outages reported as far away as Belgium. At a national level, no traffic disruptions were evident in any of the countries.

Analysis of traffic at a regional level in France shows a slight decline concurrent with the power outage in several regions, but the drops were nominal in comparison to Spain and Portugal, and traffic volumes recovered to expected levels within 90 minutes. No impact was evident at a regional level in Andorra.

It appears that Morocco may have been impacted in some fashion by the power outage, or at least Orange Maroc was. In a post on X, the provider stated (translated) “Internet traffic has been disrupted following a massive power outage in Spain and Portugal, which is affecting international connections.” Cloudflare Radar shows that traffic from the network fell sharply around 12:00 UTC, 90 minutes after the power outage began, with a full outage beginning around 15:00 UTC.

Power restoration in Spain had already started as this post was being written, and full recovery will likely take hours to days. As power is restored, Internet traffic and other metrics will recover as well. The current state of Internet connectivity in Spain and Portugal can be tracked on Cloudflare Radar.

The Cloudflare Radar team is constantly monitoring for Internet disruptions, sharing our observations on the Cloudflare Radar Outage Center, via social media, and in posts on blog.cloudflare.com. Follow us on social media at @CloudflareRadar (X), noc.social/@cloudflareradar (Mastodon), and radar.cloudflare.com (Bluesky), or contact us via email.

While rotating credentials used by the R2 Gateway service (R2's API frontend) to authenticate with our storage infrastructure, the R2 engineering team inadvertently deployed the new credentials (ID and key pair) to a development instance of the service instead of production. When the old credentials were deleted from our storage infrastructure (as part of the key rotation process), the production R2 Gateway service did not have access to the new credentials. This ultimately resulted in R2’s Gateway service not being able to authenticate with our storage backend. There was no data loss or corruption that occurred as part of this incident: any in-flight uploads or mutations that returned successful HTTP status codes were persisted.

Once the root cause was identified and we realized we hadn’t deployed the new credentials to the production R2 Gateway service, we deployed the updated credentials and service availability was restored. 

This incident happened because of human error and lasted longer than it should have because we didn’t have proper visibility into which credentials were being used by the Gateway Worker to authenticate with our storage infrastructure. 

We’re deeply sorry for this incident and the disruption it may have caused to you or your users. We hold ourselves to a high standard and this is not acceptable. This blog post exactly explains the impact, what happened and when, and the steps we are taking to make sure this failure (and others like it) doesn’t happen again.

The primary incident window occurred between 21:38 UTC and 22:45 UTC.

The following table details the specific impact to R2 and Cloudflare services that depend on, or interact with, R2:

All timestamps referenced are in Coordinated Universal Time (UTC).

R2’s architecture is primarily composed of three parts: R2 production gateway Worker (serves requests from S3 API, REST API, Workers API), metadata service, and storage infrastructure (stores encrypted object data).

The R2 Gateway Worker uses credentials (ID and key pair) to securely authenticate with our distributed storage infrastructure. We rotate these credentials regularly as a best practice security precaution.

Our key rotation process involves the following high-level steps:

1. Create a new set of credentials (ID and key pair) for our storage infrastructure. At this point, the old credentials are maintained to avoid downtime during credential change over.

2. Set the new credential secret for the R2 production gateway Worker using the wrangler secret put command.

3. Set the new updated credential ID as an environment variable in the R2 production gateway Worker using the wrangler deploy command. At this point, new storage credentials start being used by the gateway Worker.

4. Remove previous credentials from our storage infrastructure to complete the credential rotation process.

5. Monitor operational dashboards and logs to validate change over.

The R2 engineering team uses Workers environments to separate production and development environments for the R2 Gateway Worker. Each environment defines a separate isolated Cloudflare Worker with separate environment variables and secrets. 

Critically, both wrangler secret put and wrangler deploy commands default to the default environment if the --env command line parameter is not included. In this case, due to human error, we inadvertently omitted the --env parameter and deployed the new storage credentials to the wrong Worker (default environment instead of production). To correctly deploy storage credentials to the production R2 Gateway Worker, we need to specify --env production.

The action we took on step 4 above to remove the old credentials from our storage infrastructure caused authentication errors, as the R2 Gateway production Worker still had the old credentials. This is ultimately what resulted in degraded availability.

The decline in R2 availability metrics was gradual and not immediately obvious because there was a delay in the propagation of the previous credential deletion to storage infrastructure. This accounted for a delay in our initial discovery of the problem. Instead of relying on availability metrics after updating the old set of credentials, we should have explicitly validated which token was being used by the R2 Gateway service to authenticate with R2's storage infrastructure.

Overall, the impact on read availability was significantly mitigated by our intermediate cache that sits in front of storage and continued to serve requests.

Once we identified the root cause, we were able to resolve the incident quickly by deploying the new credentials to the production R2 Gateway Worker. This resulted in an immediate recovery of R2 availability.

This incident happened because of human error and lasted longer than it should have because we didn’t have proper visibility into which credentials were being used by the R2 Gateway Worker to authenticate with our storage infrastructure.

We have taken immediate steps to prevent this failure (and others like it) from happening again:

• Added logging tags that include the suffix of the credential ID the R2 Gateway Worker uses to authenticate with our storage infrastructure. With this change, we can explicitly confirm which credential is being used.

• Related to the above step, our internal processes now require explicit confirmation that the suffix of the new token ID matches logs from our storage infrastructure before deleting the previous token.

• Require that key rotation takes place through our hotfix release tooling instead of relying on manual wrangler command entry which introduces human error. Our hotfix release deploy tooling explicitly enforces the environment configuration and contains other safety checks.

• While it’s been an implicit standard that this process involves at least two humans to validate the changes ahead as we progress, we’ve updated our relevant SOPs (standard operating procedures) to include this explicitly.

• In Progress: Extend our existing closed loop health check system that monitors our endpoints to test new keys, automate reporting of their status through our alerting platform, and ensure global propagation prior to releasing the gateway Worker.

• In Progress: To expedite triage on any future issues with our distributed storage endpoints, we are updating our observability platform to include views of upstream success rates that bypass caching to give clearer indication of issues serving requests for any reason.

The list above is not exhaustive: as we work through the above items, we will likely uncover other improvements to our systems, controls, and processes that we’ll be applying to improve R2’s resiliency, on top of our business-as-usual efforts. We are confident that this set of changes will prevent this failure, and related credential rotation failure modes, from occurring again. Again, we sincerely apologize for this incident and deeply regret any disruption it has caused you or your users.

The incident occurred due to human error and insufficient validation safeguards during a routine abuse remediation for a report about a phishing site hosted on R2. The action taken on the complaint resulted in an advanced product disablement action on the site that led to disabling the production R2 Gateway service responsible for the R2 API.  

Critically, this incident did not result in the loss or corruption of any data stored on R2. 

We’re deeply sorry for this incident: this was a failure of a number of controls, and we are prioritizing work to implement additional system-level controls related not only to our abuse processing systems, but so that we continue to reduce the blast radius of any system- or human- action that could result in disabling any production service at Cloudflare.

All customers using Cloudflare R2 would have observed a 100% failure rate against their R2 buckets and objects during the primary incident window. Services that depend on R2 (detailed in the table below) observed heightened error rates and failure modes depending on their usage of R2.

The primary incident window occurred between 08:14 UTC to 09:13 UTC, when operations against R2 had a 100% error rate. Dependent services (detailed below) observed increased failure rates for operations that relied on R2.

From 09:13 UTC to 09:36 UTC, as R2 recovered and clients reconnected, the backlog and resulting spike in client operations caused load issues with R2's metadata layer (built on Durable Objects). This impact was significantly more isolated: we observed a 0.09% increase in error rates in calls to Durable Objects running in North America during this window. 

The following table details the impacted services, including the user-facing impact, operation failures, and increases in error rates observed:

The incident timeline, including the initial impact, investigation, root cause, and remediation, are detailed below.

All timestamps referenced are in Coordinated Universal Time (UTC).

At the R2 service level, our internal Prometheus metrics showed R2’s SLO near-immediately drop to 0% as R2’s Gateway service stopped serving all requests and terminated in-flight requests.

The slight delay in failure was due to the product disablement action taking 1–2 minutes to take effect as well as our configured metrics aggregation intervals:

For context, R2’s architecture separates the Gateway service, which is responsible for authenticating and serving requests to R2’s S3 & REST APIs and is the “front door” for R2 — its metadata store (built on Durable Objects), our intermediate caches, and the underlying, distributed storage subsystem responsible for durably storing objects. 

During the incident, all other components of R2 remained up: this is what allowed the service to recover so quickly once the R2 Gateway service was restored and re-deployed. The R2 Gateway acts as the coordinator for all work when operations are made against R2. During the request lifecycle, we validate authentication and authorization, write any new data to a new immutable key in our object store, then update our metadata layer to point to the new object. When the service was disabled, all running processes stopped.

While this means that all in-flight and subsequent requests fail, anything that had received a HTTP 200 response had already succeeded with no risk of reverting to a prior version when the service recovered. This is critical to R2’s consistency guarantees and mitigates the chance of a client receiving a successful API response without the underlying metadata and storage infrastructure having persisted the change.  

Due to human error and insufficient validation safeguards in our admin tooling, the R2 Gateway service was taken down as part of a routine remediation for a phishing URL.

During a routine abuse remediation, action was taken on a complaint that inadvertently disabled the R2 Gateway service instead of the specific endpoint/bucket associated with the report. This was a failure of multiple system level controls (first and foremost) and operator training. 

A key system-level control that led to this incident was in how we identify (or "tag") internal accounts used by our teams. Teams typically have multiple accounts (dev, staging, prod) to reduce the blast radius of any configuration changes or deployments, but our abuse processing systems were not explicitly configured to identify these accounts and block disablement actions against them. Instead of disabling the specific endpoint associated with the abuse report, the system allowed the operator to (incorrectly) disable the R2 Gateway service. 

Once we identified this as the cause of the outage, remediation and recovery was inhibited by the lack of direct controls to revert the product disablement action and the need to engage an operations team with lower level access than is routine. The R2 Gateway service then required a re-deployment in order to rebuild its routing pipeline across our edge network.

Once re-deployed, clients were able to re-connect to R2, and error rates for dependent services (including Stream, Images, Cache Reserve and Vectorize) returned to normal levels.

We have taken immediate steps to resolve the validation gaps in our tooling to prevent this specific failure from occurring in the future.

We are prioritizing several work-streams to implement stronger, system-wide controls (defense-in-depth) to prevent this, including how we provision internal accounts so that we are not relying on our teams to correctly and reliably tag accounts. A key theme to our remediation efforts here is around removing the need to rely on training or process, and instead ensuring that our systems have the right guardrails and controls built-in to prevent operator errors.

These work-streams include (but are not limited to) the following:

• Actioned: deployed additional guardrails implemented in the Admin API to prevent product disablement of services running in internal accounts.

• Actioned: Product disablement actions in the abuse review UI have been disabled while we add more robust safeguards. This will prevent us from inadvertently repeating similar high-risk manual actions.

• In-flight: Changing how we create all internal accounts (staging, dev, production) to ensure that all accounts are correctly provisioned into the correct organization. This must include protections against creating standalone accounts to avoid re-occurrence of this incident (or similar) in the future.

• In-flight: Further restricting access to product disablement actions beyond the remediations recommended by the system to a smaller group of senior operators.

• In-flight: Two-party approval required for ad-hoc product disablement actions. Going forward, if an investigator requires additional remediations, they must be submitted to a manager or a person on our approved remediation acceptance list to approve their additional actions on an abuse report. 

• In-flight: Expand existing abuse checks that prevent accidental blocking of internal hostnames to also prevent any product disablement action of products associated with an internal Cloudflare account.  

• In-flight: Internal accounts are being moved to our new Organizations model ahead of public release of this feature. The R2 production account was a member of this organization, but our abuse remediation engine did not have the necessary protections to prevent acting against accounts within this organization.

We’re continuing to discuss & review additional steps and effort that can continue to reduce the blast radius of any system- or human- action that could result in disabling any production service at Cloudflare.

We understand this was a serious incident, and we are painfully aware of — and extremely sorry for — the impact it caused to customers and teams building and running their businesses on Cloudflare.

This is the first (and ideally, the last) incident of this kind and duration for R2, and we’re committed to improving controls across our systems and workflows to prevent this in the future.

Below, we present a summary of key findings, and then explore them in more detail in subsequent sections.

• Global Internet traffic grew 17.2% in 2024. 🔗

• Google maintained its position as the most popular Internet service overall. OpenAI remained at the top of the Generative AI category. Binance remained at the top of the Cryptocurrency category. WhatsApp remained the top Messaging platform, and Facebook remained the top Social Media site. 🔗

• Global traffic from Starlink grew 3.3x in 2024, in line with last year’s growth rate. After initiating service in Malawi in July 2023, Starlink traffic from that country grew 38x in 2024. As Starlink added new markets, we saw traffic grow rapidly in those locations. 🔗

• Googlebot, Google’s web crawler, was responsible for the highest volume of request traffic to Cloudflare in 2024, as it retrieved content from millions of Cloudflare customer sites for search indexing. 🔗

• Traffic from ByteDance’s AI crawler (Bytespider) gradually declined over the course of 2024. Anthropic’s AI crawler (ClaudeBot) first started showing signs of ongoing crawling activity in April, then declined after an initial peak in May & June. 🔗

• 13.0% of TLS 1.3 traffic is using post-quantum encryption. 🔗

• Globally, nearly one-third of mobile device traffic was from Apple iOS devices. Android had a >90% share of mobile device traffic in 29 countries/regions; peak iOS mobile device traffic share was over 60% in eight countries/regions. 🔗

• Globally, nearly half of web requests used HTTP/2, with 20.5% using HTTP/3. Usage of both versions was up slightly from 2023. 🔗

• React, PHP, and jQuery were among the most popular technologies used to build websites, while HubSpot, Google, and WordPress were among the most popular vendors of supporting services and platforms. 🔗

• Go surpassed NodeJS as the most popular language used for making automated API requests. 🔗

• Google is far and away the most popular search engine globally, across all platforms. On mobile devices and operating systems, Baidu is a distant second. Bing is a distant second across desktop and Windows devices, with DuckDuckGo second most popular on macOS. Shares vary by platform and country/region. 🔗

• Google Chrome is far and away the most popular browser overall. While this is also true on macOS devices, Safari usage is well ahead of Chrome on iOS devices. On Windows, Edge is the second most popular browser as it comes preinstalled and is the initial default. 🔗

• 225 major Internet disruptions were observed globally in 2024, with many due to government-directed regional and national shutdowns of Internet connectivity. Cable cuts and power outages were also leading causes. 🔗

• Aggregated across 2024, 28.5% of IPv6-capable requests were made over IPv6. India and Malaysia were the strongest countries, at 68.9% and 59.6% IPv6 adoption respectively. 🔗

• The top 10 countries ranked by Internet speed all had average download speeds above 200 Mbps. Spain was consistently among the top locations across the measured Internet quality metrics. 🔗

• 41.3% of global traffic comes from mobile devices. In nearly 100 countries/regions, the majority of traffic comes from mobile devices. 🔗

• 20.7% of TCP connections are unexpectedly terminated before any useful data can be exchanged. 🔗

• 6.5% of global traffic was mitigated by Cloudflare's systems as being potentially malicious or for customer-defined reasons. In the United States, the share of mitigated traffic grew to 5.1%, while in South Korea, it dropped slightly to 8.1%. In 44 countries/regions, over 10% of traffic was mitigated. 🔗

• The United States was responsible for over a third of global bot traffic. Amazon Web Services was responsible for 12.7% of global bot traffic, and 7.8% came from Google. 🔗

• Globally, Gambling/Games was the most attacked industry, slightly ahead of 2023’s most targeted industry, Finance. 🔗

• Log4j, a vulnerability discovered in 2021, remains a persistent threat and was actively targeted throughout 2024. 🔗

• Routing security, measured as the share of RPKI valid routes and the share of covered IP address space, continued to improve globally throughout 2024. We saw a 4.7% increase in RPKI valid IPv4 address space in 2024, and a 6.4% increase in RPKI valid routes in 2024. 🔗

• An average of 4.3% of emails were determined to be malicious in 2024, although this figure was likely influenced by spikes observed in March, April, and May. Deceptive links and identity deception were the two most common types of threats found in malicious email messages. 🔗

• Over 99% of the email messages processed by Cloudflare Email Security from the .bar, .rest, and .uno top level domains (TLDs) were found to be either spam or malicious in nature. 🔗

Over the last four years (2020, 2021, 2022, 2023), we have aggregated perspectives from Cloudflare Radar into an annual Year In Review, illustrating the Internet’s patterns across multiple areas over the course of that year. The Cloudflare Radar 2024 Year In Review microsite continues that tradition, featuring interactive charts, graphs, and maps you can use to explore and compare notable Internet trends observed throughout this past year.

Cloudflare’s network currently spans more than 330 cities in over 120 countries/regions, serving an average of over 63 million HTTP(S) requests per second for millions of Internet properties, in addition to handling over 42 million DNS requests per second on average. The resulting data generated by this usage, combined with data from other complementary Cloudflare tools, enables Radar to provide unique near-real time perspectives on the patterns and trends around security, traffic, performance, and usage that we observe across the Internet. 

The 2024 Year In Review is organized into five sections: Traffic, Adoption & Usage, Connectivity, Security, and Email Security and covers the period from January 1 to December 1, 2024. We have incorporated several new metrics this year, including AI bot & crawler traffic, search engine and browser market share, connection tampering, and “most dangerous” top level domains (TLDs). To ensure consistency, we have kept underlying methodologies consistent with previous years’ calculations. Trends for 200 countries/regions are available on the microsite; smaller or less populated locations are excluded due to insufficient data. Some metrics are only shown worldwide, and are not displayed if a country/region is selected. 

Below, we provide an overview of the content contained within the major Year In Review sections (Traffic, Adoption & Usage, Connectivity, Security, and Email Security), along with notable observations and key findings. In addition, we have also published a companion blog post that specifically explores trends seen across Top Internet Services.

The key findings and associated discussion within this post only provide a high-level perspective on the unique insights that can be found in the Year in Review microsite. Visit the microsite to explore the various datasets and metrics in more detail, including trends seen in your country/region, how these trends have changed as compared to 2023, and how they compare to other countries/regions of interest. Surveying the Internet from this vantage point provides insights that can inform decisions on everything from an organization’s security posture and IT priorities to product development and strategy. 

An inflection point for Internet traffic arguably occurred thirty years ago. The World Wide Web went mainstream in 1994, thanks to the late 1993 release of the NCSA Mosaic browser for multiple popular operating systems, which included support for embedded images. In turn, “heavier” (in contrast to text-based) Internet content became the norm, and coupled with the growth in consumption through popular online services and the emerging consumer ISP industry, Internet traffic began to rapidly increase, and that trend has continued to the present.

To determine the traffic trends over time for the Year in Review, we use the average daily traffic volume (excluding bot traffic) over the second full calendar week (January 8-15) of 2024 as our baseline. (The second calendar week is used to allow time for people to get back into their “normal” school and work routines after the winter holidays and New Year’s Day. The percent change shown in the traffic trends chart is calculated relative to the baseline value — it does not represent absolute traffic volume for a country/region. The trend line represents a seven-day trailing average, which is used to smooth the sharp changes seen with data at a daily granularity. To compare 2024’s traffic trends with 2023 data and/or other locations, click the “Compare” icon at the upper right of the graph.

Throughout the first half of 2024, worldwide Internet traffic growth appeared to be fairly limited, within a percent or two on either side of the baseline value through mid-August. However, at that time growth clearly began to accelerate, climbing consistently through the end of November, growing 17.2% for the year. This trend is similar to those also seen in 2023 and 2022, as we discussed in the 2023 Year in Review blog post.

^{Internet traffic trends in 2024, worldwide}

The West African country of Guinea experienced the most significant Internet traffic growth seen in 2024, reaching as much as 350% above baseline. Traffic growth didn’t begin in earnest until late February, and reached an initial peak in early April. It remained between 100% and 200% above baseline until September, when it experienced several multi-week periods of growth. While the September-November periods of traffic growth also occurred in 2023, they peaked at under 90% above baseline.

The impact of significant Internet outages is also clearly visible when looking at data across the year. Two significant Internet outages in Cuba are clearly visible as large drops in traffic in October and November. A reported “complete disconnection” of the national electricity system on the island occurred on October 18, lasting just over three days. Just a couple of weeks later, on November 6, damage from Hurricane Rafael caused widespread power outages in Cuba, resulting in another large drop in Internet traffic. Traffic has remained lower as Cuba’s electrical infrastructure continues to struggle.

^{Internet traffic trends in 2024, Cuba}

As we frequently discuss in Cloudflare Radar blog and social media posts, government-directed Internet shutdowns occur all too frequently, and the impact of these actions are also clearly visible when looking at long-term traffic data. In Bangladesh, the government ordered the shutdown of mobile Internet connectivity on July 18, in response to student protests. Shortly after mobile networks were shut down, fixed broadband networks were taken offline as well, resulting in a near complete loss of Internet traffic from the country. Connectivity gradually returned over the course of several days, between July 23-28.

^{Internet traffic trends in 2024, Bangladesh}

As we also noted last year, the celebration of major holidays can also have a visible impact on Internet traffic at a country level. For example, in Muslim countries including Indonesia and the United Arab Emirates, the celebration of Eid al-Fitr, the festival marking the end of the fast of Ramadan, is visible as a noticeable drop in traffic around April 9-10. 

^{Internet traffic trends in 2024, Indonesia and United Arab Emirates}

Over the last several years, the Year In Review has ranked the most popular Internet services. These rankings cover an “overall” perspective, as well as a dozen more specific categories, based on analysis of anonymized query data of traffic to our 1.1.1.1 public DNS resolver from millions of users around the world. For the purposes of these rankings, domains that belong to a single Internet service are grouped together.

Google once again held the top spot overall, supported by its broad portfolio of services, as well as the popularity of the Android mobile operating system (more on that below). Meta properties Facebook, Instagram, and WhatsApp also held spots in the top 10.

Generative AI continued to grow in popularity throughout 2024, and in this category, OpenAI again held the top spot, building on the continued success and popularity of ChatGPT. Within Social Media, the top five remained consistent with 2023’s and 2022’s ranking, including Facebook, TikTok, Instagram, X, and Snapchat.

These categorical rankings, as well as trends seen by specific services, are explored in more detail in a separate blog post, From ChatGPT to Temu: ranking top Internet services in 2024.

SpaceX’s Starlink continues to be the leading satellite Internet service provider, bringing connectivity to unserved or underserved areas. In addition to opening up new markets in 2024, Starlink also announced relationships to provide in-flight connectivity to multiple airlines, and on cruise ships and trains, as well as enabling subscribers to roam with their subscription using the Starlink Mini.

We analyzed aggregate Cloudflare traffic volumes associated with Starlink's primary autonomous system (AS14593) to track the growth in usage of the service throughout 2024. Similar to the traffic trends discussed above, the request volume shown on the trend line in the chart represents a seven-day trailing average. Comparisons with 2023 data can be shown by clicking the “Compare” icon at the upper right of the graph. Within comparative views, the lines are scaled to the maximum value shown.

On a worldwide basis, steady, consistent growth was seen across the year, though it accelerates throughout November. This acceleration may have been driven by traffic associated with customer-specific large software updates. 

^{Starlink traffic growth worldwide in 2024}

In many locations, there is pent-up demand for “alternative” connectivity providers such as Starlink, and in these countries/regions, we see rapid traffic growth when service becomes available, such as in Zimbabwe. Service availability was announced on September 7, and traffic from the country began to grow rapidly almost immediately thereafter.

^{Starlink traffic growth in Zimbabwe in 2024}

In new markets, traffic growth continues after that initial increase. For example Starlink service became available in Malawi in July 2023, and throughout 2024, Starlink traffic from the country grew 38x. While Malawi’s 38x increase is impressive, other countries also experienced significant growth. In the Eastern European country of Georgia, service became available on November 1, 2023. After a slow ramp, traffic began to take off growing over 100x through 2024. In Paraguay, service availability was announced on December 21, and began to grow at the beginning of January, registering an increase of over 900x across the year.

^{Starlink traffic growth in Malawi in 2024}

Cloudflare Radar shows users Internet traffic trends over a selected period of time, but at a country/region or network level. However, as we did in 2023, we again wanted to look at the traffic Cloudflare saw over the course of the full year from the entire IPv4 Internet. To do so, we can use Hilbert curves, which allow us to visualize a sequence of IPv4 addresses in a two-dimensional pattern that keeps nearby IP addresses close to each other, making them useful for surveying the Internet's IPv4 address space.

Using a Hilbert curve, we can visualize aggregated IPv4 request traffic to Cloudflare from January 1 through December 1, 2024. Within the visualization, we aggregate IPv4 addresses at a /20 level, meaning that at the highest zoom level, each square represents traffic from 4,096 IPv4 addresses. This aggregation is done to keep the amount of data used for the visualization manageable. (While we would like to create a similar visualization for IPv6 traffic, the enormity of the full IPv6 address space would make associated traffic very hard to see in such a visualization, especially as such a small amount has been allocated for assignment by the Regional Internet Registries.)

Within the visualization, IP addresses are grouped by ownership, and for much of the IP address space shown there, a mouseover at the default zoom level will show the Regional Internet Registry (RIR) that the address block belongs to. However, there are also a number of blocks that were assigned prior to the existence of the RIR system, and for these, they are labeled with the name of the organization that owns them. Progressive zooming ultimately shows the autonomous system and country/region that the IP address block is associated with, as well as its share of traffic relative to the maximum. (If a country/region is selected, only the IP address blocks associated with that location are visible.) Overall traffic shares are indicated by shading based on a color scale, and although a number of large unshaded blocks are visible, this does not necessarily mean that the associated address space is unused, but rather that it may be used in a way that does not generate traffic to Cloudflare.

^{Hilbert curve showing aggregated 2024 traffic to Cloudflare across the IPv4 Internet}

Warmer orange/red shading within the visualization represents areas of higher request volume, and buried within one of those areas is the IP address block that had the maximum request volume to Cloudflare during 2024. As it was in 2023, this address block was 66.249.64.0/20, which belongs to Google, and is one of several used by the Googlebot web crawler to retrieve content for search indexing. This use of that address space is a likely explanation for the high request volume, given the number of web properties on Cloudflare’s network.

^{Zoomed Hilbert curve view showing the IPv4 address block that generated the highest volume of requests}

In addition to Google, owners of other prefixes in the top 20 include Alibaba, Microsoft, Amazon, and Apple. To explore the IPv4 Internet in more detail, we encourage you to go to the Year in Review microsite and explore it by dragging and zooming to move around IPv4 address space.

AI bots and crawlers have been in the news throughout 2024 as they voraciously consume content to train ever-evolving models. Controversy has followed them, as not all bots and crawlers respect content owner directives to restrict crawling activity. In July, Cloudflare enabled customers to block these bots and crawlers with a single click, and during Birthday Week we introduced AI Audit to give website owners even more visibility into and control over how AI platforms access their content. 

Tracking traffic trends for AI bots can help us better understand their activity over time — observing which are the most aggressive and have the highest volume of requests, which perform crawls on a regular basis, etc. The new AI bot & crawler traffic graph on Radar’s Traffic page, launched in September, provides insight into these traffic trends gathered over the selected time period for the top known AI bots. 

Looking at traffic trends from two of those bots, we can see some interesting patterns. Bytespider is a crawler operated by ByteDance, the Chinese owner of TikTok, and is reportedly used to download training data for ByteDance’s Large Language Models (LLMs). Bytespider’s crawling activity trended generally downwards over the course of 2024, with end-of-November activity approximately 80-85% lower than that seen at the start of the year. ClaudeBot is Anthropic’s crawler, which downloads training data for its LLMs that power AI products like Claude. Traffic from ClaudeBot appeared to be mostly non-existent through mid-April, except for some small spikes that possibly represent test runs. Traffic became more consistently non-zero starting in late April, but after an early spike, trailed off through the remainder of the year.

^{Traffic trends for AI crawlers Bytespider and ClaudeBot in 2024}

Traffic trends for the full list of AI bots & crawlers can be found in the Cloudflare Radar Data Explorer.

The term “post-quantum” refers to a new set of cryptographic techniques designed to protect data from adversaries that have the ability to capture and store current data for decryption by sufficiently powerful quantum computers in the future. The Cloudflare Research team has been exploring post-quantum cryptography since 2017.

In October 2022, we enabled post-quantum key agreement on our network by default, but use of it requires that browsers and clients support it as well. In 2024, Google's Chrome 124 enabled it by default on April 17, and adoption grew rapidly following that release, increasing from just over 2% of requests to around 12% within a month, and ended November at 13%. We expect that adoption will continue to grow into and during 2025 due to support in other Chromium-based browsers, growing default support in Mozilla Firefox, and initial testing in Apple Safari.

^{Growth trends in post-quantum encrypted TLS 1.3 traffic during 2024}

The two leading mobile device operating systems globally are Apple’s iOS and Google’s Android, and by analyzing information in the user agent reported with each request, we can get insight into the distribution of traffic by client operating system throughout the year. Again, we found that Android is responsible for the majority of mobile device traffic when aggregated globally, due to the wide distribution of price points, form factors, and capabilities.

Similar to 2023’s findings, Android was once again responsible for just over two-thirds of mobile device traffic. Looking at the top countries for Android traffic, we find a greater than 95% share in Sudan, Bangladesh, Turkmenistan, Malawi, Papua New Guinea, Syria, and Yemen, up from just two countries in 2023. Similar to last year, we again found that countries/regions with higher levels of Android usage are largely in Africa, Oceania/Asia, and South America, and that many have lower levels of gross national income per capita. In these countries/regions, the availability of lower priced “budget” Android devices supports increased adoption.

^{Global distribution of mobile device traffic by operating system in 2024}

In contrast, iOS adoption tops out in the 65% range in Jersey, the Faroe Islands, Guernsey, and Denmark. Adoption rates of 50% or more were seen in a total of 26 countries/regions, including Norway, Sweden, Australia, Japan, the United States, and Canada. These locations likely have a greater ability to afford higher priced devices, owing to their comparatively higher gross national income per capita.

^{Countries/regions with the largest share of iOS traffic in 2024}

HTTP (HyperText Transfer Protocol) is the core protocol that the web relies upon. HTTP/1.0 was first standardized in 1996, HTTP/1.1 in 1999, and HTTP/2 in 2015. The most recent version, HTTP/3, was completed in 2022, and runs on top of a new transport protocol known as QUIC. By running on top of QUIC, HTTP/3 can deliver improved performance by mitigating the effects of packet loss and network changes, as well as establishing connections more quickly. HTTP/3 also provides encryption by default, which mitigates the risk of attacks. 

Current versions of desktop and mobile Google Chrome (and Chromium-based variants), Mozilla Firefox, and Apple Safari all support HTTP/3 by default. Cloudflare makes HTTP/3 available for free to all of our customers, although not every customer chooses to enable it.

Analysis of the HTTP version negotiated for each request provides insight into the distribution of traffic by the various versions of the protocol aggregated across the year. (“HTTP/1.x” aggregates requests made over HTTP/1.0 and HTTP/1.1.) At a global level, 20.5% of requests in 2024 were made using HTTP/3. Another 29.9% of requests were made over the older HTTP/1.x versions, while HTTP/2 remained dominant, accounting for the remaining 49.6%.

^{Global distribution of traffic by HTTP version in 2024}

Looking at version distribution geographically, we found eight countries/regions sending more than a third of their requests over HTTP/3, with Reunion, Sri Lanka, Mongolia, Greece, and North Macedonia comprising the top five as shown below. Eight other countries/regions, including Iran, Ireland, Hong Kong, and China, sent more than half of their requests over HTTP/1.x throughout 2024. More than half of requests were made over HTTP/2 in a total of 147 countries/regions.

^{Countries/regions with the largest shares of HTTP/3 traffic in 2024}

Modern websites and applications are extremely complex, built on and integrating on a mix of frameworks, platforms, services, and tools. In order to deliver a seamless user experience, developers must ensure that all of these components happily coexist with each other. Using Cloudflare Radar’s URL Scanner, we again scanned websites associated with the top 5000 domains to identify the most popular technologies and services used across a dozen different categories. 

In looking at core technologies used to build websites, React had a commanding lead over Vue.js and other JavaScript frameworks, PHP was the most popular programming technology, and jQuery’s share was 10x other popular JavaScript libraries.

Third-party services and platforms are also used by websites and applications to support things like analytics, content management, and marketing automation. Google Analytics remained the most widely used analytics provider, WordPress had a greater than 50% share among content management systems, and for marketing automation providers, category leader HubSpot had nearly twice the usage share of Marketo and MailChimp.

^{Top website technologies, JavaScript frameworks category in 2024}

Many dynamic websites and applications are built on automated API calls, and we can use our unique visibility into Web traffic to identify the top languages these API clients are written in. Applying heuristics to API-related requests determined to not be coming from a person using a browser or native mobile application helps us to identify the language used to build the API client.

Our analysis found that almost 12% of automated API requests are made by Go-based clients, with NodeJS, Python, Java, and .NET holding smaller shares. Compared to 2023, Go’s share increased by approximately 40%, allowing it to capture the top spot, while NodeJS’s share fell by just over 30%. Python and Java also saw their shares increase, while .NET’s fell.

^{Most popular API client languages in 2024}

Protecting and accelerating websites and applications for millions of customers, Cloudflare is in a unique position to measure search engine market share data. Our methodology uses HTTP’s referer header to identify the search engine sending traffic to customer sites and applications. The market share data is presented as an overall aggregate, as well as broken out by device type and operating system. (Device type and operating system data is derived from the User-Agent and Client Hints headers accompanying a content request.)

Aggregated at a global level, Google referred the most traffic to Cloudflare customers, with a greater than 88% share across 2024. Yandex, Baidu, Bing, and DuckDuckGo round out the top five, all with single digit percentage shares. 

^{Overall worldwide search engine market share in 2024}

However, when drilling down by location or platform, differences are apparent in the top search engines and their shares. For example, in South Korea, Google is responsible for only two-thirds of referrals, while local platform Naver drives 29.2%, with local portal Daum also in the top five at 1.3%.

^{Overall search engine market share in South Korea in 2024}

Google’s dominance is also blunted a bit on Windows devices, where it drives only 80% of referrals globally. Unsurprisingly, Bing holds the second spot for Windows users, with a 10.4% share. Yandex, Yahoo, and DuckDuckGo round out the top 5, all with shares below 5%.

^{Overall worldwide search engine market share for Windows devices in 2024}

For additional details, including search engines aggregated under “Other”, please refer to the quarterly Search Engine Referral Reports on Cloudflare Radar.

Similar to our ability to measure search engine market share, Cloudflare is also in a unique position to measure browser market share. Our methodology uses information from the User-Agent and Client Hints headers to identify the browser making content requests, along with the associated operating system. Browser market share data is presented as an overall aggregate, as well as broken out by device type and operating system. Note that the shares of browsers available on both desktop and mobile devices, such as Chrome or Safari, are presented in aggregate.

Globally, we found that 65.8% of requests came from Google’s Chrome browser across 2024, and that just 15.5% came from Apple’s Safari browser. Microsoft Edge, Mozilla Firefox, and the Samsung Internet browser rounded out the top five, all with shares below 10%.

^{Overall worldwide web browser market share in 2024}

Similar to the search engine statistics discussed above, differences are clearly visible when drilling down by location or platform. In some countries where iOS holds a larger market share than Android, Chrome remains the leading browser, but by a much lower margin. For example, in Sweden, Chrome’s share fell to 56.2%, while Safari’s increased to 22.5%. In Norway, Chrome fell to just 50%, while Safari grew to 25.6%.

^{Overall web browser market share in Norway in 2024}

As the default browser on devices running iOS, Apple Safari was the most popular browser for iOS devices, commanding an 81.7% market share across the year, with Chrome at just 16.1%. And despite being the preinstalled default browser on Windows devices, Edge held just a 17.3% share, in comparison to Chrome’s 68.5%

^{Overall worldwide web browser market share for iOS devices in 2024}

For additional details, including browsers aggregated under “Other”, please refer to the quarterly Browser Market Share Reports on Cloudflare Radar.

Throughout 2024, as we have over the last several years, we have written frequently about observed Internet outages, whether due to cable cuts, unspecified technical issues, government-directed shutdowns, or a number of other reasons covered in our quarterly summary posts (Q1, Q2, Q3). The impacts of these outages can be significant, including significant economic losses and severely limited communications. The Cloudflare Radar Outage Center tracks these Internet outages, and uses Cloudflare traffic data for insights into their scope and duration.

Some of the outages seen through the year were short-lived, lasting just a few hours, while others stretched on for days or weeks. In the latter category, an Internet outage in Haiti dragged on for eight days in September because repair crews were barred from accessing a damaged submarine cable due to a business dispute, while shutdowns of mobile and fixed Internet providers in Bangladesh lasted for approximately 10 days in July. In the former category, Iraq frequently experienced multi-hour nationwide Internet shutdowns intended to prevent cheating on academic exams — these contribute to the clustering visible in the timeline during June, July, August, and September.

Within the timeline on the Year in Review microsite, hovering over a dot will display metadata about that outage, and clicking on it will open a page with additional information. Below the map and timeline, we have added a bar graph illustrating the recorded reasons associated with the observed outages. In 2024, over half were due to government-directed shutdowns. If a country/region is selected, only outages and reasons for that country/region will be displayed.

^{Over 200 Internet outages were observed around the world during 2024}

The IPv4 protocol still used by many Internet-connected devices was developed in the 1970s, and was never meant to handle the vast and growing scale of the modern Internet. An initial specification for its successor, IPv6, was published in December 1995, evolving to a draft standard three years later, offering an expanded address space intended to better support the expected growth in the number of Internet-connected devices. At this point, available IPv4 space has long since been exhausted, and connectivity providers use solutions like Network Address Translation to stretch limited IPv4 resources. Hungry for IPv4 address space as their businesses and infrastructure grow, cloud and hosting providers are acquiring blocks of IPv4 address space for as much as \$30 - \$50 per address. 

Cloudflare has been a vocal and active advocate for IPv6 since 2011, when we announced our Automatic IPv6 Gateway, which enabled free IPv6 support for all of our customers. In 2014, we enabled IPv6 support by default for all of our customers, but not all customers choose to keep it enabled for a variety of reasons. Note that server-side support is only half of the equation for driving IPv6 adoption, as end user connections need to support it as well. (In reality, it is a bit more complex than that, but server and client side support across applications, operating systems, and network environments are the two primary requirements. From a network perspective, implementing IPv6 also brings a number of other benefits.) By analyzing the IP version used for each request made to Cloudflare, aggregated throughout the year, we can get insight into the distribution of traffic by the various versions of the protocol.

At a global level, 28.5% of IPv6-capable (“dual-stack”) requests were made over IPv6, up from 26.4% in 2023. India was again the country with the highest level of IPv6 adoption, at 68.9%, carried in large part by 94% IPv6 adoption at Reliance Jio, one of the country’s largest Internet service providers. India was followed closely by Malaysia, where 59.6% of dual-stacked requests were made over IPv6 during 2024, thanks to strong IPv6 adoption rates across leading Internet providers within the country. IPv6 adoption in India was up from 66% in 2023, and in Malaysia, it was up from 57.3% last year. Saudi Arabia was the only other country with an IPv6 adoption rate above 50% this year, at 51.8%, whereas that list also included Vietnam, Greece, France, Uruguay, and Thailand in 2023. Thirty four countries/regions, including many in Africa, still have IPv6 adoption rates below 1%, while a total of 96 countries/regions have adoption rates below 10%.

^{Global distribution of traffic by IP version in 2024}

^{Countries/regions with the largest shares of IPv6 traffic in 2024}

As more and more of our everyday lives move online, including entertainment, work, education, finance, shopping, and even basic social and personal interaction, the quality of our Internet connections is arguably more important than ever, necessitating higher connection speeds and lower latency. Although Internet providers continue to evolve their service portfolios to offer increased connection speeds and reduced latency in order to support growth in use cases like videoconferencing, live streaming, and online gaming, consumer adoption is often mixed due to cost, availability, or other issues. By aggregating the results of speed.cloudflare.com tests taken during 2024, we can get a geographic perspective on connection quality metrics including average download and upload speeds, and average idle and loaded latencies, as well as the distribution of the measurements.

In 2024, Spain was a leader in download speed (292.6 Mbps) and upload speed (192.6 Mbps) metrics, and placed second globally for loaded latency (78.6 ms). (Loaded latency is the round-trip time when data-heavy applications are being used on the network.) Spain’s leadership in these connection quality metrics is supported by the strong progress that the country has made towards achieving the EU’s “Digital Decade” objectives, including fixed very high capacity network (VHCN) deployment, fiber-to-the-premises (FTTP) coverage, and 5G coverage with the latter two reaching 95.2% and 92.3% respectively. High speed fiber broadband connections are also relatively affordable, with research showing major providers offering 100 Mbps, 300 Mbps, 600 Mbps, and 1 Gbps packages, with the latter priced between €30 and €46 per month. The figures below for Spain show the largest clusters of speed measurements around the 100 Mbps mark, with slight bumps also visible around 300 Mbps, suggesting that the former package has the highest subscription rate, followed by the latter. Further, they show these connections are also relatively low latency, with 87% of idle latency measurements below 50 ms and 65% of loaded latency measurements below 100 ms, providing users with good gaming and videoconferencing/streaming experiences.

^{Measured download/upload speed distribution in Spain in 2024}

^{Measured idle/loaded latency distribution in Spain in 2024}

With approximately 70% of the world’s population using smartphones, and 91% of Americans owning a smartphone, these mobile devices have become an integral part of both our personal and professional lives, providing us with Internet access from nearly any place at any time. In some countries/regions, mobile devices primarily connect to the Internet via Wi-Fi, while other countries/regions are “mobile first”, where 4G/5G services are the primary means of Internet access.

Analysis of information contained with the user agent reported with each request to Cloudflare enables us to categorize it as coming from a mobile, desktop, or other type of device. Aggregating this categorization throughout the year at a global level, we found that 41.3% of traffic came from mobile devices, with 58.7% coming from desktop devices such as laptops and “classic” PCs. These traffic shares were in line with those measured in both 2023 and 2022, suggesting that mobile device usage has achieved a “steady state”. Over 77% of traffic came from mobile devices in Sudan, Cuba, and Syria, making them the countries/regions with the largest mobile device traffic share in 2024. Other countries/regions that had more than 50% of traffic come from mobile devices were concentrated in the Middle East/Africa, the Asia Pacific region, and South/Central America. 

^{Global distribution of traffic by device type in 2024}

^{Countries/regions with the largest shares of mobile device usage in 2024}

Cloudflare is in a unique position to help measure the health and behaviors of Internet networks around the world. One way we do this is passively measuring rates of connections to Cloudflare that appear anomalous, meaning that they are unexpectedly terminated before any useful data exchange occurs. The underlying causes of connection anomalies are varied and range from DoS attacks to quirky client behavior to third-party connection tampering (e.g., when a network monitors and selectively disrupts connections to filter content).

Connection anomalies are symptoms — visible signs that “something abnormal” is happening in a network, but the underlying root cause is not always clear from the outset. However, we can gain a better understanding by incorporating previously-reported network behaviors, active measurements and on-the-ground reports, and macro trends across networks. Additional details on such analysis can be found in the blog posts A global assessment of third-party connection tampering and Bringing insights into TCP resets and timeouts to Cloudflare Radar.

Insights into TCP connection anomalies were launched on Cloudflare Radar in September, with the plot lines in the associated graph corresponding to the stage of the TCP connection in which the connection anomalously closed (using shorthand, the first three messages we typically receive from the client in a TCP connection are “SYN” and “ACK” packets to establish a connection, and then a “PSH” packet indicating the requested resource). In aggregate globally, over 20% of connections to Cloudflare were terminated unexpectedly, with the largest share (nearly half) being closed “Post SYN” — that is, after our server has received a client’s SYN packet, but before we have received a subsequent acknowledgement (ACK) from the client or any useful data that would follow the acknowledgement. These terminations can often be attributed to DoS attacks or Internet scanning. Post-ACK (3.1% globally) and Post-PSH (1.4% globally) anomalies are more often associated with connection tampering, especially when they occur at high rates in specific networks.

^{Trends in TCP connection anomalies by stage in 2024}

To protect customers from threats posed by malicious bots used to attack websites and applications, Cloudflare mitigates this attack traffic using DDoS mitigation techniques or Web Application Firewall (WAF) Managed Rules. For a variety of other reasons, customers may also want Cloudflare to mitigate traffic using techniques like rate-limiting requests, or blocking all traffic from a given location, even if it isn’t malicious. Analyzing traffic to Cloudflare’s network throughout 2024, we looked at the overall share that was mitigated for any reason, as well as the share that was blocked as a DDoS attack or by WAF Managed Rules. 

In 2024, 6.5% of global traffic was mitigated, up almost one percentage point from 2023. Just 3.2% was mitigated as a DDoS attack, or by WAF Managed Rules, a rate slightly higher than in 2023. More than 10% of the traffic originating from 44 countries/regions had mitigations generally applied, while DDoS/WAF mitigations were applied to more than 10% of the traffic originating from just seven countries/regions.

At a country/region level, Albania had one of the highest mitigated traffic shares throughout the year, at 42.9%, while Libya had one of the highest shares of traffic that was mitigated as a DDoS attack or by WAF Managed Rules, at 19.2%. In 2023’s Year in Review blog post, we highlighted the United States and Korea. This year, the share of mitigated traffic grew to 5.0% in the United States (up from 3.65% in 2023), while in South Korea, it dropped slightly to 8.1%, down from 8.36%.

^{Trends in mitigated traffic worldwide in 2024}

Bot traffic describes any non-human Internet traffic, and by monitoring traffic suspected to be from bots site and application owners can spot and, if necessary, block potentially malicious activity. However, not all bots are malicious — bots can also be helpful, and Cloudflare maintains a list of verified bots that includes those used for things like search engine indexing, performance testing, and availability monitoring. Regardless of intent, we analyzed where bot traffic was originating from in 2024, using the IP address of a request to identify the network (autonomous system) and country/region associated with the bot making the request. Cloud platforms remained among the leading sources of bot traffic due to a number of factors. These include the ease of using automated tools to quickly provision compute resources, the relatively low cost of using these compute resources in an ephemeral manner, the broadly distributed geographic footprint of cloud platforms, and the platforms’ high-bandwidth Internet connectivity.

Globally, we found that 68.5% of observed bot traffic came from the top 10 countries in 2024, with the United States responsible for half of that total, over 5x the share of second place Germany. (In comparison to 2023, the US share was up slightly, while Germany’s was down slightly.) Among cloud platforms that originate bot traffic, Amazon Web Services was responsible for 12.7% of global bot traffic, and 7.8% came from Google. Microsoft, Hetzner, Digital Ocean, and OVH all also contributed more than a percent each.

^{Global bot traffic distribution by source country in 2024}

^{Global bot traffic distribution by source network in 2024}

The industries targeted by attacks often shift over time, depending on the intent of the attackers. They may be trying to cause financial harm by attacking ecommerce sites during a busy shopping period, gain an advantage against opponents by attacking an online game, or make a political statement by attacking government-related sites. To identify industry-targeted attack activity during 2024, we analyzed mitigated traffic for customers that had an associated industry and vertical within their customer record. Mitigated traffic was aggregated weekly by source country/region across 19 target industries.

Companies in the Gambling/Games industry were, in aggregate, the most attacked during 2024, with 6.6% of global mitigated traffic targeting the industry. The industry was slightly ahead of Finance, which led 2023’s aggregate list. (Both industries are shown at 6.6% in the Summary view due to rounding.)  Gambling/Games sites saw the largest shares of mitigated traffic in January and the first week of February, possibly related to National Football League playoffs in the United States, heading into the Super Bowl.

Attacks targeting Finance organizations were most active in May, reaching a peak of 15.3% of mitigated traffic the week of May 13. This is in line with the figure in our DDoS threat report for Q2 2024 that shows that Financial Services was the most attacked industry by request volume during the quarter in South America and the Middle East region.

As we have seen in the past, peak attack activity varied by industry on a weekly basis. The highest peaks for the year were seen in attacks targeting People & Society organizations (19.6% of mitigated traffic, week of January 1), the Autos & Vehicles industry (29.7% of mitigated traffic, week of January 15), and the Real Estate industry (27.5% of mitigated traffic, week of August 26).

^{Global mitigated traffic share by industry in 2024, summary view}

In December 2021, we published a series of blog posts about the Log4j vulnerability, highlighting the threat that it posed, our observations of attempted exploitation, and the steps we took to protect customers. Two years on, in our 2023 Year in Review, we noted that even as an older vulnerability, Log4j remained a top target for attacks during 2023, with related attack activity significantly higher than other commonly exploited vulnerabilities.

In 2024, three years after the initial Log4j disclosure, we found that Log4j remains an active threat. This year, we compared normalized daily attack activity for Log4j with attack activity for Atlassian Confluence Code Injection, a vulnerability we examined in the 2023 Year in Review, as well as aggregated daily attack activity for multiple CVEs related to Authentication Bypass and Remote Code Execution vulnerabilities published in 2024.

Log4j attack activity appeared to trend generally upwards across the year, with several significant spikes visible during the first half of the year, and then again in October and November. In terms of the difference in activity, Log4j ranges from approximately 4x to over 20x the activity seen for Atlassian Confluence Code Injection, and as much as 100x the aggregated activity seen for Authentication Bypass or Remote Code Injection vulnerabilities.  

^{Global attack activity trends for commonly exploited vulnerabilities in 2024}

As the routing protocol that underpins the Internet, Border Gateway Protocol (BGP) communicates routes between networks, enabling traffic to flow between source and destination. BGP, however, relies on trust between networks, and incorrect information shared between peers, whether or not it was shared intentionally, can send traffic to the wrong place, potentially with malicious results. Resource Public Key Infrastructure (RPKI) is a cryptographic method of signing records that associate a BGP route announcement with the correct originating autonomous system (AS) number, providing a way of ensuring that the information being shared originally came from a network that is allowed to do so. (It is important to note that this is only half of the challenge of implementing routing security, because network providers also need to validate these signatures and filter out invalid announcements to prevent sharing them further.)

Cloudflare has long been an advocate for routing security, including being a founding participant in the MANRS CDN and Cloud Programme and providing a public tool that enables users to test whether their Internet provider has implemented BGP safely. Building on insights available in the Routing page on Cloudflare Radar, we analyzed data from RIPE NCC's RPKI daily archive to determine the share of RPKI valid routes (as opposed to those route announcements that are invalid or whose status is unknown) and how that share has changed over the course of 2024, as well as determining the share of IP address space covered by valid routes. The latter metric is of interest because a route announcement covering a significant amount of IP address space (millions of IPv4 addresses, for example) has a greater potential impact than an announcement covering a small block of IP address space (hundreds of IPv4 addresses, for example).

At a global level during 2024, we saw a 6.4 percentage point increase (from 43.4% to 49.8%) in valid IPv4 routes, and a 3.2 percentage point increase (from 53.7% to 56.9%) in valid IPv6 routes. Given the trajectory, it is likely that over half of IPv4 routes will be RPKI valid by the end of calendar year 2024. Looking at the global share of IP address space covered by valid routes, we saw a 4.7 percentage point increase (from 38.9% to 43.6%) for IPv4, and a 3.3 percentage point increase (from 57.6% to 60.9%) for IPv6.

^{Shares of global RPKI valid routing entries by IP version in 2024}

^{Shares of globally announced IP address space covered by RPKI valid routes in 2024}

Spain started 2024 with less than half of its routes (both IPv4 and IPv6) RPKI valid. However, the share of valid routes grew significantly on February 15, when AS12479 (Orange Espagne) signed records associated with 98% of their IP address prefixes that were previously in an “unknown” (or NotFound) state of RPKI validity, thus converting these prefixes from unknown to valid. That drove an immediate increase for IPv4 to 76%, reaching 81% validity by December 1, and an immediate increase for IPv6 to 91%, reaching 92.9% validity by December 1. A notable change in covered IP address space was observed in Cameroon, where covered IPv4 space more than doubled at the end of January, growing from 32% to 82%. This was due to AS36912 (Orange Cameroun) signing records associated with all of their IPv4 address prefixes, changing the associated IP address space to RPKI valid. 

^{IPv4 and IPv6 shares of RPKI valid routes for Spain in 2024}

^{Share of IPv4 address space covered by RPKI valid routes for Cameroon in 2024}

Despite the growing enterprise use of collaboration/messaging apps, email remains an important business application and is a very attractive entry point into enterprise networks for attackers. Attackers will send targeted malicious emails that attempt to impersonate an otherwise legitimate sender (such as a corporate executive), that try to get the user to click on a deceptive link, or that contain a dangerous attachment, among other types of threats. Cloudflare Email Security protects customers from email-based attacks, including those carried out through targeted malicious email messages. During 2024, an average of 4.3% of emails analyzed by Cloudflare were found to be malicious. Aggregated at a weekly level, spikes above 14% were seen in late March, early April, and mid-May. We believe that these spikes were related to targeted “backscatter” attacks, where the attacker flooded a target with undeliverable messages, which then bounced the messages to the victim, whose email had been set as the reply-to: address.

^{Global malicious email share trends in 2024}

Attackers use a variety of techniques, which we refer to as threat categories, when they use malicious email messages as an attack vector. These categories are defined and explored in detail in our phishing threats report. In our analysis of malicious emails, we have found that such messages may contain multiple types of threats. In reviewing a weekly aggregation of threat activity trends for these categories, we found that, averaged across 2024, 42.9% of malicious email messages contained deceptive links, with the share reaching 70% at times throughout the year. Activity for this thread category was spiky, with low points seen in the March to May timeframe, and a general downward trend visible from July through November.

Identity deception was a similarly active threat category, with such threats also found in up to 70% of analyzed emails several weeks throughout the year. Averaged across 2024, 35.1% of emails contained attempted identity deception. The activity pattern for this threat category appears to be somewhat similar to deceptive links, with a number of the peaks and valleys occurring during the same weeks. At times, identity deception was a more prevalent threat in analyzed emails than deceptive links, as seen in the graph below.

Among other threat categories, extortion saw the most significant change throughout the year. After being found in 86% of malicious emails during the first week of January, its share gradually trended lower throughout the year, finishing November under 10%.

^{Global malicious email threat category trends for Deceptive Links and Identity Deception in 2024}

In March 2024, we launched a set of email security insights on Cloudflare Radar, including visibility into so-called “dangerous domains” — those top level domains (TLDs) that were found to be the sources of the most spam or malicious email among messages analyzed by Cloudflare Email Security. The analysis is based on the sending domain’s TLD, found in the From: header of an email message. For example, if a message came from sender@example.com, then example.com is the sending domain, and .com is the associated TLD.

In aggregate across 2024, we found that the .bar, .rest, and .uno TLDs were the “most dangerous”, each with over 99% of analyzed email messages characterized as either spam or malicious. (These TLDs are all at least a decade old, and each sees at least some usage, with between 20,000 and 60,000 registered domain names.)  Sorting by malicious email share, the .ws ccTLD (country code top level domain) belonging to Western Samoa came out on top, with over 90% of analyzed emails categorized as malicious. Sorting by spam email share, .quest is the biggest offender, with over 88% of emails originating from associated domains characterized as spam.

^{TLDs originating the largest total shares of malicious and spam email in 2024}

The Internet is an amazingly complex and dynamic organism, constantly changing, growing, and evolving.

With the Cloudflare Radar 2024 Year In Review, we are providing insights into the change, growth, and evolution that we have measured and observed throughout the year. Trend graphs, maps, tables, and summary statistics provide our unique perspectives on Internet traffic, Internet quality, and Internet security, and how key metrics across these areas vary around the world and over time.

We strongly encourage you to visit the Cloudflare Radar 2024 Year In Review microsite and explore the trends for your country/region, and to consider how they impact your organization so that you are appropriately prepared for 2025. In addition, for insights into the top Internet services across multiple industry categories, we encourage you to read the companion Year in Review blog post, From ChatGPT to Temu: ranking top Internet services in 2024.

If you have any questions, you can contact the Cloudflare Radar team at radar@cloudflare.com or on social media at @CloudflareRadar (X), https://noc.social/@cloudflareradar (Mastodon), and radar.cloudflare.com (Bluesky).

As it is every year, it truly is a team effort to produce the data, microsite, and content for our annual Year in Review, and I’d like to acknowledge those team members that contributed to this year’s effort. Thank you to: Jorge Pacheco, Sabina Zejnilovic, Carlos Azevedo, Mingwei Zhang (Data Analysis); André Jesus, Nuno Pereira (Front End Development); João Tomé (Most popular Internet services); Jackie Dutton, Kari Linder, Guille Lasarte (Communications); Eunice Giles (Brand Design); Jason Kincaid (blog editing); and Paula Tavares (Engineering Management), as well as countless other colleagues for their answers, edits, support, and ideas.

When cable cuts occur, whether submarine or terrestrial, they often result in observable disruptions to Internet connectivity, knocking a network, city, or country offline. This is especially true when there is insufficient resilience or alternative paths — that is, when a cable is effectively a single point of failure. Associated observations of traffic loss resulting from these disruptions are frequently covered by Cloudflare Radar in social media and blog posts. However, two recent cable cuts that occurred in the Baltic Sea resulted in little-to-no observable impact to the affected countries, as we discuss below, in large part because of the significant redundancy and resilience of Internet infrastructure in Europe.

On Sunday, November 17 2024, the BCS East-West Interlink submarine cable connecting Sventoji, Lithuania and Katthammarsvik, Sweden was reportedly damaged around 10:00 local (Lithuania) time (08:00 UTC). A Data Center Dynamics article about the cable cut quotes the CTO of Telia Lietuva, the telecommunications provider that operates the cable, and notes “The Lithuanian cable carried about a third of the nation's Internet capacity, but capacity was carried via other routes.”

As the Cloudflare Radar graphs below show, there was no apparent impact to traffic volumes in either country at the time that the cables were damaged. The NetFlows graphs represent the number of bytes that Cloudflare sends to users and clients in response to their requests.

Internet quality metrics for both countries show changes in measured bandwidth and latency throughout the day on Sunday, but with no sudden anomalous shifts visible around the time of the cable cut. (The loss of connectivity associated with a cable cut potentially manifests itself as an increase in latency and concurrent decrease in bandwidth due to loss of capacity.) The latency graph for Sweden does show an increase in latency, but it began before the cable cut occurred, is similar to a pattern visible several hours earlier, and is matched by an increase in measured bandwidth, so it is unlikely to be related to the cable cut event.

BGP announcements are a way for network providers to communicate routing information to other networks, and announcement activity observed on Telia Lietuva’s autonomous systems around the time of the cable cut may be related to the re-routing referenced in the article. No change in announced IP address space was visible for any of these autonomous systems, suggesting no loss of connectivity as the capacity was re-routed.

Telegeography’s submarinecablemap.com illustrates, at least in part, the resilience in connectivity enjoyed by these two countries. In addition to the damaged cable, it shows that Lithuania is connected to neighboring Latvia as well as to the Swedish mainland. Over 20 submarine cables land in Sweden, connecting it to multiple countries across Europe. In addition to the submarine resilience, network providers in both countries can take advantage of terrestrial fiber connections to neighboring countries, such as those illustrated in a European network map from Arelion (formerly Telia), which is only one of the large European backbone providers.

Less than a day later, the C-Lion1 submarine cable, which connects Helsinki, Finland and Rostock Germany was reportedly damaged during the early morning hours of Monday, November 18. Cinia, the telecommunications company that owns the cable, said that the cable stopped working at about 02:00 UTC. 

In this situation as well, as the Cloudflare Radar graphs below show, there was no apparent impact to traffic volumes in either country at the time that the cables were damaged. The Finland graphs, week-on-week, show fewer bytes transferred and fewer HTTP requests, but that difference is present before the cable cut at 02:00 UTC. However, the trend of the current line does not change after the cable cut, so the two events would appear unrelated. 

By looking at volume-related metrics, alone, Internet connectivity would appear to be unaffected by the cable cut.

If, however, we change perspective and look at Internet quality, a brief yet interesting change is visible for Finland around the reported time of the cable damage, though it isn’t clear whether it is related in any way. Just after midnight, median measured bandwidth, previously consistent around 50 Mbps begins to grow, peaking just over 200 Mbps around 03:00 UTC. Around that same time, measured median latency also begins to drop, falling from around 30 ms to a low of 13 ms, also around 03:00 UTC. Median bandwidth returned to normal levels around 06:00 UTC, while latency took about two hours longer to return to normal levels.  These observed  improvements in bandwidth and latency could have been due to traffic being re-routed to along paths with better connectivity to measurement endpoints, but because the shifts began before the cable damage occurred, and recovered shortly thereafter, that is unlikely to be the root cause.

In Germany, a brief minor increase in median bandwidth peaked around 02:45 UTC, while no notable changes were observed in latency.

From a routing perspective, there was no notable BGP announcement activity observed for top autonomous systems in either Finland or Germany around 02:00 on November 18, and total announced IP address space aggregated at a country level also demonstrated no change.

Telegeography’s submarinecablemap.com shows that both Finland and Germany also have significant redundancy and resilience from a submarine cable perspective, with over 10 cables landing in Finland, and nearly 10 landing in Germany, including Atlantic Crossing-1 (AC-1), which connects to the United States over two distinct paths. Terrestrial fiber maps from Arelion and eunetworks (as just two examples) show multiple redundant fiber routes within both countries, as well as cross-border routes to other neighboring countries, enabling more resilient Internet connectivity.

As we have discussed in multiple prior blog posts (Jersey, 2016; AAE-1/SMW5, 2022; WACS/MainOne/SAT3/ACE, 2024; EASSy/Seacom, 2024), cable cuts often cause significant disruptions to Internet connectivity, in many cases because they represent a concentrated point of vulnerability, whether for an individual network provider, city/state, or country. These disruptions are often quite lengthy as well, due to the time needed to marshal repair resources, identify the location of the damage, etc. Although it is not always feasible due to financial or geographic constraints, building redundant and resilient network architecture, at multiple levels, is a best practice. This includes the sending traffic over multiple physical cables (both submarine and terrestrial), connecting to multiple peer and upstream network providers, and even avoiding single points of failure in core Internet resources like DNS servers.

The Cloudflare Radar team continually monitors the status of Internet connectivity in countries/regions around the world, and we share our observations on the Cloudflare Radar Outage Center, via social media, and in posts on blog.cloudflare.com. Follow us on social media at @CloudflareRadar (X), https://noc.social/@cloudflareradar (Mastodon), and radar.cloudflare.com (Bluesky), or contact us via email.

We observed traffic starting to drop at 13:21 UTC, just ahead of the reported start time. By 13:28 UTC, it was approximately 95% lower than pre-incident levels. Recovery appeared to start at 13:31 UTC, and by 13:40 UTC, the reported end time of the incident, it had reached approximately 50% of pre-incident levels.

^{Traffic from OVHcloud (AS16276) to Cloudflare}

Cloudflare generally exchanges most of our traffic with OVHcloud over peering links. However, as shown below, peered traffic volume during the incident fell significantly. It appears that some small amount of traffic briefly began to flow over transit links from Cloudflare to OVHcloud due to sudden changes in which Cloudflare data centers we were receiving OVHcloud requests. (Peering is a direct connection between two network providers for the purpose of exchanging traffic. Transit is when one network pays an intermediary network to carry traffic to the destination network.) 

Because we peer directly, we exchange most traffic over our private peering sessions with OVHcloud. Instead, we found OVHcloud routing to Cloudflare dropped entirely for a few minutes, then switched to just a single Internet Exchange port in Amsterdam, and finally normalized globally minutes later.

As the graphs below illustrate, we normally see the largest amount of traffic from OVHcloud in our Frankfurt and Paris data centers, as OVHcloud has large data center presences in these regions. However, in that shift to transit, and the shift to an Amsterdam Internet Exchange peering point, we saw a spike in traffic routed to our Amsterdam data center. We suspect the routing shifts are the earliest signs of either internal BGP reconvergence, or general network recovery within AS16276, starting with their presence nearest our Amsterdam peering point.

The postmortem published by OVHcloud noted that the incident was caused by “an issue in a network configuration mistakenly pushed by one of our peering partner[s]” and that “We immediately reconfigured our network routes to restore traffic.” One possible explanation for the backbone incident may be a BGP route leak by the mentioned peering partner, where OVHcloud could have accepted a full Internet table from the peer and therefore overwhelmed their network or the peering partner’s network with traffic, or caused unexpected internal BGP route updates within AS16276.

Upon investigating what route leak may have caused this incident impacting OVHcloud, we found evidence of a maximum prefix-limit threshold being breached on our peering with Worldstream (AS49981) in Amsterdam. 

Oct 30 13:16:53  edge02.ams01 rpd[9669]: RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer 141.101.65.53 (External AS 49981) changed state from Established to Idle (event PrefixLimitExceeded) (instance master)

As the number of received prefixes exceeded the limits configured for our peering session with Worldstream, the BGP session automatically entered an idle state. This prevented the route leak from impacting Cloudflare’s network. In analyzing BGP Monitoring Protocol (BMP) data from AS49981 prior to the automatic session shutdown, we were able to confirm Worldstream was sending advertisements with AS paths that contained their upstream Tier 1 transit provider.

During this time, we also detected over 500,000 BGP announcements from AS49981, as Worldstream was announcing routes to many of their peers, visible on Cloudflare Radar.

Worldstream later posted a notice on their status page, indicating that their network experienced a route leak, causing routes to be unintentionally advertised to all peers:

“Due to a configuration error on one of the core routers, all routes were briefly announced to all our peers. As a result, we pulled in more traffic than expected, leading to congestion on some paths. To address this, we temporarily shut down these BGP sessions to locate the issue and stabilize the network. We are sorry for the inconvenience.”

We believe Worldstream also leaked routes on an OVHcloud peering session in Amsterdam, which caused today’s impact.

Cloudflare has written about impactful route leaks before, and there are multiple methods available to prevent BGP route leaks from impacting your network. One is setting max prefix-limits for a peer, so the BGP session is automatically torn down when a peer sends more prefixes than they are expected to. Other forward-looking measures are Autonomous System Provider Authorization (ASPA) for BGP, where Resource Public Key Infrastructure (RPKI) helps protect a network from accepting BGP routes with an invalid AS path, or RFC9234, which prevents leaks by tying strict customer, peer, and provider relationships to BGP updates. For improved Internet resilience, we recommend that network operators follow recommendations defined within MANRS for Network Operators.

As we have noted in the past, this post is intended as a summary overview of observed and confirmed disruptions, and is not an exhaustive or complete list of issues that have occurred during the quarter. 

A larger list of detected traffic anomalies is available in the Cloudflare Radar Outage Center.

Having said that, the third quarter of 2024 was particularly active, with quite a few significant Internet disruptions. Unfortunately, governments continued to impose nationwide Internet shutdowns intended to prevent cheating on exams. Damage to both terrestrial and submarine cables impacted Internet connectivity across Africa and in other parts of the world. Damage caused by an active hurricane season caused Internet outages across the Caribbean and in multiple parts of the United States. Because Internet connectivity is dependent on reliable electrical power, both planned and unplanned power outages in South America and Africa resulted in multi-hour Internet disruptions. Military action continued to cause Internet outages in affected countries, as did infrastructure maintenance, fire, and a purported cyberattack. The quarter also saw several noteworthy Internet disruptions that did not have verified causes.

Over the past several years, we have seen multiple governments around the world implement Internet shutdowns in response to protests within their countries. Some shutdowns are more targeted, affecting only (a subset of) mobile Internet providers, while others are more aggressive, effectively cutting off Internet connectivity at a national level. In addition, we all too frequently see governments implement nationwide multi-hour Internet shutdowns in an effort to prevent students from cheating on national exams. Unfortunately, governments were active in both respects during the third quarter, as we observed multiple government directed Internet shutdowns. Several were covered in our August 1 blog post, A recent spate of Internet disruptions.

Violent student protests in Bangladesh against quotas in government jobs and rising unemployment rates led the government to order the nationwide shutdown of mobile Internet connectivity on July 18, reportedly to “ensure the security of citizens.” This government-directed shutdown ultimately became a near-complete Internet outage for the country, as broadband networks were taken offline as well. At a country level, Internet traffic in Bangladesh dropped to near zero just before 21:00 local time (15:00 UTC). Announced IP address space from the country dropped to near zero at that time as well, meaning that nearly every network in the country was disconnected from the Internet.

Traffic and announced IP address space at a national level began to recover around 18:00 local time (12:00 UTC) on July 23, and continued over the next several days, as fixed broadband connectivity was restored, with mobile connectivity returning on July 28. The initial restoration was characterized as a “trial run”, prioritizing banking, commercial sectors, technology firms, exporters, outsourcing providers and media outlets, according to the state minister for post, telecommunication and information technology.

Ahead of this nationwide shutdown, we observed outages across several Bangladeshi network providers, perhaps foreshadowing what was to come. At AS24389 (Grameenphone), a complete Internet outage started at 01:30 local time on July 18 (19:30 UTC on July 17), with a total loss of both Internet traffic and announced IP address space.

The outage at AS25245 (Banglalink) started at 02:15 local time on July 18 (20:15 UTC on July 17) as both Internet traffic and announced IP address space dropped to zero.

At AS24432 (Robi Axiata), an Internet outage was observed starting around 06:30 local time on July 18 (00:30 UTC), with both Internet traffic and announced IP address space disappearing at that time.

Internet traffic at AS58715 (Earth Telecommunication) began to fall at 18:00 local time on July 18 (12:00 UTC), reaching zero four hours later. Announced IP address space began to fall at 21:00 local time (15:00 UTC), and was completely gone by 21:25 local time (15:25 UTC).

AS63526 (Carnival Internet) was one of the last to fall before the complete shutdown, losing traffic at 20:45 local time (14:45 UTC), and seeing all of its announced IP address space withdrawn over the following hour.

These mobile connectivity outages lasted from July 18 through July 28. Just a few days after connectivity was restored, additional clashes between police and protestors drove the government to order mobile Internet connectivity to be shut down again. As shown in the graphs below, traffic on these mobile network providers dropped between 13:30 and 14:15 local time (07:30 to 08:15 UTC) on Sunday, August 4.

These protests ultimately led the government to order a full Internet shutdown in the country, with both traffic and announced IP address space dropping precipitously around 10:30 local time (04:30 UTC) on Monday, August 5. However, the shutdown appeared to be short-lived, as broadband connectivity began to recover around 13:20 local time (07:20 UTC), with mobile connectivity being restored around 14:00 local time (08:00 UTC).

Both Iraq and Iraqi Kurdistan (the autonomous Kurdistan region in the northern part of the country) regularly implement government directed Internet shutdowns to prevent cheating on secondary and baccalaureate exams. Within Iraqi Kurdistan, we observed two sets of exam-related Internet shutdowns during the third quarter. The impacts of the shutdowns are visible on traffic from networks that operate within the region, as well as on the country-level graphs for Iraq.

The first round of shutdowns occurred in July, impacting AS59625 (KorekTel), AS21277 (Newroz Telecom), AS48492 (IQ Online), and AS206206 (KNET) between 06:00 - 08:00 local time (03:00 - 05:00 UTC) on July 3, 7, 10, and 14. This is consistent with shutdowns observed in the second quarter, as well as in June 2023. None of the impacted networks experienced a drop in announced IP address space during these shutdowns.

The second set of shutdowns in Iraqi Kurdistan took place across multiple days during the back half of August. On August 17, 19, 21, 24, 26, 28, and 31, all four network providers were again impacted, as seen in the graphs below, with traffic dropping between 06:00 - 08:00 local time (03:00 - 05:00 UTC).

In Iraq, a second round of exams for 12th graders resulted in over two weeks of regular Internet shutdowns across the country occurring between 06:00 - 08:00 local time (03:00 - 05:00 UTC) on multiple days between August 29 and September 16, intended to prevent cheating on second ministerial exams for secondary education. Both HTTP traffic and announced IP address space from Iraq dropped during these shutdowns, as seen in the graphs below.

(Note that the red annotation bar visible on September 11 & 12 on both the country and network-level graphs below highlights an internal data pipeline issue, and is not associated with an Internet shutdown in Iraq.)

This round of government-directed shutdowns impacted multiple local network providers, including AS58322 (Halasat), AS51684 (AsiaCell), AS203214 (HulumTele), AS199739 (Earthlink), and AS59588 (ZAINAS). In reviewing the distribution of mobile device and desktop traffic at a network level, gaps were observed during the shutdowns on AS58322 and AS199739, and to a lesser extent, AS203214, suggesting that these networks were completely offline, while AS56184 and AS59588 remained at least partially online. (This is also corroborated by complete or partial loss of announced IP address space across these networks during the shutdowns.)

A first round of exam-related Internet shutdowns took place in Syria earlier this year, between May 26 and June 13, and were discussed in our Exam-ining recent Internet shutdowns in Syria, Iraq, and Algeria blog post. A second set of exams, and the associated Internet shutdowns requested by the Ministry of Education, began on July 25 and ran through August 8, as specified in the schedule published by Syrian Telecom on its Facebook page.

The length of the shutdowns varied by day — they all began at 07:00 local time (04:00 UTC), but the end times ranged between 09:45 -10:30 local time (06:45 - 07:30 UTC). The graphs below show the impact at a country level, as well as to AS29256 (Syrian Telecom), the primary telecommunications provider within the country.

These shutdowns were also covered in our August 1 blog post, A recent spate of Internet disruptions.

On August 12, a round of baccalaureate exams began in Mauritania, and in an effort to prevent student cheating on the exams, the government instituted multiple Internet shutdowns that impacted several major mobile providers. Two shutdowns were observed on August 12, between 08:00 - 12:00 local time (08:00 - 12:00 UTC) and between 15:00 - 19:00 local time (15:00 - 19:00 UTC), and an additional one was observed on August 13, between 08:00 - 12:30 local time (08:00 - 12:30 UTC). Impacted network providers included AS37508 (Mattel), AS37541 (Chinguitel), and AS29544 (Mauritel). Announced IP address space for these networks remained unchanged during the shutdown periods, suggesting that that mobile subscriber connectivity was disabled, as opposed to the networks effectively being disconnected from the Internet, as we have seen in other countries.

Exam-related Internet shutdowns are, unfortunately, not new to Mauritania, as authorities in the country also implemented them between 2017 and 2020.

On July 14, MTN Eswatini (AS327765) informed customers via a post on X that “connection to the internet and data services is currently intermittent, because of fiber cable breaks resulting from wildfires.” This apparent connection disruption was visible in Cloudflare Radar between 19:30 and 20:15 local time (17:30 and 18:15 UTC).

In Cameroon, a fiber cut that occurred on August 4 during sanitation work disrupted mobile connectivity for Cameroon Telecommunications (AS15964 (Camtel)) customers for over half a day. According to a (translated) post on X from Camtel, “We inform you that due to the sanitation work carried out in the city of Yaoundé, at the place called Cradat, our Voice and Data services have been temporarily interrupted on the entire mobile network.” The observed disruption occurred between 03:00 - 16:30 local time (02:00 - 15:30 UTC). Although it initially started during a time when traffic was lower overnight anyway, both request and bytes traffic remained lower than the same time a week prior during the duration of the disruption.

The Liberia Telecommunications Authority posted an announcement to their Facebook page on August 21 noting that “We have been informed by the CCL that the ACE Cable is experiencing interruptions.” (The Africa Coast to Europe (ACE) submarine cable connects multiple countries along the West Coast of Africa to Portugal and Europe.) The announcement further noted that the first signs of interruption occurred at 01:00 local time (and UTC), and that Lonestar Cell MTN (AS37410) was among the providers that had been “gravely affected” by the cut.

We observed traffic on Lonestar Cell MTN dropping just after 01:00, in line with the announcement. The network experienced a complete outage lasting over a day and a half, before traffic started to recover at 14:00 local time (and UTC) on August 22. In a Facebook post on August 22, Lonestar Cell MTN confirmed that Internet service had been restored, and that customer accounts would be credited with 500 MB of data for free.

A September 7 post on X from Airtel Niger alerted customers to Internet service disruptions caused by cuts on international fiber optic cables. As a land-locked country, Niger is dependent on terrestrial connections to networks in neighboring countries, but it isn’t clear which connection or country Airtel Niger’s post was referencing.

Two significant Internet disruptions were observed around the time of Airtel Niger’s post that we believe are related to the referenced fiber cuts. The first occurred between 18:00 - 21:00 local time (17:00 - 20:00 UTC) on September 6, visible at a country level and at a network level as well on AS37531 (Airtel Niger) and AS37233 (Orange Niger / Zamani Telecom). The second disruption occurred between 10:45 - 12:00 local time (09:45 - 11:00 UTC) on September 7, visible at a country level as well as on those two networks. 

Internet disruptions related to submarine cable failures often take a significant amount of time to resolve because of the challenges repair crews face in getting to, and accessing, the damaged portion of the cable, as it is frequently located deep underwater in the middle of an ocean. A September 14 submarine cable failure that impacted Digicel Haiti (AS27653) lasted for over a week for a similar, but slightly different, reason.

A significant loss of traffic on Digicel Haiti was first observed at 08:00 local time (12:00 UTC) on September 14. On September 16, Digicel Haiti posted a press release confirming that since September 14, a failure had been detected on an international submarine cable belonging to Cable and Wireless, and that the cable damage occurred at Kaliko Beach Club (the property is reportedly used as a cable entry point). Digicel noted that their technicians went to the scene of the damage immediately, but were denied access, apparently because of a business dispute dating back to 2021. The release also explained that technical teams had taken temporary steps to ensure the continuity of essential services, which prevented the incident from resulting in a complete loss of connectivity. On September 22, a subsequent press release posted by Digicel Haiti announced the restoration of Internet services as of 02:00 local time (06:00 UTC), and referenced vandalism as the cause of the cable damage.

Reported damage to the “backbone wire” or “main cable” of an upstream provider resulted in a brief Internet outage for Kyrgyzstan Internet provider Megacom (AS50223) of September 25. AS12389 (Rostelecom) is listed as Megacom’s only upstream provider.

The outage lasted for only an hour, between 15:45 and 16:45 local time (09:45 - 10:45 UTC), dropping both traffic and announced IP address space to zero. At a country level, traffic dropped as much as 72% as compared to the previous week. Given the complete loss of both traffic and IP address space, the damage likely occurred on the connection between Megacom and Rostelecom.

An active hurricane season during July, August, and September resulted in infrastructure damage caused by multiple hurricanes disrupting Internet connectivity in multiple places across the Caribbean and Southeastern United States.

At the start of the third quarter, Grenada and Saint Vincent and the Grenadines both suffered significant damage from Hurricane Beryl, reportedly causing destruction of infrastructure, buildings, agriculture, and the natural environment.

On July 1, traffic from Grenada dropped significantly at 10:00 local time (14:00 UTC), just ahead of landfall on Grenada’s Carriacou Island. The most significant impacts to traffic were seen for approximately the first 24 hours, though traffic did not return to expected pre-storm levels until around 10:00 local time (14:00 UTC) on July 5.

Internet traffic in Saint Vincent and the Grenadines was also disrupted by Hurricane Beryl, also falling at 10:00 local time (14:00 UTC). Similar to Grenada, the most significant impact was seen in the first 24 hours, with consistent gradual recovery seen after that time. However, traffic did not return to expected pre-storm levels until July 11.

As Hurricane Beryl continued across the Caribbean, it passed Jamaica on July 3. The associated damage that it caused impacted Internet connectivity on the island, with traffic dropping significantly around 14:00 local time (19:00 UTC). As the graph below shows, the disruption was preceded by higher than normal traffic volumes, presumably due to residents looking for information about Beryl. The disruption lasted nearly a week, with traffic returning to expected levels on July 10.

The following month, damage from Tropical Storm Ernesto caused power outages across the U.S. Virgin Islands, resulting in disruptions to Internet connectivity. Traffic from the islands dropped precipitously at 22:00 local time on August 13 (02:00 UTC on August 14) and remained lower for over two days, before returning to expected pre-storm levels around 11:00 local time (15:00 UTC) on August 16.

Over the course of the following few days, Ernesto strengthened from a tropical storm into a hurricane, but had weakened by the time it hit Bermuda on August 16/17. In this case, damage was reportedly limited to power outages, downed trees, and flooding, but even this limited damage disrupted Internet connectivity on the island. As the storm made landfall on the island, traffic levels dropped over 80% at 22:00 local time on August 16 (01:00 UTC on August 17). Traffic levels remained depressed for about two and a half days, recovering to expected levels around 09:00 local time (12:00 UTC) on August 19.

Heavy rains in Nepal at the end of September resulted in flooding and landslides across much of the country, which in turn resulted in power outages and Internet disruptions. One such disruption believed to be associated with the impacts of the storm was observed on September 28, when AS23752 (Nepal Telecom), AS45650 (Vianet), AS139922 (Dishhome), and AS17501 (Worldlink) all saw traffic drop 50 - 70% between 14:15 - 16:00 local time (08:30 - 10:15 UTC).

A disruption to traffic from AS11427 (Charter Communications/Spectrum) in Texas that occurred between 12:30 and 19:30 local time on July 9 (17:30 - 00:30 UTC) was caused by “a third-party infrastructure issue caused by the impact of Hurricane Beryl”, according to a July 9 post on X from the provider. Spectrum acknowledged the issue shortly after it began, and followed up again after service had been restored.

Hurricane Helene made landfall in northern Florida as a Category 4 storm late in the evening (local time) on September 26, and over the following hours and days, continued north through Georgia, South Carolina, and North Carolina, and into Tennessee. Even as it weakened, it caused historic flooding and damage to roads, homes, power lines, and telecommunications infrastructure. Below, we review the traffic impacts observed at a state level in three of the most impacted states, as well as exploring the impact at a network level for selected providers. (Doug Madory at Kentik published an excellent blog post exploring the impact of Helene from the perspective of their data, and the networks referenced below were informed by that post.)

Helene entered Georgia early morning on Friday, September 27, and by midday (local time), peak traffic was approximately 20% lower than peak levels seen in the days ahead of the storm. (The lower peaks on September 28 & 29 are likely due to it being a weekend.) At a state level, peak traffic remained lower over the following week, with more recovery seen heading into the week of October 6.

One of the most significantly impacted network providers in Georgia was AS11240 (ATC Broadband), which saw traffic start to drop around 22:00 local time on September 26 (02:00 UTC on September 27). Subscribers and customers experienced a near complete outage until around 08:00 local time on September 30 (12:00 UTC), when traffic volumes slowly started to recover. The normal diurnal traffic pattern became more clear in the following days, with peak traffic levels continuing to increase over the next week as well.

Other network providers in Georgia that experienced significant impacts include AS400511 (Clearwave Fiber), AS394473 (Brantley Telephone Company), AS40285 (Northland Cable Television), AS15313 (Pembroke Telephone Company), and AS397118 (Glenwood Telephone Company).

The midday traffic peak on September 27 in South Carolina was just 65% of the preceding days, with the peaks remaining lower over the following two weekend days. Traffic remained somewhat lower during the week following Helene, with peak increases becoming more evident the week of October 6.

At AS19212 (Piedmont Rural Telephone) in South Carolina, traffic began to fall rapidly around midnight local time on September 27 (04:00 UTC), reaching a state of near complete outage over the next eight hours. A gradual recovery is visible over the following several days, with a more regular pattern becoming evident on October 1, with rapid growth over the following week, accelerating towards the end of the week.

Other network providers in South Carolina, including AS397068 (Carolina Connect), AS10279 (West Carolina Communications), AS20222 & AS21898 (TruVista), and AS14615 (Rock Hill Telephone), also experienced significant disruptions to connectivity in the wake of Helene.

Although a drop in traffic is visible in the graph for North Carolina on September 27, it occurs after a midday peak in line with previous days, and the magnitude is not as significant as that seen in South Carolina and Georgia. Traffic peaks over the following week are in line with the week preceding Helene’s arrival, with higher peaks seen the week of October 6.

North Carolina providers AS53488 (Morris Broadband) and AS53274 (Skyrunner) both experienced multi-day disruptions, likely related to damage from Helene. However, these disruptions took Morris Broadband completely offline several times over the course of a week — the announced IP address space graph below shows three distinct drops to zero, aligning with outages visible in the traffic graph, when the network was effectively disconnected from the Internet. A similar but less severe pattern was seen at Skyrunner, which lost 75-80% of announced IP address space for a two-day period covering September 27-29, aligning with an outage visible in the associated traffic graph.

Other impacted network providers in North Carolina included AS22191 (Wilkes Communications) and AS23118 (Skyline Telephone).

A nationwide power outage in Venezuela on August 30 was, according to President Nicolás Maduro, the result of an attack on the Guri Reservoir, Venezuela's largest hydroelectric project. A published report indicated that all 24 of the country's states reported a total or partial loss of electricity supply. The loss of power unsurprisingly caused an Internet disruption, with country-level traffic dropping 82%, starting around 04:45 local time (08:45 UTC). Traffic began to increase as electricity returned to various parts of the country throughout the day, and returned to expected levels just after midnight local time on August 31 (04:00 UTC). 

On August 30, Kenya Power Care posted a Customer Alert on its Facebook page, issued at 21:57 local time (18:57 UTC), stating that “We have lost power supply to various parts of the country except North Rift region and sections of Western region.” Approximately a half hour before that alert, Kenya’s Internet traffic began to drop, falling as much as 61%. Just two hours later, Kenya Power Care posted a follow up, stating “Following the partial outage affecting several parts of the country this evening, we are pleased to report that power supply has now been restored to the entire Western region, as well as parts of Central Rift, South Nyanza, and Nairobi regions.” However, traffic did not return to expected levels for several more hours, taking until 06:00 local time (03:00 UTC).

A week later, on September 6, Kenya Power Care posted another similar Customer Alert, noting that “We are experiencing a power outage affecting several parts of the country, except sections of North Rift and Western regions.” This alert was issued at 09:20 local time (06:20 UTC), and follows a drop in Internet traffic that started around 09:00 local time (06:00 UTC). Traffic dropped approximately 45% during this power outage, and returned to expected levels around 16:00 local time (13:00 UTC). Traffic recovery aligns with a subsequent Customer Alert posted on Facebook, where Kenya Power Care stated “We are glad to report that normal electricity supply was restored across the country as at 3:49pm”.

A statement from Energy and Petroleum Cabinet Secretary Opiyo Wandayi, shared on Facebook by Kenya Power Care, explained the cause of the power outage: “Today, Friday 6th September 2024 at 8.56 am, the 220kV High Voltage Loiyangalani transmission line tripped at Suswa substation while evacuating 288MW from Lake Turkana Wind Power (LTWP) plant. This was followed by a trip on the Ethiopia – Kenya 500kV DC interconnector that was then carrying 200MW, resulting to a total loss of 488MW…” 

According to a (translated) September 7 post on X from CENACE, the national electricity operator in Ecuador, “We inform the public that due to a fault in the Molino substation bar, which is connected to the Paute generation, there has been a power outage in some provinces of the country. Cenace's technical team, in coordination with the distribution companies, is working to gradually restore electrical service. It is estimated that it will take 3 to 4 hours maximum for the supply to return to normal.” The post was published at 09:53 local time (14:53 UTC), approximately an hour after Internet traffic from the country began to drop. Traffic returned to expected levels just under four hours later, at around 12:30 local time (17:30 UTC), in line with CENACE’s predicted time for power to be fully restored.

On September 18/19, the first of several planned nightly power outages to enable needed grid maintenance in Ecuador disrupted Internet connectivity. Traffic dropped by over 60% as compared to the same time the prior week starting around 21:30 local (02:30 UTC), with the power outages reportedly scheduled for 22:00 - 06:00 local time. Internet traffic recovered to expected levels around 06:00 local time (11:00 UTC) as power was restored. Similar power cuts were reportedly planned from September 23 to September 27, but these power outages did not appear to impact traffic levels in Ecuador as compared to the previous week. 

Senegal’s power company, Senelec, posted a communiqué on X on September 12 that stated (translated) “Senelec informs its valued customers that an incident that occurred this morning at the Hann substation resulted in the loss of the OMVS interconnected network and disruptions to electricity distribution.” This disruption to electricity distribution also resulted in a disruption to Internet traffic, which dropped sharply at 13:00 local time (13:00 UTC), falling as much as 80%. Traffic recovered to expected levels by 20:00 local time (20:00 UTC) around the same time that Senelec posted a followup about the incident that stated (translated) “Effective restoration of electricity supply in all localities.”

As we discussed above, Internet users in Syria were impacted by an exam-related Internet shutdown from 07:00 - 10:15 local time (04:00 - 07:15 UTC) on July 30. However, just an hour after connectivity was restored, another disruption occurred, as seen in both the traffic and announced IP address space graphs below. According to a (translated) Facebook post from Syrian Telecom, “...during the periodic maintenance of one of the air conditioners in one of the technical halls, an explosion occurred, which caused the internet circuits to be temporarily out of service.” Traffic remained depressed for approximately eight hours, recovering to expected levels around 19:00 local time (16:00 UTC).

Roskomnadzor, Russia’s Internet regulate, blamed a brief disruption in traffic observed in Russia and on AS12389 (Rostelecom) on August 21 on a distributed denial-of-service (DDoS) attack that targeted Russian telecommunications operators. The disruption was brief, lasting from around 13:45 until 14:30 Moscow time (10:45 - 11:30 UTC). Roskomnadzor subsequently stated "As of 3 PM Moscow time, the attack has been repelled, and services are operating normally." The disruption reportedly impacted messaging services Telegram and WhatsApp, as well as Wikipedia, Yandex, VKontakte, telecom support services, and mobile banking apps. Some experts questioned the official explanation, suggesting instead that the disruption was due to centralized interference from Roskomnadzor.

We have covered Internet disruptions related to the ongoing conflict in Gaza multiple times since October 2023, both on Cloudflare Radar’s presence on X, and on the Cloudflare blog (1, 2, 3). In many of these cases, Paltel (AS12975) has posted notices on social media regarding service disruptions and outages. On September 8, Paltel posted a message on its Facebook page, stating (translated) “We regret to announce the suspension of home internet services in the central and southern areas of the Gaza Strip, due to the ongoing aggression.”

Within the Gaza, Rafah, Deir al-Balah Governorates, we observed a sharp drop in traffic at 18:00 local time (16:00 UTC). The impact appeared to be most significant in Rafah and Deir al-Balah. Traffic returned to expected levels around 23:00 local time (21:00 UTC), and Paltel confirmed the service restoration in a subsequent Facebook post, stating (translated) “We would like to announce the return of home Internet services in central and southern Gaza Strip to the way it was before it was interrupted hours ago.”

Israeli airstrikes targeting the Lebanese capital of Beirut on September 28 likely knocked local network provider Solidere (AS42852) offline for several hours. The graph below shows a loss of traffic starting around 12:15 local time (10:15 UTC), at the same time a complete loss of announced IP address space occurred. Most of Solidere’s IP address space started to get announced again at 14:45 local time (12:45 UTC), and a slight increase in traffic was seen at that time as well. Traffic levels fully recovered just after 18:00 local time (16:00 UTC), and announced IP address space had stabilized by that time as well. 

A fire near a data center in Blida Province, Algeria disrupted connectivity on AS327931 (Djezzy) at 13:00 and local time (12:00 UTC) on July 24. According to a (translated) X post from Djezzy, “Djezzy announced fluctuations in its services in some areas of the country, as it was a victim of a fire that broke out on Wednesday, July 24, 2024, in a warehouse of one of the companies located near its technical center in the state of Blida.” The post from Djezzy predicted that “97% of the sites will be restored by around 3 am [July 25]”, but traffic did not return to expected levels until the end of the day on July 25.

On Monday, September 30, customers on Verizon’s mobile network in multiple cities across the United States reported experiencing a loss of connectivity. Impacted phones showed “SOS” instead of the usual bar-based signal strength indicator, and customers complained of an inability to make or receive calls on their mobile devices. Although initial reports of connectivity problems started around 09:00 ET (13:00 UTC), we didn’t see a noticeable change in request volume at an ASN level until about two hours later. AS6167 (CELLCO) is the autonomous system used by Verizon for its mobile network.

Just before 12:00 ET (16:00 UTC), Verizon published a social media post acknowledging the problem, stating “We are aware of an issue impacting service for some customers. Our engineers are engaged, and we are working quickly to identify and solve the issue.” As the graph below shows, a slight decline (-5%) in HTTP traffic as compared to traffic at the same time a week prior is first visible around 11:00 ET (15:00 UTC), and request volume fell as much as 9% below expected levels at 13:45 ET (17:45 UTC).

Media reports listed cities including Chicago, Indianapolis, New York City, Atlanta, Cincinnati, Omaha, Phoenix, Denver, Minneapolis, Seattle, Los Angeles, and Las Vegas as being most impacted. Traffic graphs illustrating the impacts seen in these cities can be found in our Impact of Verizon’s September 30 outage on Internet traffic blog post.

Traffic appeared to return to expected levels around 17:15 ET (21:15 UTC). At 19:18 ET (23:18 UTC), a social media post from Verizon noted “Verizon engineers have fully restored today's network disruption that impacted some customers. Service has returned to normal levels.”

On July 31, Pakistan experienced a wide-scale Internet disruption that lasted approximately two hours, between 13:30 - 15:30 local time (08:30 - 10:30 UTC). Traffic only dropped ~45% at a country level, but AS17557 (PTCL) experienced a near complete loss of traffic, while traffic at AS24499 (Telenor Pakistan) dropped nearly 90%. Together, the two network providers serve an estimated nine million users, and are among the top five Internet service providers in the country.

The actual cause of the disruption is disputed. It was reported that the Pakistan Telecommunication Authority (PTA) attributed the disruptions to a technical glitch in the international submarine cable affecting the Pakistan Telecommunication Company Limited (PTCL) network. However, another published report noted “According to our sources, the government’s latest firewall edition to block the content was misconfigured, resulting in Internet connectivity disruption.” Additional details can be found in our August 1 blog post, A recent spate of Internet disruptions.

On August 14, subscribers of UK service provider Vodafone (AS25135) reported problems accessing both mobile and landline Internet connections. Starting around 11:00 local time (10:00 UTC), we observed traffic starting to drop, ultimately falling 43% below the same time the prior week. The disruption was fairly short-lived, as traffic returned to expected levels by 13:30 local time (12:30 UTC). Vodafone did not acknowledge the issue on social media, nor did it provide a public explanation for what caused the disruption.

Although Internet disruptions observed during the third quarter had a variety of underlying causes, those caused by power outages due to aging or insufficiently maintained electrical infrastructure are worth highlighting. Of course, widespread power outages always create a massive inconvenience for impacted populations, but over the last several years, as communication, entertainment, commerce, and more have become increasingly reliant on the Internet, the impact of these outages has become even more significant, because losing electrical power largely means losing Internet connectivity. Although mobile connectivity may still be available in some cases, it is decidedly not a complete replacement, not to mention that mobile devices will eventually need to be recharged. While addressing the underlying infrastructure issues require non-trivial amounts of time, resources, and money, governments appear to be taking steps towards doing so.

Visit Cloudflare Radar for additional insights around Internet disruptions, routing issues, Internet traffic trends, security and attacks, and Internet quality. Follow us on social media at @CloudflareRadar (X), noc.social/@cloudflareradar (Mastodon), and radar.cloudflare.com (Bluesky), or contact us via e-mail.

AS6167 (CELLCO) is the autonomous system used by Verizon for its mobile network. To better understand how the outage impacted Internet traffic on Verizon’s network, we took a look at HTTP request volume from AS6167 independent of geography, as well as traffic from AS6167 in various cities that were reported to be the most significantly impacted.

Although initial reports of connectivity problems started around 09:00 ET (13:00 UTC), we didn’t see a noticeable change in request volume at an ASN level until about two hours later. Just before 12:00 ET (16:00 UTC), Verizon published a social media post acknowledging the problem, stating “We are aware of an issue impacting service for some customers. Our engineers are engaged and we are working quickly to identify and solve the issue.”

As the Cloudflare Radar graph below shows, a slight decline (-5%) in HTTP traffic as compared to traffic at the same time a week prior is first visible around 11:00 ET (15:00 UTC). Request volume fell as much as 9% below expected levels at 13:45 ET (17:45 UTC).

Just after 17:00 ET (21:00 UTC), Verizon published a second social media post noting, in part, “Verizon engineers are making progress on our network issue and service has started to be restored.” Request volumes returned to expected levels around the same time, surpassing the previous week’s levels at 17:15 ET (21:15 UTC). At 19:18 ET (23:18 UTC), a social media post from Verizon noted “Verizon engineers have fully restored today's network disruption that impacted some customers. Service has returned to normal levels.”

Media reports listed cities including Chicago, Indianapolis, New York City, Atlanta, Cincinnati, Omaha, Phoenix, Denver, Minneapolis, Seattle, Los Angeles, and Las Vegas as being most impacted. In addition to looking at comparative traffic trends across the whole Verizon Wireless network, we also compared request volumes in the listed cities to the same time a week prior (September 23).

Declines in request traffic starting around 11:00 ET (15:00 UTC) are clearly visible in cities including Los Angeles, Seattle, Omaha, Denver, Phoenix, Minneapolis, Indianapolis, and Chicago. In contrast to other cities, Omaha’s request volume was already trending lower than last week heading into today’s outage, but its graph clearly shows the impact of today’s disruption as well. Omaha’s difference in traffic was the most significant, down approximately 30%, while other cities saw declines in the 10-20% range. 

Request traffic from Las Vegas initially appeared to exhibit a bit of volatility around 11:00 ET (15:00 UTC), but continues to track fairly closely to last week’s levels before exceeding them starting at 16:00 ET (20:00 UTC). Cincinnati was tracking slightly above last week’s request volume before the outage began, and tracked closely to the prior week during the outage period.

We observed week-over-week traffic increases during the outage period in New York and Atlanta. However, in both cities, traffic was already slightly above last week’s levels, and that trend continued throughout the day. 

Based on our observations, it appears that voice services on Verizon’s network may have been more significantly impacted than data services, as we saw some declines in request traffic across impacted cities, but none experienced full outages.

As of this writing (19:15 ET, 23:15 UTC), no specific information has been made available by Verizon regarding the root cause of the network problems.