Over the years when Cloudflare has had an outage that affected our customers we have very quickly blogged about what happened, why, and what we are doing to address the causes of the outage. Today’s post is a little different. It’s about a single customer’s website not working correctly because of incorrect action taken by Cloudflare.

Although the customer was not in any way banned from Cloudflare, or lost access to their account, their website didn’t work. And it didn’t work because Cloudflare applied a bandwidth throttle between us and their origin server. The effect was that the website was unusable.

Because of this unusual throttle there was some internal confusion for our customer support team about what had happened. They, incorrectly, believed that the customer had been limited because of a breach of section 2.8 of our Self-Serve Subscription Agreement which prohibits use of our self-service CDN to serve excessive non-HTML content, such as images and video, without a paid plan that includes those services (this is, for example, designed to prevent someone building an image-hosting service on Cloudflare and consuming a huge amount of bandwidth; for that sort of use case we have paid image and video plans).

However, this customer wasn’t breaking section 2.8, and they were both a paying customer and a paying customer of Cloudflare Workers through which the throttled traffic was passing. This throttle should not have happened. In addition, there is and was no need for the customer to upgrade to some other plan level.

This incident has set off a number of workstreams inside Cloudflare to ensure better communication between teams, prevent such an incident happening, and to ensure that communications between Cloudflare and our customers are much clearer.

Before we explain our own mistake and how it came to be, we’d like to apologize to the customer. We realize the serious impact this had, and how we fell short of expectations. In this blog post, we want to explain what happened, and more importantly what we’re going to change to make sure it does not happen again.

On February 2, an on-call network engineer received an alert for a congesting interface with Equinix IX in our Ashburn data center. While this is not an unusual alert, this one stood out for two reasons. First, it was the second day in a row that it happened, and second, the congestion was due to a sudden and extreme spike of traffic.

The engineer in charge identified the customer’s domain, tardis.dev, as being responsible for this sudden spike of traffic between Cloudflare and their origin network, a storage provider. Because this congestion happens on a physical interface connected to external peers, there was an immediate impact to many of our customers and peers. A port congestion like this one typically incurs packet loss, slow throughput and higher than usual latency. While we have automatic mitigation in place for congesting interfaces, in this case the mitigation was unable to resolve the impact completely.

The traffic from this customer went suddenly from an average of 1,500 requests per second, and a 0.5 MB payload per request, to 3,000 requests per second (2x) and more than 12 MB payload per request (25x).

The congestion happened between Cloudflare and the origin network. Caching did not happen because the requests were all unique URLs going to the origin, and therefore we had no ability to serve from cache.

A Cloudflare engineer decided to apply a throttling mechanism to prevent the zone from pulling so much traffic from their origin. Let's be very clear on this action: Cloudflare does not have an established process to throttle customers that consume large amounts of bandwidth, and does not intend to have one. This remediation was a mistake, it was not sanctioned, and we deeply regret it.

We lifted the throttle through internal escalation 12 hours and 53 minutes after having set it up.

To make sure a similar incident does not happen, we are establishing clear rules to mitigate issues like this one. Any action taken against a customer domain, paying or not, will require multiple levels of approval and clear communication to the customer. Our tooling will be improved to reflect this. We have many ways of traffic shaping in situations where a huge spike of traffic affects a link and could have applied a different mitigation in this instance.

We are in the process of rewriting our terms of service to better reflect the type of services that our customers deliver on our platform today. We are also committed to explaining to our users in plain language what is permitted under self-service plans. As a developer-first company with transparency as one of its core principles, we know we can do better here. We will follow up with a blog post dedicated to these changes later.

Once again, we apologize to the customer for this action and for the confusion it created for other Cloudflare customers.

This article will talk about our approach to network security using technologies like RPKI to sign Internet routes and protect our users and customers from route hijacks and misconfigurations. We are proud to announce we have started deploying active filtering by using RPKI for routing decisions and signing our routes.

Back in April, articles including our blog post on BGP and route-leaks were reported in the news, highlighting how IP addresses can be redirected maliciously or by mistake. While enormous, the underlying routing infrastructure, the bedrock of the Internet, has remained mostly unsecured.

At Cloudflare, we decided to secure our part of the Internet by protecting our customers and everyone using our services including our recursive resolver 1.1.1.1.

A prefix is a range of IP addresses, for instance, 10.0.0.0/24, whose first address is 10.0.0.0 and the last one is 10.0.0.255. A computer or a server usually have one. A router creates a list of reachable prefixes called a routing table and uses this routing table to transport packets from a source to a destination.  

On the Internet, network devices exchange routes via a protocol called BGP (Border Gateway Protocol). BGP enables a map of the interconnections on the Internet so that packets can be sent across different networks to reach their final destination. BGP binds the separate networks together into the Internet.

This dynamic protocol is also what makes Internet so resilient by providing multiple paths in case a router on the way fails. A BGP announcement is usually composed of a prefix which can be reached at a destination and was originated by an Autonomous System Number (ASN).

IP addresses and Autonomous Systems Numbers are allocated by five Regional Internet Registries (RIR): Afrinic for Africa, APNIC for Asia-Pacific, ARIN for North America, LACNIC for Central and South America and RIPE for Europe, Middle-East and Russia. Each one operates independently.

With more than 700,000 IPv4 routes and 40,000 IPv6 routes announced by all Internet actors, it is difficult to know who owns which resource.

There is no simple relationship between the entity that has a prefix assigned, the one that announces it with an ASN and the ones that receive or send packets with these IP addresses. An entity owning 10.0.0.0/8 may be delegating a subset 10.1.0.0/24 of that space to another operator while being announced through the AS of another entity.

Thereby, a route leak or a route hijack is defined as the illegitimate advertisement of an IP space. The term route hijack implies a malicious purpose while a route leak usually happens because of a misconfiguration.

A change in route will cause the traffic to be redirected via other networks. Unencrypted traffic can be read and modified. HTTP webpages and DNS without DNSSEC are sensitive to these exploits.

You can learn more about BGP Hijacking in our Learning Center.

When an illegitimate announcement is detected by a peer, they usually notify the origin and reconfigure their network to reject the invalid route.Unfortunately, the time to detect and act upon may take from a few minutes up to a few days, more than enough to steal cryptocurrencies, poison a DNS cache or make a website unavailable.

A few systems exist to document and prevent illegitimate BGP announcements.

The Internet Routing Registries (IRR) are semi-public databases used by network operators to register their assigned Internet resources. Some database maintainers do not check whether the entry was actually made by the owner, nor check if the prefix has been transferred to somebody else. This makes them prone to error and not completely reliable.

Resource Public Key Infrastructure (RPKI) is similar to the IRR “route” objects, but adding the authentication with cryptography.

Here’s how it works: each RIR has a root certificate. They can generate a signed certificate for a Local Internet Registry (LIR, a.k.a. a network operator) with all the resources they are assigned (IPs and ASNs). The LIR then signs the prefix containing the origin AS that they intend to use: a ROA (Route Object Authorization) is created. ROAs are just simple X.509 certificates.

If you are used to SSL/TLS certificates used by browsers to authenticate the holder of a website, then ROAs are the equivalent in the Internet routing world.

Each network operator owning and managing Internet resources (IP addresses, Autonomous System Numbers) has access to their Regional Internet Registry portal. Signing their prefixes through the portal or the API of their RIR is the easiest way to start with RPKI.

Because of our global presence, Cloudflare has resources in each of the 5 RIR regions. With more than 800 prefix announcements over different ASNs, the first step was to ensure the prefixes we were going to sign were correctly announced.

We started by signing our less used prefixes, checked if the traffic levels remained the same and then signed more prefixes. Today about 25% of Cloudflare prefixes are signed. This includes our critical DNS servers and our public 1.1.1.1 resolver.

Signing the prefixes is one thing. But ensuring that the prefixes we receive from our peers match their certificates is another.

The first part is validating the certificate chain. It is done by synchronizing the RIR databases of ROAs through rsync (although there are some new proposals regarding distribution over HTTPS), then check the signature of every ROA against the RIR’s certificate public key. Once the valid records are known, this information is sent to the routers.

Major vendors support a protocol called RPKI to Router Protocol (abbreviated as RTR). This is a simple protocol for passing a list of valid prefixes with their origin ASN and expected mask length. However, while the RFC defines 4 different secure transport methods, vendors have only implemented the insecure one. Routes sent in clear text over TCP can be tampered with.

With more than 150 routers over the globe, it would be unsafe to rely on these cleartext TCP sessions over the insecure and lossy Internet to our validator. We needed local distribution on a link we know secure and reliable.

One option we considered was to install an RPKI RTR server and a validator in each of our 150+ datacenters, but doing so would cause a significant increase in operational cost and reduce debugging capabilities.

We needed an easier way of passing an RPKI cache securely. After some system design sessions, we settled on distributing the list of valid routes from a central validator securely, distribute it via our own Content Delivery Network and use a lightweight local RTR server. This server fetches the cache file over HTTPS and passes the routes over RTR.

Rolling out this system on all our PoPs using automation was straightforward and we are progressively moving towards enforcing strict validation of RPKI signed routes everywhere.

To encourage adoption of Route Origin Validation on the Internet, we also want to provide this service to everyone, for free. You can already download our RTR server with the cache behind Cloudflare. Just configure your Juniper or Cisco router. And if you do not want to use our file of prefixes, it is compatible with the RIPE RPKI Validator Export format.

We are also working on providing a public RTR server using our own Spectrum service so that you will not have to install anything, just make sure you peer with us! Cloudflare is present on many Internet Exchange Points so we are one hop away from most routers.

A few months ago, Nick Sullivan introduced our new Nimbus Certificate Transparency Log.

In order to track the emitted certificates in the RPKI, our Crypto team created a new Certificate Transparency Log called Cirrus which includes the five RIRs root certificates as trust anchors. Certificate transparency is a great tool for detecting bad behavior in the RPKI because it keeps a permanent record of all valid certificates that are submitted to it in an append-only database that can’t be modified without detection. It also enables users to download the entire set of certificates via an HTTP API.

We use services like BGPmon and other public observation services extensively to ensure quick action if some of our prefixes are leaked. We also have internal BGP and BMP collectors, aggregating more than 60 millions routes and processing live updates.

Our filters make use of this live feed to ensure we are alerted when a suspicious route appears.

The latest statistics suggest that around 8.7% of the IPv4 Internet routes are signed with RPKI, but only 0.5% of all the networks apply strict RPKI validation.Even with RPKI validation enforced, a BGP actor could still impersonate your origin AS and advertise your BGP route through a malicious router configuration.

However that can be partially solved by denser interconnections, that Cloudflare already has through an extensive network of private and public interconnections.To be fully effective, RPKI must be deployed by multiple major network operators.

As said by Job Snijders from NTT Communications, who’s been at the forefront of the efforts of securing Internet routing:

If the Internet's major content providers use RPKI and validate routes, the impact of BGP attacks is greatly reduced because protected paths are formed back and forth. It'll only take a small specific group of densely connected organizations to positively impact the Internet experience for billions of end users.

RPKI is not a bullet-proof solution to securing all routing on the Internet, however it represents the first milestone in moving from trust based to authentication based routing. Our intention is to demonstrate that it can be done simply and cost efficiently. We are inviting operators of critical Internet infrastructure to follow us in a large scale deployment.

Subscribe to the blog for daily updates on our announcements.

As Irma is now forecast to hit southern Florida as category 5 this weekend with gusty winds reaching up to 155mph, it is also expected that Internet infrastructure in the region will suffer.

At the time of writing, we haven’t noticed any decrease in traffic in the region of Miami despite calls to evacuate.

Contrary to popular belief, Internet wasn't built for the purpose of resisting a nuclear attack. That doesn't mean that datacenters aren't built to resist catastrophic events.

The Miami datacenter housing servers for Cloudflare and other Internet operators is classified as Tier IV. What does this tiering mean? As defined by the ANSI (American National Standards Institute), a Tier IV datacenter is the stringent classification in term of redundancy of the critical components of a datacenter: power and cooling. It guarantees 99.995% uptime per year, that is only 26 minutes of unavailability. Tier IV datacenters provide this level of uptime by being connected to separate power grids, allowing their customers to connect their devices to both of these grids. They also provide fuel-powered backup generators, which can themselves be made redundant, for up to 96 hours of autonomy.

Data center facilities have already taken precautionary measures during these last days, one of them contacting their customers with the following (excerpt):

• The generators have been tested, fuel tanks are full, levels verified.

• Special arrangements have been made to make sure staff is available on site as necessary to maintain operability standards.

• We will secure hotel rooms near the sites, and have cots, MREs, and other emergency supplies on site should the situation become extreme.

Due to their importance in Internet infrastructure, Tier IV datacenters also have the most available connectivity to the Internet. This is the case for our Miami data center, which is connected to multiple Tier 1 transit providers and Internet Exchanges and will provide backup routes in case of an outage with a particular infrastructure.

As a last resort, in the event our Miami datacenter would be taken offline, our Anycast routing will smoothly reroute packets to our nearest data centers in the United States: Tampa, Atlanta and Ashburn (Washington DC).

Our technical teams will take all the necessary steps to ensure our services stay online during these unfortunate events. We’d like to remind our users to follow all precautions, and evacuate the regions as advised by the local authorities.

The strength of the Internet is its ability to interconnect all sorts of networks — big data centers, e-commerce websites at small hosting companies, Internet Service Providers (ISP), and Content Delivery Networks (CDN) — just to name a few. These networks are either interconnected with each other directly using a dedicated physical fiber cable, through a common interconnection platform called an Internet Exchange (IXP), or they can even talk to each other by simply being on the Internet connected through intermediaries called transit providers.

The Internet is like the network of roads across a country and navigating roads means answering questions like “How do I get from Atlanta to Boise?” The Internet equivalent of that question is asking how to reach one network from another. For example, as you are reading this on the Cloudflare blog, your web browser is connected to your ISP and packets from your computer found their way across the Internet to Cloudflare’s blog server.

Figuring out the route between networks is accomplished through a protocol designed 25 years ago (on two napkins) called BGP.

BGP allows interconnections between networks to change dynamically. It provides an administrative protocol to exchange routes between networks, and allows for withdrawals in the case that a path is no longer viable (when some route no longer works).

The Internet has become such a complex set of tangled fibers, neighboring routers, and millions of servers that you can be certain there is a server failing or a optical fibre being damaged at any moment. Whether it’s in a datacenter, a trench next to a railroad, or at the bottom of the ocean. The reality is that the Internet is in a constant state of flux as connections break and are fixed; it’s incredible strength is that it operates in the face of the real world where conditions constantly change.

While BGP is the cornerstone of Internet routing, it does not provide first class mechanisms to automatically deal with these events, nor does it provide tools to manage quality of service in general.

Although BGP is able to handle the coming and going of networks with grace, it wasn’t designed to deal with Internet brownouts. One common problem is that a connection enters a state where it hasn’t failed, but isn’t working correctly either. This usually presents itself as packet loss: packets enter a connection and never arrive at their destination. The only solution to these brownouts is active, continuous monitoring of the health of the Internet.

CC BY 2.0 image by Steve Jurvetson

Again, the metaphor of a system of roads is useful. A printed map may tell you the route from one city to another, but it won't tell you where there's a traffic jam. However, modern GPS applications such as Waze can tell you which roads are congested and which are clear. Similarly, Internet monitoring shows which parts of the Internet are blocked or losing packets and which are working well.

At Cloudflare we decided to deploy our own mechanisms to react to unpredictable events causing these brownouts. While most events do not fall under our jurisdiction — they are “external” to the Cloudflare network — we have to operate a reliable service by minimizing the impact of external events.

This is a journey of continual improvement, and it can be deconstructed into a few simple components:

• Building an exhaustive and consistent view of the quality of the Internet

• Building a detection and alerting mechanism on top of this view

• Building the automatic mitigation mechanisms to ensure the best reaction time

Having deployed our network in a hundred locations worldwide, we are in a unique position to monitor the quality of the Internet from a wide variety of locations. To do this, we are leveraging the probing capabilities of our network hardware and have added some extra tools that we’ve built.

By collecting data from thousands of automatically deployed probes, we have a real-time view of the Internet’s infrastructure: packet loss in any of our transit provider’s backbones, packet loss on Internet Exchanges, or packet loss between continents. It is salutary to watch this real-time view over time and realize how often parts of the Internet fail and how resilient the overall network is.

Our monitoring data is stored in real-time in our metrics pipeline powered by a mix of open-source software: ZeroMQ, Prometheus and OpenTSDB. The data can then be queried and filtered on a single dashboard to give us a clear view of the state of a specific transit provider, or one specific PoP.

Above we can see a time-lapse of a transit provider having some packet loss issues.

Here we see a transit provider having some trouble on the US West Coast on October 28, 2016.

We didn’t want to stop here. Having a real-time map of Internet quality puts us in a great position to detect problems and create alerts as they unfold. We have defined a set of triggers that we know are indicative of a network issue, which allow us to quickly analyze and repair problems.

For example, 3% packet loss from Latin America to Asia is expected under normal Internet conditions and not something that would trigger an alert. However, 3% packet loss between two countries in Europe usually indicates a bigger and potentially more impactful problem, and thus will immediately trigger alerts for our Systems Reliability Engineering and Network Engineering teams to look into the issue.

Sitting between eyeball networks and content networks, it is easy for us to correlate this packet loss with various other metrics in our system, such as difficulty connecting to customer origin servers (which manifest as Cloudflare error 522) or a sudden decrease of traffic from a local ISP.

Receiving valuable and actionable alerts is great, however we were still facing the hard to compress time-to-reaction factor. Thankfully in our early years, we’ve learned a lot from DDoS attacks. We’ve learned how to detect and auto-mitigate most attacks with our efficient automated DDoS mitigation pipeline. So naturally we wondered if we could apply what we’ve learned from DDoS mitigation to these generic internet events? After all, they do exhibit the same characteristics: they’re unpredictable, they’re external to our network, and they can impact our service.

The next step was to correlate these alerts with automated actions. The actions should reflect what an on-call network engineer would have done given the same information. This includes running some important checks: is the packet loss really external to our network? Is the packet loss correlated to an actual impact? Do we currently have enough capacity to reroute the traffic? When all the stars align, we know we have a case to perform some action.

All that said, automating actions on network devices turns out to be more complicated than one would imagine. Without going into too much detail, we struggled to find a common language to talk to our equipment with because we’re a multi-vendor network. We decided to contribute to the brilliant open-source project Napalm, coupled it with the automation framework Salt, and and improved it to bring us the features we needed.

We wanted to be able to perform actions such as configuring probes, retrieving their data, and managing complex BGP neighbor configuration regardless of the network device a given PoP was using. With all these features put together into an automated system, we can see the impact of actions it has taken:

Here you can see one of our transit provider having a sudden problem in Hong-Kong. Our system automatically detects the fault and takes the necessary action, which is to disable this link for our routing.

Our system keeps improving every day, but it is already running at a high pace and making immediate adjustments across our network to optimize performance every single day.

Here we can see actions taken during 90 days of our mitigation bot.

The impact of this is that we’ve managed to make the Internet perform better for our customers and reduce the number of errors that they'd see if they weren't using Cloudflare. One way to measure this is how often we're unable to reach a customer's origin. Sometimes origins are completely offline. However, we are increasingly at a point where if an origin is reachable we'll find a path to reach it. You can see the effects of our improvements over the last year in the graph below.

While we keep improving this resiliency pipeline every day, we are looking forward to deploying some new technologies to streamline it further: Telemetry will permit a more real-time collection of our data by moving from a pull model to a push model, and new automation languages like OpenConfig will unify and simplify our communication with network devices. We look forward to deploying these improvements as soon as they are mature enough for us to release.

At Cloudflare our mission is to help build a better internet. The internet though, by its nature and size, is in constant flux — breaking down, being added to, and being repaired at almost any given moment — meaning services are often interrupted and traffic is slowed without warning. By enhancing the reliability and resiliency of this complex network of networks we think we are one step closer to fulfilling our mission and building a better internet.

On June 17, at 08:32 UTC, our systems detected a significant packet loss between multiple destinations on one of our major transit provider backbone networks, Telia Carrier.In the timeframe where the incident was being analysed by our engineers, the loss became intermittent and finally disappeared.

Packet loss on Telia Carrier (AS1299)

Today, Jun 20, at 12:10 UTC, our systems again detected massive packet loss on one of our major transit provider backbone networks: Telia Carrier.

Packet loss on Telia Carrier (AS1299)

Typically, transit providers are very reliable and transport all of our packets from one point of the globe to the other without loss - that’s what we pay them for. In this case, our packets (and that of other Telia customers), were being dropped.

While Internet users usually take it for granted that they can reach any destination in the world from their homes and businesses, the reality is harsher than that. Our planet is big, and the Internet pipes are not always reliable. Fortunately, the Internet is mostly built on the TCP protocol which allows lost packets to be retransmitted. That is especially useful on lossy links. In most cases, you won’t notice these packets being lost and retransmitted, however, when the loss is too significant, as was the case this morning, your browser can’t do much.

Our systems caught the problem instantly and recorded it. Here is an animated map of the packet loss being detected during the event:

CloudFlare detects packet loss (denoted by thickness)

Because transit providers are usually reliable, they tend to fix their problems rather quickly. In this case, that did not happen and we had to take our ports down with Telia at 12:30 UTC. Because we are interconnected with most Tier 1 providers, we are able to shift traffic away from one problematic provider and let others, who are performing better, take care of transporting our packets.

We saw a big increase in our 522s errors. A 522 HTTP error indicates that our servers are unable to reach the origin servers of our customers. You can see the spike and the breakdown here:

Spike in 522 errors across PoPs (in reaching origin servers)

Communicating in this kind of incident is crucial and difficult at the same time. Our customers understandably expect prompt, accurate information and want the impact to stop as soon as possible. In today’s incident, we identified weaknesses in our communication: the scope of the incident was incorrectly identified in Europe only, and our response time was not adequate. We want to reassure you that we are taking all the steps to improve our communication, including implementation of automated detection and mitigation systems that can react much more quickly than any human operator. We already have such systems in place for our smaller data centers and are actively testing their accuracy and efficacy before turning them on for larger PoPs.

Taking down an important transit provider globally is not an easy decision, and many cautious steps are to be taken before doing it. The Internet and its associated protocols are a community based on mutual trust. Any weak link in the chain will cause the entire chain to fail and requires collaboration and cooperation from all parties to make it successful.

We know how important it is to communicate on our status page. We heard from our customers and took the necessary steps to improve on our communication. Our support team is working on improvements in how we update our status page and reviewing the content for accuracy as well as transparency.

Even as CloudFlare has grown to become critical Internet infrastructure sitting in front of four million Internet-facing applications, investing in greater resiliency continues to be a work-in-progress. This is achieved through a combination of greater interconnection, automated mitigation, and increased failover capacity.

We fill our cache from the public Internet using multiple transit providers (such as Telia), and deliver traffic to local eyeball networks using transit providers and multiple peering relationships. To the extent possible, we maintain buffer capacity with our providers to allow them to bear the impact of potential failures on multiple other networks. Spreading out traffic across providers allows for diversity and reduces the impact of potential outages from our upstream providers. Even so, today’s incident impacted a significant fraction of traffic that relied on the Telia backbone.

_Traffic switching from one provider to the other after we reroute_

Where possible, we try to failover traffic to a redundant provider or data center while keeping traffic within the same country.

BGP is the protocol used to route packets between autonomous networks on the Internet. While it is doing a great job at keeping interconnections alive, it has no mechanism built in to detect packet loss and performance issues on a path.

We have been working on building a mechanism (which augments BGP) to proactively detect packet loss and move traffic away from providers experiencing packet loss. Because this system is currently activated only for our most remote and smallest locations, it didn't trigger in this morning’s incident. We plan to extend the capability in the next 2 weeks to switch from a manual reaction to an automatic one in all our POPs. For example, in this screenshot, you can see our POP in Johannesburg being automatically removed from our network because of problems detected when connecting to origin servers:

Johannesburg PoP gracefully fails over to nearest PoP

We understand how critical our infrastructure is for our customers’ businesses, and so we will continue to move towards completely automated systems to deal with this type of incident. Our goal is to minimize disruptions and outages for our customers regardless of the origin of the issue.

We see today lot of new DDoS attacks from Internet to our network. Type: NTP AMP Size: >350Gbps. No issue. VAC is great :) — Oles (@olesovhcom) February 12, 2014

John-Graham Cumming on our team wrote a blog post before the attack describing how such an attack is possible by using a combination of spoofed UDP packets and vulnerable NTP servers.

During the 400Gbps attack we saw 4,259 IPv4 addresses of involved vulnerable servers that were sending attack traffic to our network. These networks were not controlled by the attacker directly but instead were running network time protocol (NTP) servers that responded to commands that would create very large responses, thus acting as a good amplification vector. Specifically, all of these servers were used by attackers to reply large packets in response to the "monlist" command.

In the aftermath of this massive attack, we decided to publish the list of networks originating these attacks hoping to have them fix the problem. Since the blog post we’ve been monitoring the networks to see whether attention to this problem has helped close the vulnerable NTP servers. The results are encouraging:

After a week and a half, more than 75% of the vulnerable servers involved in the attack are now no longer vulnerable. While in some cases the servers might be temporarily unreachable, the trend is clear: network administrators have gotten the message and are closing vulnerable NTP servers.

The people behind the openntp.org project also have noticed a massive improvement of the situation worlwide:

NTP MONLIST Amplifiers down from 490k -> 349k in the last week. http://t.co/35vLsj3DZJ to check your network. — jared mauch (@jaredmauch) February 14, 2014

Notably, we’ve seen a huge decrease from OVH, who have taken significant measures to prevent NTP attacks coming from its large installed base of servers. This is an encouraging achievement from the community, deploying tremendous efforts to solve a real problem.