In February 2021, Cloudflare launched Project Fair Shot — a program that gave our Waiting Room product free of charge to any government, municipality, private/public business, or anyone responsible for the scheduling and/or dissemination of the COVID-19 vaccine.

By having our Waiting Room technology in front of the vaccine scheduling application, it ensured that:

• Applications would remain available, reliable, and resilient against massive spikes of traffic for users attempting to get their vaccine appointment scheduled.

• Visitors could wait for their long-awaited vaccine with confidence, arriving at a branded queuing page that provided accurate, estimated wait times.

• Vaccines would get distributed equitably, and not just to folks with faster reflexes or Internet connections.

Since February, we’ve seen a good number of participants in Project Fair Shot. To date, we have helped more than 100 customers across more than 10 countries to schedule approximately 100 million vaccinations. Even better, these vaccinations went smoothly, with customers like the County of San Luis Obispo regularly dealing with more than 20,000 appointments in a day.  “The bottom line is Cloudflare saved lives today. Our County will forever be grateful for your participation in getting the vaccine to those that need it most in an elegant, efficient and ethical manner” — Web Services Administrator for the County of San Luis Obispo.

We are happy to have helped not just in the US, but worldwide as well. In Canada, we partnered with a number of organizations and the Canadian government to increase access to the vaccine. One partner stated: “Our relationship with Cloudflare went from ‘Let's try Waiting Room’ to ‘Unless you have this, we're not going live with that public-facing site.'” — CEO of Verto Health. In another country in Europe, we saw over three million people go through the Waiting Room in less than 24 hours, leading to a significantly smoother and less stressful experience. Cities in Japan, — working closely with our partner, Classmethod — have been able to vaccinate over 40 million people and are on track to complete their vaccination process across 317 cities. If you want more stories from Project Fair Shot, check out our case studies.

A European customer seeing very high amounts of traffic during a vaccination event

We are continuing to add more customers to Project Fair Shot every day to ensure we are doing all that we can to help distribute more vaccines. With the emergence of the Delta variant and others, vaccine distribution (and soon, booster shots) is still very much a real problem to keep everyone healthy and resilient. Because of these new developments, Cloudflare will be extending Project Fair Shot until at least July 1, 2022. Though we are not excited to see the pandemic continue, we are humbled to be able to provide our services and be a critical part in helping us collectively move towards a better tomorrow.

Around the world government and medical organizations are struggling with one of the most difficult logistics challenges in history: equitably and efficiently distributing the COVID-19 vaccine. There are challenges around communicating who is eligible to be vaccinated, registering those who are eligible for appointments, ensuring they show up for their appointments, transporting the vaccine under the required handling conditions, ensuring that there are trained personnel to administer the vaccine, and then doing it all over again as most of the vaccines require two doses.

Cloudflare can't help with most of that problem, but there is one key part that we realized we could help facilitate: ensuring that registration websites don't crash under load when they first begin scheduling vaccine appointments. Project Fair Shot provides Cloudflare's new Waiting Room service for free for any government, municipality, hospital, pharmacy, or other organization responsible for distributing COVID-19 vaccines. It is open to eligible organizations around the world and will remain free until at least July 1, 2021 or longer if there is still more demand for appointments for the vaccine than there is supply.

The problem of vaccine scheduling registration websites crashing under load isn't theoretical: it is happening over and over as organizations attempt to schedule the administration of the vaccine. This hit home at Cloudflare last weekend. The wife of one of our senior team members was trying to register her parents to receive the vaccine. They met all the criteria and the municipality where they lived was scheduled to open appointments at noon.

When the time came for the site to open, it immediately crashed. The cause wasn't hackers or malicious activity. It was merely that so many people were trying to access the site at once. "Why doesn't Cloudflare build a service that organizes a queue into an orderly fashion so these sites don't get overwhelmed?" she asked her husband.

Turns out, we were already working on such a feature, but not for this use case. The problem of fairly distributing something where there is more demand than supply comes up with several of our clients. Whether selling tickets to a hot concert, the latest new sneaker, or access to popular national park hikes it is a difficult challenge to ensure that everyone eligible has a fair chance.

The solution is to open registration to acquire the scarce item ahead of the actual sale. Anyone who visits the site ahead of time can be put into a queue. The moment before the sale opens, the order of the queue can be randomly (and fairly) shuffled. People can then be let in in order of their new, random position in the queue — allowing only so many at any time as the backend of the site can handle.

At Cloudflare, we were building this functionality for our customers as a feature called Waiting Room. (You can learn more about the technical details of Waiting Room in this post by Brian Batraski who helped build it.) The technology is powerful because it can be used in front of any existing web registration site without needing any code changes or hardware installation. Simply deploy Cloudflare through a simple DNS change and then configure Waiting Room to ensure any transactional site, no matter how meagerly resourced, can keep up with demand.

We planned to release it in February. Then, when we saw vaccine sites crashing under load and frustration of people eligible for the vaccine building, we realized we needed to move the launch up and offer the service for free to organizations struggling to fairly distribute the vaccine. With that, Project Fair Shot was born.

Government, municipal, hospital, pharmacy, clinic, and any other organizations charged with scheduling appointments to distribute the vaccine can apply to participate in Project Fair Shot by visiting: projectfairshot.org

The service will be free for qualified organizations at least until July 1, 2021 or longer if there is still more demand for appointments for the vaccine than there is supply. We are not experts in medical cold storage and I get squeamish at the sight of needles, so we can't help with many of the logistical challenges of distributing the vaccine. But, seeing how we could support this aspect, our team knew we needed to do all we could to help.

The superheroes of this crisis are the medical professionals who are taking care of the sick and the scientists who so quickly invented these miraculous vaccines. We're proud of the supporting role Cloudflare has played helping ensure the Internet has continued to function well when the world needed it most. Project Fair Shot is one more way we are living up to our mission of helping build a better Internet.

Today, we are excited to announce Cloudflare Waiting Room! It will first be available to select customers through a new program called Project Fair Shot which aims to help with the problem of overwhelming demand for COVID-19 vaccinations causing appointment registration websites to fail. General availability in our Business and Enterprise plans will be added in the near future.

Most of us are familiar with the concept of a waiting room, and rarely are we excited about the idea of being in one. Usually our first experience of one is at a doctor’s office — yes, you have an appointment, but sometimes the doctor is running late (or one of the patients was). Given the doctor can only see one person at a time… the waiting room was born, as a mechanism to queue up patients.

While servers can handle more concurrent requests than a doctor can, they too can be overwhelmed. If, in a pre-COVID world, you’ve ever tried buying tickets to a popular concert or event, you’ve probably encountered a waiting room online. It limits requests inbound to an application, and places these requests into a virtual queue. Once the number of users in the application has reduced, new users are let in within the defined thresholds the application can handle. This protects the origin servers supporting the application from being inundated with too many requests, while also ensuring equity from a user perspective — users who try to access a resource when the system is overloaded are not unfairly dropped and forced to reconnect, hoping to join their chance in the queue.

Given not many of us are going to live concerts any time soon, why is Cloudflare doing this now?

Well, perhaps we aren’t going to concerts, but the second order effects of COVID-19 have created a huge need for waiting rooms. First of all, given social distancing and the closing of many places of business and government, customers and citizens have shifted to online channels, putting substantially more strain on business and government infrastructure.

Second, the pandemic and the flow-on consequences of it have meant many folks around the world have come to rely on resources that they didn’t need twelve months earlier. To be specific, these are often health or government-related resources — for example, unemployment insurance websites. The online infrastructure was set up to handle a peak load that didn’t foresee the impact of COVID-19. We’re seeing a similar pattern emerge with websites that are related to vaccines.

Historically, the number of organizations that needed waiting rooms was quite small. The nature of most businesses online usually involves a more consistent user load, rather than huge crushes of people all at once. Those organizations were able to build custom waiting rooms and were integrated deeply into their application (for example, buying tickets).  With Cloudflare’s Waiting Room, no code changes to the application are necessary and a Waiting Room can be set up in a matter of minutes for any website without writing a single line of code.

Whether you are an engineering architect or a business operations analyst, setting up a Waiting Room is simple. We make it quick and easy to ensure your applications are reliable and protected from unexpected spikes in traffic.  Other features we felt were important are automatic enablement and dynamic outflow. In other words, a waiting room should turn on automatically when thresholds are exceeded and as users finish their tasks in the application, let out different sized buckets of users and intake new ones already in the queue. It should just work. Lastly, we’ve seen the major impact COVID-19 has made on users and businesses alike, especially, but not limited to, the health and government sectors. We wanted to provide another way to ensure these applications remain available and functional so all users can receive the care that they need and not errors within their browser.

We built Waiting Room on top of our edge network and our Workers product. By leveraging Workers and our new Durable Objects offerings, we were able to remove the need for any customer coding and provide a seamless, out of the box product that will ‘just work’. On top of this, we get the benefits of the scale and performance of our Workers product to ensure we maintain extremely low latency overhead, keep estimated times presented to end users accurate as can be and not keep any user in the queue longer than needed. But building a centralized system in a decentralized network is no easy task. When requests come into an application from around the world, we need to be able to get a broad, accurate view of what that load looks like inbound and outbound to a given application.

Request going through Cloudflare without a Waiting Room

These requests, as fast as they are, still take time to travel across the planet. And so, a unique edge case was presented. What if a website is getting reasonable traffic from North America and Europe, but then a sudden major spike of traffic takes place from South America - how do we know when to keep letting users into the application and when to kick in the Waiting Room to protect the origin servers from being overloaded?

Thanks to some clever engineering and our Workers product, we were able to create a system that almost immediately keeps itself synced with global demand to an application giving us the necessary insight into when we should and should not be queueing users into the Waiting Room. By leveraging our global Anycast network and over 200+ data centers, we remove any single point of failure to protect our customers' infrastructure yet also provide a great experience to end-users who have to wait a small amount of time to enter the application under high load.

Request going through Cloudflare with a Waiting Room

Setting up a Waiting Room is incredibly easy and very fast! At the easiest side of the scale, a user needs to fill out only five fields: 1) the name of the Waiting Room, 2) a hostname (which will already be pre-populated with the zone it’s being configured on), 3) the total active users that can be in the application at any given time, 4) the new users per minute allowed into the application, and 5) the session duration for any given user. No coding or any application changes are necessary.

We provide the option of using our default Waiting Room template for customers who don’t want to add additional branding. This simplifies the process of getting a Waiting Room up and running.

That’s it! Press save and the Waiting Room is ready to go!

For customers with more time and technical ability, the same process is followed, except we give full customization capabilities to our users so they can brand the Waiting Room, ensuring it matches the look and feel of their overall product.

Lastly, managing different Waiting Rooms is incredibly easy. With our Manage Waiting Room table, at a glance you are able to get a full snapshot of which rooms are actively queueing, not queueing, and/or disabled.

We are very excited to put the power of our Waiting Room into the hands of our customers to ensure they continue to focus on their businesses and customers. Keep an eye out for another blog post coming soon with major updates to our Waiting Room product for Enterprise!

In July 2020, I joined Cloudflare as a Product Management Intern on the DDoS (Distributed Denial of Service) team to enhance the benefits that Network Analytics brings to our customers. In the following, I am excited to share with you my experience with remote working as an intern, and how I acclimatized into Cloudflare. I also give details about what my work entailed and how we approached the process of Product Management.

As a long-time user of Cloudflare’s Free CDN plan myself, I was thrilled to join the company and learn what was happening behind the scenes while making its products. The entering internship class consisted of students and recent graduates from various backgrounds around the world - all with a mutual passion in helping build a better Internet.

The catch here was that 2020 would make the experience of being an intern very different. As it was the case with many other fellow interns, it was the first time I had taken up work remotely from scratch. The initial challenge was to integrate into the working environment without ever meeting colleagues in a physical office. Because everything took place online, it was much harder to pick up non-verbal cues that play a key role in communication, such as eye contact and body language.

To face this challenge, Cloudflare introduced creative and active ways in which we could better interact with one another. From the very first day, I was welcomed to an abundance of knowledge sharing talks and coffee chats with new and existing colleagues in different offices across the world. Whether it was data protection from the Legal team or going serverless with Workers, we were welcomed to afternoon seminars every week on a new area that was being pursued within Cloudflare.

Cloudflare not only retained the summer internship scheme, but in fact doubled the size of the class; this reinforced an optimistic mood within the entering class and a sense of personal responsibility. I was paired up with a mentor, a buddy, and a manager who helped me find my way quickly within Cloudflare, and without which my experience would not have been the same. Thanks to Omer, Pat, Val and countless others for all your incredible support!

Social interactions took various forms and were scheduled for all global time zones. I was invited to weekly virtual yoga sessions and intern meetups to network and discover what other interns across the world were working on. We got to virtually mingle at an “Intern Mixer” where we shared answers to philosophical prompts – what’s more, this was accompanied by an UberEats coupon for us to enjoy refreshments in our work-from-home setting. We also had Pub Quizzes with colleagues in the EMEA region to brush up on our trivia skills. At this uncertain time of the year, part of which I spent in complete self-isolation, these gatherings helped create a sense of belonging within the community, as well as an affinity towards the colleagues I interacted with.

My internship also offered a unique learning experience from the Product Management perspective. I took on the task of increasing the value of Network Analytics by giving customers and internal stakeholders improved  transparency in the traffic patterns and attacks taking place. Network Analytics is Cloudflare’s packet- and bit-oriented dashboard that provides visibility into network- and transport-layer attacks which are mitigated across the world. Among various updates I led in visibility features is the new trends insights. During this time the dashboard was also extended to Enterprise customers on the Spectrum service, Cloudflare's L4 reverse-proxy that provides DDoS protection against attacks and facilitates network performance.

I was at the intersection of multiple teams that contributed to Network Analytics from different angles, including user interface, UX research, product design, product content and backend engineering, among many others. The key to a successful delivery of Network Analytics as a product, given its interdisciplinary nature, meant that I actively facilitated communication and collaboration across experts in these teams as well as reflected the needs of the users.

I spent the first month of the internship approaching internal stakeholders, namely Customer Support engineers, Solutions Engineers, Customer Success Managers, and Product Managers, to better understand the common pain points. Given their past experience with customers, their insights revealed how Network Analytics could both leverage the existing visibility features to reduce overhead costs on the internal support side and empower users with actionable insights. This process also helped ensure that I didn’t reinvent wheels that had already been explored by existing Product Managers.

I then approached customers to enquire about desired areas for improvements. An example of such a desired improvement was that the display of data in the dashboard was not helping users infer any meaning regarding next steps. It did not answer questions like: What do these numbers represent in retrospect, and should I be concerned? Discussing these aspects helped validate the needs, and we subsequently came up with rough solutions to address them, such as dynamic trends view. Over the calls, we confirmed that - especially from those who rarely accessed the dashboard - having an overview of these numbers in the form of a trends card would incentivize users to log in more often and get more value from the product.

Trends Insights

The 1:1 dialogues were incredibly helpful in understanding how Network Analytics could be more effectively utilized, and guided ways for us to better surface the performance of our DDoS mitigation tools to our customers. In the first few weeks of the internship, I shadowed customer calls of other products; this helped me gain the confidence, knowledge, and language appropriate in Cloudflare’s user research. I did a run-through of the interview questions with a UX Researcher, and was informed on the procedure for getting in touch with appropriate customers. We even had bilingual calls where the Customer Success Manager helped translate the dialogues real-time.

In the following weeks, I synthesized these findings into a Product Requirements Document and lined up the features according to quarterly goals that could now be addressed in collaboration with other teams. After a formal review and discussion with Product Managers, engineers, and designers, we developed and rolled out each feature to the customers on a bi-weekly basis. We always welcomed feedback before and after the feature releases, as the goal wasn’t to have an ultimate final product, but to deliver incremental enhancements to meet the evolving needs of our customers.

Of course, all my interactions, including customer and internal stakeholder calls, were all held remotely. We all embraced video conferencing and instant chat messengers to make it feel as though we were physically close. I had weekly check-ins with various colleagues including my managers, Network Analytics team, DDoS engineering team, and DDoS reports team, to ensure that things were on track. For me, the key to working remotely was the instant chat function, which was not as intrusive as a fully fledged meeting, but a quick and considerate way to communicate in a tightly-knit team.

Product Management is a growth process - both for the corresponding individual and the product. As an individual, you grow fast through creative thinking, problem solving and incessant curiosity to better understand a product in the shoes of a customer. At the same time, the product continues to evolve and grow as a result of synergy between experts from diverse fields and customer feedback. Products are used and experienced by people, so it is a no-brainer that maintaining constant and direct feedback from our customers and internal stakeholders are what bolsters their quality.

It was an incredible opportunity to have been a part of an organization that represents one of the largest networks. Network Analytics is a window into the efforts led by Cloudflare engineers and technicians to help secure the Internet, and we are ambitious to scale the transparency across further mitigation systems in the future.

The internship was a successful immersive experience into the world of Network Analytics and Product Management, even in the face of a pandemic. Owing to Cloudflare’s flexibility and ready access to resources for remote work, I was able to adapt to the work environment from the first day onwards and gain an authentic learning experience into how products work. As I now return to university, I look back on an internship that significantly added to my personal and professional growth. I am happy to leave behind the latest evolution of Network Analytics dashboard with hopefully many more to come. Thanks to Cloudflare and all my colleagues for making this possible!

Now that we’re a long way through 2020, let’s take a look at automated traffic, which makes up almost 40% of total Internet traffic.

This blog post is a high-level overview of bot traffic on Cloudflare’s network. Cloudflare offers a comprehensive Bot Management tool for Enterprise customers, along with an effective free tool called Bot Fight Mode. Because of the tremendous amount of traffic that flows through our network each day, Cloudflare is in a unique position to analyze global bot trends.

In this post, we will cover the basics of bot traffic and distinguish between automated requests and other human requests (What Is A Bot?). Then, we’ll move on to a global overview of bot traffic around the world (A RoboBird’s Eye View, A Bot Day and Bots All Over The World), and dive into North American traffic (A Look into North American Traffic).  Lastly, we’ll finish with an overview of how the coronavirus pandemic affected global traffic, and we’ll take a deeper look at European traffic (Bots During COVID-19 In Europe).

On average, Cloudflare processes 18 million HTTP requests every second. This is a great opportunity to understand how bots shape the Internet, how much infrastructure is dedicated to these automated requests, and why our customers need a great bot management solution.

Cloudflare groups traffic into four bot-related categories:

1. Verified2. Definitely automated3. Likely automated4. Likely human

Our goal is to stop malicious and unwanted bots from harming our customers, while giving customers the opportunity to control how other automated traffic is managed.

We label each request that comes into Cloudflare with a “bot score” 1 through 99, where a lower score means that a request probably came from a bot. A higher score means that a request probably came from a human. This score is available in our Firewall, logs, and Workers, giving customers the flexibility to act on any score.

Cloudflare also maintains a challenge platform that customers can choose to deploy on suspected bots. You’ll recognize these as CAPTCHA challenges or JavaScript challenges. In fact, having the score available in Firewall Rules means that customers can take any action they choose. This platform can be used for mitigation, ensuring that unwanted traffic is stopped in its tracks.

To learn more about how Bot Management interacts with our firewall, check out our support page.

We track successes and failures during these challenges, which ultimately allows us to improve our detection systems. Assuming that our challenges are solvable by humans, effective detections should have low solve rates, given that they are usually presented to bots.

Verified bots are registered in an internal verified bot directory. These good bots power search engines and monitoring tools. Good bots enable our customers’ web pages to be found by search engines, for example.

For known non-verified bots (such as a scraper using a simple curl library), we keep a similar directory that is managed by our heuristics engine. If not otherwise verified, we consider requests caught by this engine to be definitely automated.

Our machine learning engine provides another way to identify potential bots. This engine identifies requests with a high probability of automation and marks them as likely automated. This detection mechanism benefits from models built on data from our global network.

If a request is not marked as automated, we mark it as likely human and pass along the bot score from our machine learning system.

We also have a behavioral analysis engine and a JavaScript detections engine. You can learn more about these systems by checking out Alex Bocharov’s previous post on Cloudflare Bot Management.

The two bot definitions for automated traffic are somewhat complementary. Requests caught by heuristic detections will not count towards machine learning detections. Requests that are reliably caught by our machine learning detections won’t need to be registered in our known heuristics bot directory. Because of this, we combine these two together when we discuss “automated traffic” in general.

Data from this piece comes from information about Cloudflare’s customers, analyzed between January 15, 2020 and July 31, 2020.

First, let’s get a basic understanding of the traffic on our network.

Figure 1.1 Traffic type on Cloudflare’s network.

Figure 1.1 has a global breakdown regarding classification; 60.6% of traffic is likely human, 19.3% is likely automated, 18.1% is definitely automated and only 2.1% is from verified bots. In total, 39.5% of requests we score come from some kind of bot.

Regular traffic fluctuates throughout the day. Do bots follow suit? Let’s check. Figure 2.1 represents traffic deviation from the average hourly traffic. An increase of 10% would mean that the hour is 10% busier than the average hour (measuring requests per hour). We include the total overall traffic in this chart to serve as a comparison to other types of traffic.

Figure 2.1 Hourly traffic as a deviation from the average hour.

Figure 2.2 Bot classification over an average day. 

We can clearly see a difference between human traffic and bot traffic. Human traffic varies heavily, but predictably, throughout the day. We can see a 15% decrease in human traffic early in the day, between midnight and 05:00 UTC, corresponding to the end of business hours in the Americas, and up to a 25% increase during business hours, 14:00 to 17:00 UTC, where traffic is highest. Conversely, bot traffic is more consistent. Slow hours still see a smaller drop than overall traffic, and busy hours are less busy. The difference between good and bad bots is also apparent: good bots are even more consistent, with small fluctuations in hourly traffic.

But why would this happen? A large portion of bots, good and bad, perform the same task across the Internet. Bad bots may be scraping websites or looking to infect unprotected machines, and they will do this with little intervention from human operators. Good bots could be doing some of these operations, but less frequently and in a more targeted fashion. A good bot scraping a website may be doing so to add it to a search engine, while a bad bot will do the same thing at a much higher rate, for other reasons.

A lot of bots follow business hours. For example, sneaker bots—focused on nabbing exclusive items from sneaker stores—will naturally be active when new products launch.

This difference in volume does not mean that our classifications are affected: our scores remain consistent throughout the day, as Figure 2.1 shows.

Figure 2.3 Daily traffic as a deviation from the average day. Grouped by day of week.

Figure 2.4 Bot classification over an average week.

We can also see that good bots don’t take weekends off. Weekdays and weekends have fairly marked differences for most traffic, but good bots keep a consistent schedule. Whereas a typical weekday is slightly above average, we can see a drop of about 4% in overall traffic. This does not fully apply to verified bots, which only see a small 1% drop in traffic.

Now that we’ve taken a look at global traffic, let’s dig a little deeper.

Different regions have distinct traffic landscapes regarding automated traffic.

**Figure 3.1** Traffic type by region.

Figure 3.1 breaks down traffic by region, letting us peek into where each type of traffic comes from. North America stands out as a major automated traffic source; over 50% of definitely automated traffic comes from there, and they also contribute almost 80% of all verified bot traffic. Europe makes up the second largest chunk of traffic, followed by Asia.

Figure 3.2 Traffic classification within each region.

Looking at regional breakdown of traffic in Figure 3.2, we can see just how much North American traffic is automated, well above the global average.

As the largest source of automated traffic, North America deserves a closer look.

First, we’ll start with a breakdown of each country.

Figure 3.3 Percentage of traffic within North America.

Most of our requests in North America come from just three countries—the United States, Canada and Mexico. These account for 98% of all requests from North America, 97% of all requests from likely human sources and 100% of requests from verified bots. The United States alone accounts for 88% of total requests, 82% of requests from likely human sources, 96% of requests from definitely automated sources, 88% of requests from likely automated traffic sources and  98% of requests from verified bot.

However, this alone does not mean that the United States has an unusual amount of activity. These countries have a combined population of roughly 497 million people. The United States accounts for 66.5% of that, Mexico 25.9% and Canada 7.6%. With this context, we can see that the United States is overrepresented in terms of raw requests, but underrepresented in terms of how much of that traffic is likely to be human. Conversely, Canadian traffic is more likely to be human.

Let's take another look at each country.

Figure 3.4 Percentage of traffic within each country.

Over half of the traffic from the United States is automated in some way, which is a clear departure from trends in Mexico and Canada.

So far, we’ve seen how much the United States contributes to automated traffic. If we want to go deeper, a good place to start is by understanding how these bots get online. We can do this by examining the networks from which the traffic originates. Networks are identified by Autonomous System Numbers, or ASNs. These form the backbone of the Internet infrastructure.

Think of these as Internet Service Providers, but facing inward towards the network instead of outward towards end consumers. ISPs like Comcast and Verizon are examples of residential ASNs, where we expect mostly human traffic. Cloud providers such as Google and Amazon are also ASNs, but targeted towards cloud services. We expect most of these requests to be automated in some way.

Looking at traffic on the ASN level is important because we can identify cloud-based traffic, or traffic using residential proxies, among others.

Let's take a look at which ASNs are associated with visitors in the United States. We’ll restrict ourselves to “eyeball” traffic, which is the term we use for requests coming from site visitors.

Figure 4.1 Top ASN in the United States.

From figure 4.1 we can clearly see the impact that cloud services have on traffic; 11.5% of all eyeball traffic comes from Amazon and Google.

Figure 4.2 Top ASN in the United States for verified bot traffic.

Verified bots operate in a different landscape, coming from cloud providers such as Amazon, Google, Microsoft, Advanced Hosting and Wowrack.

Figure 4.3 Top ASN in the United States for likely and definitely automated traffic.

Automated traffic has a variety of ASNs. Cloud providers such as Amazon, Google and Microsoft make up the 30% of automated traffic. Comcast also makes up a significant portion of traffic at 4.8%, indicating that some bots come from residential services.

Lockdowns and limits on public events came as a consequence of the ongoing coronavirus pandemic. Many people have been working from home, and even those who do not have this option are using the Internet in new ways. Overall, this has meant that Cloudflare’s network has grown tremendously.

But how does this impact bot traffic? First let’s get an idea of how it impacted traffic in general. Countries were impacted by the virus at different times, so we expect to see differences, right?

Figure 5.1 Total traffic across all regions.

Figure 5.1 has just the traffic increase. Globally, we are seeing an average increase of 10%, while North America saw an increase of over 40% compared to the beginning of the year. Some regions did not change much, such as Africa and Asia, while others, such as Europe saw an increased period, but has since normalized to previous levels.

Let's look at a few countries, so we can understand what this looks like.

Figure 5.2 Daily traffic evolution for Italy, the United Kingdom and Portugal, overlaid with Europe.

Figure 5.2 shows daily traffic relative to January 15, when data collection started. For comparison, we have overall European traffic, and three selected countries: Italy, the United Kingdom and Portugal. Italy was picked because it was one of the first countries in Europe to face the worst of the coronavirus and enact lockdown measures. The United Kingdom took another strategy, with an initial focus on herd immunity, and enacted measures later than the others. Portugal is somewhere in between, locking down later than Italy, in slightly different circumstances.

At the beginning of the year, traffic kept stable and fluctuations kept in line with the European average. As lockdown measures began, traffic increased. Italy was first out of these countries, rising a few weeks before the others, and keeping well above average. Eventually, all countries saw a growth in traffic, followed by a stabilization. Italy seems to have adjusted to a normal, with its growth in line with the European average. Portugal has also stabilized, but with busier weekdays. Conversely, the United Kingdom showed no signs of stopping, exceeding a growth of 40% compared to the beginning of the year.

Figure 5.3 Daily definitely automated traffic evolution for Italy, the United Kingdom and Portugal, overlaid with Europe.

Definitely automated traffic did not have that much of a pronounced variation. Italian traffic kept steady throughout, and Portugal had a rather large increase. The biggest one, however, was the United Kingdom, which tripled its initial count.

Figure 5.4 Verified bot traffic evolution for Italy, the United Kingdom and Portugal, overlaid with Europe. 

Verified bot traffic is steady, except in Italy, with a massive increase between March and May. What could be the cause of this? Are these a few zones, getting a massive number of requests?

Figure 5.5 Verified bot traffic in Italy for the top 10 000 zones, relative to January 15th 2020.

Well, no. If we only examine the top 10,000 zones (by total verified bot requests), we can still see a massive increase in traffic for other zones. So, what’s happening?

Let's look at user agents. We can separate the top 10 user agents during the bump, and see how they evolve over time.

Figure 5.6 Verified bot traffic in Italy for the top 10 user agents, relative to January 15th 2020.

We can see that these 10 user agents are responsible for the majority of verified traffic coming from Italy.

Figure 5.7 Verified bot traffic in Italy for the top user agent, relative to January 15 2020.

In fact, most of this increase is from a single user agent. This instance of Google image proxy anonymizes image requests from Gmail, which explains its popularity.

Where does this increase come from? Did this bot suddenly appear and disappear?

Not quite. One thing to keep in mind when dealing with bots is that they cross borders easily. As a proxy service, this bot is making calls on behalf of the end user – people opening emails. These requests will originate from a data center, which can be anywhere in the world. To see this in action, let's take a look at traffic for this bot in a few select countries.

Figure 5.8. Countries of origin for GoogleImageProxy.

We can see that the global average barely budges. It appears that Google may be moving image proxy traffic between data centers and during the period we observed above that traffic was coming from Italy.

With Cloudflare’s global reach, we’re in a position to understand how bots behave.

The first half of 2020 saw a massive increase in web traffic of around 35% since the beginning of the year, driven by the ongoing coronavirus pandemic, and some bots have taken advantage of it.

We explained how bot management works for our customers, and how we distinguish between likely automated and human traffic.

We showed an overview of how much of our global traffic is automated, and how bots change their behavior throughout the day and the week. Notably, 39.4% of all traffic Cloudflare processes comes from a suspected automated source.

A regional overview of automated traffic lets us know which regions were the source of traffic from likely automated agents. North America, Europe and Asia were the primary sources of traffic, and also of automated traffic in particular.

We then focused on North America, where the majority of automated traffic originates. The United States alone accounted for the majority of requests, over half of which come from automated sources.

To explore this further, we briefly dived into ASN traffic in the United States, so we could see where these requests were coming from. ASNs like Comcast and AT&T were the top ASNs for overall traffic, but unsurprisingly, data centers like Google and Amazon AWS were the main drivers of automated traffic.

Finally, we examined how the coronavirus has impacted traffic in Europe, with a deeper dive on Italian traffic. This led to some interesting insights on verified bot traffic, which saw a massive increase in Italy for a few months.

This post is a small peek into bot management at Cloudflare. In the future, we hope to expand this series of blog posts on bot management, exposing even more insights about bots on the Internet.

In the first quarter of 2020, within a matter of weeks, our way of life shifted. We’ve become reliant on online services more than ever. Employees that can are working from home, students of all ages and grades are taking classes online, and we’ve redefined what it means to stay connected. The more the public is dependent on staying connected, the larger the potential reward for attackers to cause chaos and disrupt our way of life. It is therefore no surprise that in Q1 2020 (January 1, 2020 to March 31, 2020) we reported an increase in the number of attacks—especially after various government authority mandates to stay indoors—shelter-in-place went into effect in the second half of March.

In Q2 2020 (April 1, 2020 to June 30, 2020), this trend of increasing DDoS attacks continued and even accelerated:

• The number of L3/4 DDoS attacks observed over our network doubled compared to that in the first three months of the year.

• The scale of the largest L3/4 DDoS attacks increased significantly. In fact, we observed some of the largest attacks ever recorded over our network.

• We observed more attack vectors being deployed and attacks were more geographically distributed.

Gatebot is Cloudflare’s primary DDoS protection system. It automatically detects and mitigates globally distributed DDoS attacks. A global DDoS attack is an attack that we observe in more than one of our edge data centers. These attacks are usually generated by sophisticated attackers employing botnets in the range of tens of thousand to millions of bots.

Sophisticated attackers kept Gatebot busy in Q2. The total number of global L3/4 DDoS attacks that Gatebot detected and mitigated in Q2 doubled quarter over quarter. In our Q1 DDoS report, we reported a spike in the number and size of attacks. We continue to see this trend accelerate through Q2; over 66% of all global DDoS attacks in 2020 occurred in the second quarter (nearly 100% increase). May was the busiest month in the first half of 2020, followed by June and April. Almost a third of all L3/4 DDoS attacks occurred in May.

In fact, 63% of all L3/4 DDoS attacks that peaked over 100 Gbps occurred in May. As the global pandemic continued to heighten around the world in May, attackers were especially eager to take down websites and other Internet properties.

A DDoS attack’s strength is equivalent to its size—the actual number of packets or bits flooding the link to overwhelm the target. A ‘large’ DDoS attack refers to an attack that peaks at a high rate of Internet traffic. The rate can be measured in terms of packets or bits. Attacks with high bit rates attempt to saturate the Internet link, and attacks with high packet rates attempt to overwhelm the routers or other in-line hardware devices.

Similar to Q1, the majority of L3/4 DDoS attacks that we observed in Q2 were also relatively ‘small’ with regards to the scale of Cloudflare’s network. In Q2, nearly 90% of all L3/4 DDoS attacks that we saw peaked below 10 Gbps. Small attacks that peak below 10 Gbps can still easily cause an outage to most of the websites and Internet properties around the world if they are not protected by a cloud-based DDoS mitigation service.

Similarly, from a packet rate perspective, 76% of all L3/4 DDoS attacks in Q2 peaked up to 1 million packets per second (pps). Typically, a 1 Gbps Ethernet interface can deliver anywhere between 80k to 1.5M pps. Assuming the interface also serves legitimate traffic, and that most organizations have much less than a 1 Gbps interface, you can see how even these ‘small’ packet rate DDoS attacks can easily take down Internet properties.

In terms of duration, 83% of all attacks lasted between 30 to 60 minutes. We saw a similar trend in Q1 with 79% of attacks falling in the same duration range. This may seem like a short duration, but imagine this as a 30 to 60 minute cyber battle between your security team and the attackers. Now it doesn’t seem so short. Additionally, if a DDoS attack creates an outage or service degradation, the recovery time to reboot your appliances and relaunch your services can be much longer; costing you lost revenue and reputation for every minute.

This quarter, we saw an increasing number of large scale attacks; both in terms of packet rate and bit rate. In fact, 88% of all DDoS attacks in 2020 that peaked above 100 Gbps were launched after shelter-in-place went into effect in March. Once again, May was not just the busiest month with the most number of attacks, but also the greatest number of large attacks above 100 Gbps.

From the packet perspective, June took the lead with a whopping 754 million pps attack. Besides that attack, the maximum packet rates stayed mostly consistent throughout the quarter with around 200 million pps.

The 754 million pps attack was automatically detected and mitigated by Cloudflare. The attack was part of an organized four-day campaign that lasted from June 18 to the 21. As part of the campaign, attack traffic from over 316,000 IP addresses targeted a single Cloudflare IP address.

Cloudflare’s DDoS protection systems automatically detected and mitigated the attack, and due to the size and global coverage of our network, there was no impact to performance. A global interconnected network is crucial when mitigating large attacks in order to be able to absorb the attack traffic and mitigate it close to the source, whilst also continuing serving legitimate customer traffic without inducing latency or service interruptions.

When we look at the L3/4 DDoS attack distribution by country, our data centers in the United States received the most number of attacks (22.6%), followed by Germany (4.4%), Canada (2.7%) and Great Britain (2.6%).

However when we look at the total attack bytes mitigated by each Cloudflare data center, the United States still leads (34.9%), but followed by Hong Kong (6.6%), Russia (6.5%), Germany (4.5%) and Colombia (3.7%). The reason for this change is due to the total amount of bandwidth that was generated in each attack. For instance, while Hong Kong did not make it to the top 10 list due to the relatively small number of attacks that was observed in Hong Kong (1.8%), the attacks were highly volumetric and generated so much attack traffic that pushed Hong Kong to the 2nd place.

When analyzing L3/4 DDoS attacks, we bucket the traffic by the Cloudflare edge data center locations and not by the location of the source IP. The reason is when attackers launch L3/4 attacks they can ‘spoof’ (alter) the source IP address in order to obfuscate the attack source. If we were to derive the country based on a spoofed source IP, we would get a spoofed country. Cloudflare is able to overcome the challenges of spoofed IPs by displaying the attack data by the location of Cloudflare’s data center in which the attack was observed. We’re able to achieve geographical accuracy in our report because we have data centers in over 200 cities around the world.

An attack vector is a term used to describe the attack method. In Q2, we observed an increase in the number of vectors used by attackers in L3/4 DDoS attacks. A total of 39 different types of attack vectors were used in Q2, compared to 34 in Q1. SYN floods formed the majority with over 57% in share, followed by RST (13%), UDP (7%), CLDAP (6%) and SSDP (3%) attacks.

SYN flood attacks aim to exploit the handshake process of a TCP connection. By repeatedly sending initial connection request packets with a synchronize flag (SYN), the attacker attempts to overwhelm the router’s connection table that tracks the state of TCP connections. The router replies with a packet that contains a synchronized acknowledgment flag (SYN-ACK), allocates a certain amount of memory for each given connection and falsely waits for the client to respond with a final acknowledgment (ACK). Given a sufficient number of SYNs that occupy the router’s memory, the router is unable to allocate further memory for legitimate clients causing a denial of service.

No matter the attack vector, Cloudflare automatically detects and mitigates stateful or stateless DDoS attacks using our 3 pronged protection approach comprising of our home-built DDoS protection systems:

1. Gatebot - Cloudflare's centralized DDoS protection systems for detecting and mitigating globally distributed volumetric DDoS attacks. Gatebot runs in our network’s core data center. It receives samples from every one of our edge data centers, analyzes them and automatically sends mitigation instructions when attacks are detected. Gatebot is also synchronized to each of our customers’ web servers to identify its health and triggers accordingly, tailored protection.

2. dosd (denial of service daemon) - Cloudflare’s decentralized DDoS protection systems. dosd runs autonomously in each server in every Cloudflare data center around the world, analyzes traffic, and applies local mitigation rules when needed. Besides being able to detect and mitigate attacks at super fast speeds, dosd significantly improves our network resilience by delegating the detection and mitigation capabilities to the edge.

3. flowtrackd (flow tracking daemon) - Cloudflare’s TCP state tracking machine for detecting and mitigating the most randomized and sophisticated TCP-based DDoS attacks in unidirectional routing topologies. flowtrackd is able to identify the state of a TCP connection and then drops, challenges or rate-limits packets that don’t belong to a legitimate connection.

In addition to our automated DDoS protection systems, Cloudflare also generates real-time threat intelligence that automatically mitigates attacks. Furthermore, Cloudflare provides its customers firewall, rate-limiting and additional tools to further customize and optimize their protection.

As Internet usage continues to evolve for businesses and individuals, expect DDoS tactics to adapt as well. Cloudflare protects websites, applications, and entire networks from DDoS attacks of any size, kind, or level of sophistication.

Our customers and industry analysts recommend our comprehensive solution for three main reasons:

• Network scale: Cloudflare’s 37 Tbps network can easily block attacks of any size, type, or level of sophistication. The Cloudflare network has a DDoS mitigation capacity that is higher than the next four competitors—combined.

• Time-to-mitigation: Cloudflare mitigates most network layer attacks in under 10 seconds globally, and immediate mitigation (0 seconds) when static rules are preconfigured. With our global presence, Cloudflare mitigates attacks close to the source with minimal latency. In some cases, traffic is even faster than over the public Internet.

• Threat intelligence: Cloudflare’s DDoS mitigation is powered by threat intelligence harnessed from over 27 million Internet properties on it. Additionally, the threat intelligence is incorporated into customer facing firewalls and tools in order to empower our customers.

Cloudflare is uniquely positioned to deliver DDoS mitigation with unparalleled scale, speed, and smarts because of the architecture of our network. Cloudflare’s network is like a fractal—every service runs on every server in every Cloudflare data center that spans over 200 cities globally. This enables Cloudflare to detect and mitigate attacks close to the source of origin, no matter the size, source, or type of attack.

To learn more about Cloudflare’s DDoS solution contact us or get started.

You can also join an upcoming live webinar where we will be discussing these trends, and strategies enterprises can implement to combat DDoS attacks and keep their networks online and fast. You can register here.

This was an internship unlike any other. With a backdrop of a pandemic, protests, and a puppy that interrupted just about every Zoom meeting, it was also an internship that demonstrated Cloudflare’s leadership in giving students meaningful opportunities to explore their interests and contribute to the company’s mission: to help build a better Internet.

For the past twelve weeks, I’ve had the pleasure of working as a Legal Intern at Cloudflare. A few key things set this internship apart from even those in which I’ve been able to connect with people in-person:

• Communication

• Community

• Commingling

• Collaboration

Ever since I formally accepted my internship, the Cloudflare team has been in frequent and thorough communication about what to expect and how to make the most of my experience. This approach to communication was in stark contrast to the approach taken by several other companies and law firms. The moment COVID-19 hit, Cloudflare not only reassured me that I’d still have a job, the company also doubled down on bringing on more interns. Comparatively, a bunch of my fellow law school students were left in limbo: unsure of if they had a job, the extent to which they’d be able to do it remotely, and whether it would be a worthwhile experience.

This approach has continued through the duration of the internship. I know I speak for my fellow interns when I say that we were humbled to be included in company-wide initiatives to openly communicate about the trying times our nation and particularly members of communities of color have experienced this summer. We weren’t left on the sidelines but rather invited into the fold. I’m so grateful to my manager, Jason, for clearing my schedule to participate in Cloudflare’s “Day On: Learning and Inclusion.” On June 18, the day before Juneteenth, Cloudflare employees around the world joined together for transformative and engaging sessions on how to listen, learn, participate, and take action to be better members of our communities. That day illustrated Cloudflare’s commitment to fostering communication as well as to building community and diversity.

The company’s desire to foster a sense of community pervades each team. Case in point, members of the Legal, Policy, and Trust & Safety (LPT) team were ready and eager to help my fellow legal interns and me better understand the team’s mission and day-to-day activities. I went a perfect 11/11 on asks to LPT members for 1:1 Zoom meetings -- these meetings had nothing to do with a specific project but were merely meant to create a stronger community by talking with employees about how they ended up at this unique company.

From what I’ve heard from fellow interns, this sense of community was a common thread woven throughout their experiences as well. Similarly, other interns shared my appreciation for being given more than just “shadowing” opportunities. We were invited to commingle with our teammates and encouraged to take active roles in meetings and on projects.

In my own case, I got to dive into exciting research on privacy laws such as the GDPR and so much more. This research required that I do more than just be a fly on the wall, I was invited to actively converse and brief folks directly involved with making key decisions for the LPT. For instance, when Tilly came on in July as Privacy Counsel, I had the opportunity to brief her on the research I’d done related to Data Privacy Impact Assessments (DPIAs). In the same way, when Edo and Ethan identified some domain names that likely infringed on Cloudflare’s trademark, my fellow intern, Elizabeth, and I were empowered to draft WIPO complaints per the Uniform Domain Name Dispute Resolution Policy. Fingers crossed our work continues Cloudflare’s strong record before the WIPO (here’s an example of a recent favorable division). These seemingly small tasks introduced me to a wide range of fascinating legal topics that will inform my future coursework and, possibly, even my career goals.

Finally, collaboration distinguished this internship from other opportunities. By way of example, I was assigned projects that required working with others toward a successful outcome. In particular, I was excited to work with Jocelyn and Alissa on research related to the intersection of law and public policy. This dynamic duo fielded my queries, sent me background materials, and invited me to join meetings with stakeholders. This was a very different experience from previous internships in which collaboration was confined to just an email assigning the research and a cool invite to reach out if any questions came up. At Cloudflare, I had the support of a buddy, a mentor, and my manager on all of my assignments and general questions.

When I walked out of Cloudflare’s San Francisco office back in December after my in-person interview, I was thrilled to potentially have the opportunity to return and help build a better Internet. Though I’ve yet to make it back to the office due to COVID-19 and, therefore, worked entirely remotely, this internship nevertheless allowed me and my fellow interns to advance Cloudflare’s mission.

Whatever normal looks like in the following weeks, months, and years, so long as Cloudflare prioritizes communication, community, commingling, and collaboration, I know it will be a great place to work.

We’ve seen a massive shift from users accessing the Internet from busy office districts to spread out urban areas.

Although there are a slew of speed testing tools out there, none of them give you precise insights into how they came to those measurements and how they map to real-world performance. With speed.cloudflare.com, we give you insights into what we’re measuring and how exactly we calculate the scores for your network connection. Best of all, you can easily download the measurements from right inside the tool if you’d like to perform your own analysis.

We also know you care about privacy. We believe that you should know what happens with the results generated by this tool. Many other tools sell the data to third parties. Cloudflare does not sell your data. Performance data is collected and anonymized and is governed by the terms of our Privacy Policy. The data is used anonymously to determine how we can improve our network, both in terms of capacity as well as to help us determine which Internet Service Providers to peer with.

The test has three main components: download, upload and a latency test. Each measures  different aspects of your network connection.

For starters we run you through a basic download test. We start off downloading small files and progressively move up to larger and larger files until the test has saturated your Internet downlink. Small files (we start off with 10KB, then 100KB and so on) are a good representation of how websites will load, as these typically encompass many small files such as images, CSS stylesheets and JSON blobs.

For each file size, we show you the measurements inside a table, allowing you to drill down. Each dot in the bar graph represents one of the measurements, with the thin line delineating the range of speeds we've measured. The slightly thicker block represents the set of measurements between the 25th and 75th percentile.

Getting up to the larger file sizes we can see true maximum throughput: how much bandwidth do you really have? You may be wondering why we have to use progressively larger files. The reason is that download speeds start off slow (this is aptly called slow start) and then progressively gets faster. If we were to use only small files we would never get to the maximum throughput that your network provider supports, which should be close to the Internet speed your ISP quoted you when you signed up for service.

The maximum throughput on larger files will be indicative of how fast you can download large files such as games (GTA V is almost 100 GB to download!) or the maximum quality that you can stream video on (lower download speed means you have to use a lower resolution to get continuous playback). We only increase download file sizes up to the absolute minimum required to get accurate measurements: no wasted bandwidth.

Upload is the opposite of download: we send data from your browser to the Internet. This metric is more important nowadays with many people working from home: it directly affects live video conferencing. A faster upload speed means your microphone and video feed can be of higher quality, meaning people can see and hear you more clearly on videoconferences.

Measurements for upload operate in the same manner: we progressively try to upload larger and larger files up until the point we notice your connection is saturated.

Speed measurements are never 100% consistent, which is why we repeat them. An easy way for us to report your speed would be to simply report the fastest speed we see. The problem is that this will not be representative of your real-world experience: latency and packet loss constantly fluctuates, meaning you can't expect to see your maximum measured performance all the time.

To compensate for this, we take the 90th percentile of measurements, or p90 and report that instead of the absolute maximum speed that we measured. Taking the 90th percentile is a more accurate representation in that it discounts peak outliers, which is a much closer approximation of what you can expect in terms of speeds in the real world.

Download and upload are important metrics but don't paint the entire picture of the quality of your Internet connection. Many of us find ourselves interacting with work and friends over videoconferencing software more than ever. Although speeds matter, video is also very sensitive to the latency of your Internet connection. Latency represents the time an IP packet needs to travel from your device to the service you're using on the Internet and back. High latency means that when you're talking on a video conference, it will take longer for the other party to hear your voice.

But, latency only paints half the picture. Imagine yourself in a conversation where you have some delay before you hear what the other person says. That may be annoying but after a while you get used to it. What would be even worse is if the delay differed constantly: sometimes the audio is almost in sync and sometimes it has a delay of a few seconds. You can imagine how often this would result into two people starting to talk at the same time. This is directly related to how stable your latency is and is represented by the jitter metric. Jitter is the average variation found in consecutive latency measurements. A lower number means that the latencies measured are more consistent, meaning your media streams will have the same delay throughout the session.

We've designed speed.cloudflare.com to be as transparent as possible: you can click into any of the measurements to see the average, median, minimum, maximum measurements, and more. If you're interested in playing around with the numbers, there's a download button that will give you the raw results we measured.

The entire speed.cloudflare.com backend runs on Workers, meaning all logic runs entirely on the Cloudflare edge and your browser, no server necessary! If you're interested in seeing how the benchmarks take place, we've open-sourced the code, feel free to take a peek on our Github repository.

We hope you'll enjoy adding this tool to your set of network debugging tools. We love being transparent and our tools reflect this: your network performance is more than just one number. Give it a whirl and let us know what you think.

As we wrapped up the first quarter of 2020, we set out to understand if and how DDoS attack trends have shifted during this unprecedented time of global shelter in place. Since then, traffic levels have increased by over 50% in many countries, but have DDoS attacks increased as well?

Traffic increases are often observed during holiday seasons. During holidays, people may spend more time online; whether shopping, ordering food, playing online games or a myriad of other online activities. This higher usage translates into higher revenue per minute for the companies that provide those various online services.

Downtime or service degradation during these peak times could result in user churn and loss of significant revenue in a very short time. ITIC estimates that the average cost of an outage is $5,600 per minute, which extrapolates to well over $300K per hour. It is therefore no surprise that attackers capitalize on the opportunity by launching a higher number of DDoS attacks during the holiday seasons.

The current pandemic has a similar cause and effect. People are forced to stay home. They have become more reliant on online services to accomplish their daily tasks which has generated a surge in the Internet traffic and DDoS attacks.

Most of the attacks that we observed in Q1 2020 were relatively small, as measured by their bit rates. As shown in the figure below, in Q1 2020, 92% of the attacks were under 10 Gbps, compared to 84% in Q4 2019.

Diving deeper, an interesting shift can be observed in the distribution of attacks below 10 Gbps in Q1, as compared to the previous quarter. In Q4, 47% of network-layer DDoS attacks peaked below 500 Mbps, whereas in Q1 they increased to 64%.

From a packet rate perspective, the majority of the attacks peaked below 1 million packets per second (pps). This rate, along with their bit rate, indicates that attackers are no longer focusing their efforts and resources to generate high-rate floods -- bits or packets per second.

However, it's not only the packet and bit rates that are decreasing, but also the attack durations. The figure below illustrates that 79% of DDoS attacks in Q1 lasted between 30 to 60 minutes, compared to 60% in Q4, which represents a 19% increase.

These three trends could be explained by the following:

• Launching DDoS attacks is cheap and you don’t need much technical background. DDoS-as-a-service tools have provided a possible avenue for bad actors with little to no technical expertise to launch DDoS attacks quickly, easily, in a cost-effective manner and with limited bandwidth. According to Kaspersky, DDoS attack services can cost as little as $5 for a 300-second attack (5 minutes). Additionally, amateur attackers can also easily leverage free tools to generate floods of packets. As we'll see in the next section, almost 4% of all DDoS attacks in Q1 were generated using variations of the publicly available Mirai code

• While an attack under 10 Gbps might seem small, it can still be enough to affect underprotected Internet properties. Smaller and quicker attacks might prove to deliver a high ROI for attackers to extort a ransom from companies in lieu of not disrupting the availability of the Internet property.

While the majority of the attacks were under 10 Gbps, larger attacks are still prevalent. The below graph shows a trend in the largest bit-rate of network-layer DDoS attacks that Cloudflare has observed and mitigated in Q4 2019 and Q1 2020. The largest attack for the quarter was observed during March and peaked just above 550 Gbps.

A persistent attacker is one that does not give up when their attacks fail; they try and try again. They launch multiple attacks on their target, often utilizing multiple attack vectors. In the Q4 2019 holiday season, attackers persisted and launched as many as 523 DDoS attacks in one day against a single Cloudflare IP. Each Cloudflare IP under attack was targeted by as many as 4.6 DDoS attacks every day on average.

During Q1, as the world entered COVID-19 lockdown, we observed a significant increase in the number of attacks compared to the monthly average. The last time we saw such an increase was in the Q4 2019 holiday season. However, an interesting difference is that attackers seem less persistent now than during the holidays. In Q1 2020, the average persistence rate dropped as low as 2.2 attacks per Cloudflare IP address per day, with a maximum of 311 attacks on a single IP; 40% less than the previous holiday quarter.

Throughout the past two quarters, the average number of attack vectors employed in DDoS attacks per IP per day has been mostly steady at approximately 1.4, with a maximum of 10.

Over the past quarter, we've seen over 34 different types of attack vectors on L3/4. SYN attacks formed the majority (60.1%) in Q1, followed by ACK attacks with 12.4%, and in third place, CLDAP (5.3%). Together, SYN & ACK DDoS attacks (TCP) form 72% of all L3/4 attack vectors in Q1.

The number of DDoS attacks in March 2020 increased as compared to January and February. Attackers found the crisis period to be an opportune time to launch an increased number of DDoS attacks, as illustrated below.

Furthermore, as various government authorities started mandating lockdowns and shelter-in-place orders, attackers resorted to increasing the number of large-sized attacks in the latter half of March. There were 55% more attacks observed in the second half of month (March 16-31) as compared to the first half (March 1-15). Additionally, 94% of attacks peaking at 300-400 Gbps were launched in the month of March.

With the ever shifting DDoS landscape, it is important to have a DDoS protection solution which is comprehensive and adaptive. In context with the attack insights illustrated above, here’s how Cloudflare stays ahead of these shifts to protect our customers.

• As attacks shrink in rate and duration, Time To Mitigate SLAs as long as 15 minutes provided by legacy vendors are just not practical anymore. Cloudflare mitigates network layer DDoS attacks under 10 seconds in most cases, which is especially critical for the increasingly shorter attacks. Read more about the recent enhancements to our DDoS detection and mitigation systems that allow us to automatically detect and mitigate DDoS attacks so quickly at scale.

• An increasing number of DDoS attacks are localized, which implies that legacy DDoS solutions which adopt a scrubbing center approach are not a feasible solution, as they are limited in their global coverage as well as act as a choke point, as DDoS traffic needs to be hauled back and forth from them. Cloudflare’s unique distributed architecture empowers every one of its data centers, spanning across 200 cities globally, to provide full DDoS mitigation capabilities.

• Large distributed volumetric attacks still exist and are employed by resourceful attackers when the opportunity is rife. An attack exceeding 1 Tbps can be expected in the future, so the ability to mitigate large DDoS attacks is a key aspect of today’s DDoS solution. Cloudflare has one of the most interconnected networks in the world with a capacity of over 35 Tbps which allows it to mitigate even the largest DDoS attacks. This massive network capacity concomitant with the globally distributed architecture allows Cloudflare to mitigate attacks, both small and large, closer to the source.

To learn more about Cloudflare’s DDoS solution contact us or get started.

Cloudflare is watching carefully as Internet traffic patterns around the world alter as people alter their daily lives through home-working, cordon sanitaire, and social distancing. None of these traffic changes raise any concern for us. Cloudflare's network is well provisioned to handle significant spikes in traffic. We have not seen, and do not anticipate, any impact on our network's performance, reliability, or security globally.

That holds true today; our network is performing as expected under increased load. Overall the Internet has shown that it was built for this: designed to handle huge changes in traffic, outages, and a changing mix of use. As we are well into April I thought it was time for an update.

Here's a chart showing the relative change in Internet use as seen by Cloudflare since the beginning of the year. I've calculated moving average of the trailing seven days for each country and use December 29, 2019 as the reference point.

On this chart the highest growth in Internet use has been in Portugal: it's currently running at about a 50% increase with Spain close behind followed by the UK. Italy flattened out at about a 40% increase in usage towards the end of March and France seems to be plateauing at a little over 30% up on the end of last year.

It's interesting to see how steeply Internet use grew in the UK, Spain and Portugal (the red, yellow and blue lines rise very steeply), with Spain and Portugal almost in unison and the UK lagging by about two weeks.

Looking at some other major economies we see other, yet similar patterns.

Similar increases in utilization are seen here. The US, Canada, Australia and Brazil are all running at between 40% and 50% the level of use at the beginning of the year.

We measure the TCP RTT (round trip time) between our servers and visitors to Internet properties that are Cloudflare customers. This gives us a measure of the speed of the networks between us and end users, and if the RTT increases it is also a measure of congestion along the path.

Looking at TCP RTT over the last 90 days can help identify changes in congestion or the network. Cloudflare connects widely to the Internet via peering (and through the use of transit) and we connect to the largest number of Internet exchanges worldwide to ensure fast access for all users.

Cloudflare is also present in 200 cities worldwide; thus the TCP RTT seen by Cloudflare gives a measure of the performance of end-user networks within a country. Here's a chart showing the median and 95th percentile TCP RTT in the UK in the last 90 days.

What's striking in this chart is that despite the massive increase in Internet use (the grey line), the TCP RTT hasn't changed significantly. From our vantage point UK networks are coping well.

Here's the situation in Italy:

The picture here is slightly different. Both median and 95th percentile TCP RTT increased as traffic increased. This indicates that networks aren't operating as smoothly in Italy. It's noticeable, though, that as traffic has plateaued the TCP RTT has improved somewhat (take a look at the 95th percentile) indicating that ISPs and other network providers in Italy have likely taken action to improve the situation.

This doesn't mean that Italian Internet is in trouble, just that it's strained more than, say, the Internet in the UK.

The Internet has seen incredible, sudden growth in traffic but continues to operate well. What Cloudflare sees reflects what we've heard anecdotally: some end-user networks are feeling the strain of the sudden change of load but are working and helping us all cope with the societal effects of COVID-19.

It's hard to imagine another utility (say electricity, water or gas) coping with a sudden and continuous increase in demand of 50%.

The Internet has been vital to our response to the COVID-19 crisis: enabling researchers to communicate with the rest of the world, connecting resources with people who need them, and sharing data about the spread.

It’s been amazing to see some of the projects people have stood up on Cloudflare Workers to assist during this crisis. Workers allows you to get set up in minutes, it’s fast and scalable out of the box, and there’s no infrastructure to maintain or scale, which is great if you want to create a project quickly.

To support critical web projects that help in the fight against the COVID-19 pandemic, we’re giving free access to our Cloudflare Workers compute platform through Project Galileo. We believe sites, apps, APIs, and tools that can help people with COVID-19 are exactly the type of critically important projects that Project Galileo was designed to support.    

One of the earliest impacts of the COVID-19 crisis was the switch that many organizations made to a fully remote model. As that happened, and we realized that many organization’s VPNs were not up to the task of scaling to support this increased load, Cloudflare made Cloudflare for Teams free through at least September 1, 2020.

If you’re working on a COVID-19 related project, follow the Project Galileo link and submit a request — we’ll get back to you as quickly as we can. And if you’re interested in getting started with Workers, there are some links at the bottom of the post that will help.

Amidst all the devastating news, it’s been really inspiring to see  developers jump in and build tools to help the public through the pandemic. We are excited to share a few of the stories they’ve shared with us.

API-COVID-19-In, built by Amod Malviya, is an API for tracking COVID-19 cases in India, sourced from The Ministry of Health and Family Welfare and separately from unofficial sources.

"I created api-covid19-in to make it easier for people working all over India to contribute to fighting this situation — be it by creating mass transparency (the aggregate data API), or detecting patterns (the crowd sourced patient data API), or planning (hospital beds API)".

• Very simple to be up & running. From the first code being written, to being up and running happened in less than an hour.

• Better than the alternatives of maintaining an Origin (higher cost), or exposing it via Github pages (can't do compute on every call).

• Not having to be worried about scaling or performance.

A few weeks ago, a Belgian grassroots movement of makers started to brainstorm on how they can fight the COVID-19 crisis. One of the projects is MakeFaceMasks. They have created a DIY manual to sew masks, which has been approved by the Belgian Government.

• We could automate our development/translation flow. This allowed us to quickly generate translated versions of the website.

• Websites are deployed automatically with Github Actions.

• Handle the load: the day we launched we immediately attracted 100,000 unique visitors with any downtime.

Mask a Hero NY is a volunteer-run site that matches medical professionals that need Personal Protective Equipment (PPE) during the COVID-19 pandemic with people that can donate it.

"We launched it about 2 weeks ago. The COVID-19 situation in New York is very worrying. My friends that are doctors are doing everything they can to help and they saw a lot of people on Facebook groups offering to donate small amounts of PPE, but it was hard for these people to know where it was needed the most and coordinate pickups. So my friends reached out to me to see if I could help. I pulled in my colleague MJ, and we designed and built the site in about 2 days.

The site has been a big success so far. It has facilitated over 27,000 mask donations already with a lot more to come this week. It's been featured on NBC News, CBS, MSNBC, on Katie Couric's social media and newsletter, some NY-area newspapers, and more. That matters because each feature has been followed by an increase in donation submissions. The site has facilitated donations to a variety of large and small hospitals and medical departments that are feeling the strain during this time. We're really proud of the impact so far but want to do even more to help these medical professionals."

"When we built the site, we wanted the absolute easiest and most straightforward tech stack. It's a 4-page site with no dynamic information. A static site generator was the obvious choice, so I chose Jekyll. Then for hosting, the last thing I want to deal with on a static site is complex server configuration and uptime. Workers Sites is super easy to deploy - I just run wrangler publish after a Jekyll build. Workers Sites handles cache breaking and has Cloudflare's caching built-in. Most importantly, I don't have to worry about the site going down. We've seen big traffic spikes after being featured in the media. The site has never gotten slower and I don't have to worry. Cloudflare Workers Sites lets us concentrate on helping the people that need it instead of spending time managing hosting."

The COVID Tracking Project collects and publishes the most complete testing data available for US states and territories. The project emerged from a tweet of a Google Sheets spreadsheet, where someone was keeping tabs on the testing from each state.

“I had been making something similar but Jeff Hammerbacher had a more complete version. After Jeff combined forces with Alexis Madrigal I thought it best to use the data they had. Since we’ve used Google Sheets to power websites in the past I thought I should spin up a quick service that fetches the sheet data from Google and make it available as JSON for people to use.”  

“Google often requires an API key or has some strange formatting. I just wanted an array of values that reflected the sheet rows. No long complicated URL. I picked Cloudflare Workers because it works really well as a serverless proxy.

At first the Worker was just a simple proxy, making an API request for every Worker request. Then I added cf: { cacheEverything: true, cacheTtl: 120 } to the fetch() options so Cloudflare could cache the fetch result. Caching the source is great but still requires having to decode, modify and serialize on every request. Some endpoints requested XML from AWS. Since it takes some time to parse really big XML strings we started seeing errors that the process was taking longer than 50ms CPU time. Cloudflare had to (generously) increase our limits to keep things running smoothly.

Not wanting consumers of our API to be kept waiting while the servers crunched the data on every request we started using Cloudflare Key Value storage for saving the parsed and serialized result. We put a TTL limit (like an hour) on every file saved to the KV store. On a new request we return the previous generated result from the cache first and then lookup the TTL of the item and if it’s more than 5 minutes old we make a new request and save it to the cache for next time. This way the user gets a fast result before we update an entry. If no user makes a request for an hour the cached item expires and the next request has to wait for a full process before response but that doesn’t happen for the popular endpoint/query options.”

If you’re building a resource to help others cope with COVID-19, and getting started with Workers, below are a few resources to get you started:

• Workers Sites: allows you to deploy your static site directly to Cloudflare’s network, with a few simple commands. Get started with our tutorial, or video.

• Tutorials: check out our tutorials to get started with Workers. We’ve highlighted a couple below that we think might be especially useful to you:

• Localize a website: make your website accessible to an even greater audience by translating it to other languages.

• Chat bot: with more people using chat for remote communication, chat bots can be a great way to make information more easily accessible at the public’s fingertips.

• Template gallery: our template gallery is designed to help you build with Workers by providing building blocks such as code snippets and boilerplate. For example, if you are writing an API, we suggest getting started using our Apollo GraphQL server boilerplate.

• HTMLRewriter API: the HTMLRewriter is a streaming HTML parser with an easy to use selector based JavaScript API for DOM manipulation, available in the Cloudflare Workers runtime. With so much disparate information on the web, many services that provide data about COVID-19 rely on scraping and aggregating data from multiple sources. See an example of the HTMLRewriter in action here to learn how you can use it to extract information from the web.

• Want to help, but not sure what to build? Our Built with Workers gallery features projects utilizing Workers today to give you an idea of the possibilities of what you can build with Workers.

These changes in daily life are showing up as changes in patterns of Internet use around the world. In this blog post I take a look at changing patterns in northern Italy, South Korea and the Seattle area of Washington state.

To understand how Internet use is changing, it’s first helpful to start with what a normal pattern looks like. Here’s a chart of traffic from our Dallas point of presence in the middle of January 2020.

This is a pretty typical pattern. If you look carefully you can see that Internet use is down a little at the weekend and that Internet usage is diurnal: Internet use drops down during the night and then picks up again in the morning. The peaks occur at around 2100 local time and the troughs in the dead of night at around 0300. This sort of pattern repeats worldwide with the only real difference being whether a peak occurs in the early morning (at work) or evening (at home).

Now here’s Seattle in the first week of January this year. I’ve zoomed in to a single week so we see a little more of the bumpiness of traffic during the day but it’s pretty much the same story.

Now let’s zoom out to the time period January 15 to March 12. Here’s what the chart looks like for traffic coming from Cloudflare’s Seattle PoP over that period (the gaps in the chart are just missing data in the measurement tool I’m using).

Focus in on the beginning of the chart. Looks like the familiar diurnal pattern with quieter weekends. But around January 30 something changes. There’s a big spike of traffic and traffic stays elevated. The weekends aren’t so clear either. The first reported case of COVID-19 was on January 21 in the Seattle area.

Towards the end of February, the first deaths occurred in Washington state. In early March a small number of employees of Facebook, Microsoft and Amazon in the Seattle area were confirmed to be infected. At this point, employers began encouraging or requiring their staff to work from home. If you focus on the last part of the chart and compare it with the first two things stand out: Internet usage has grown greatly and the night time troughs are less evident. People seem to be using the Internet more and for more hours.

Throughout the period there are also days with double spikes of traffic. If I zoom into the period March 5 to March 12 it’s interesting to compare with the week in January above.

Firstly, traffic is up about 40% and nighttime troughs are now above the levels seen in January during the day. The traffic is also spiky and continues through the weekend at similar levels to the week.

Next we can zoom in on traffic to residential ISPs in the Seattle area. Here’s a chart showing the first three days of this week (March 9 to March 11) compared to Monday to Wednesday a month prior in early February (February 10 to February 12).

Traffic to residential ISPs appears to be up about 5% month on month during the work day. We might have expected this to be higher given the number of local companies asking employees to work from home but many appear to be using VPNs that route all Internet traffic back through the corporate gateway.

Turning to Italy, and in particular northern Italy, where there has been a serious outbreak of COVID-19 leading to first a local quarantine and then a national one. Most of the traffic in northern Italy is served from our Milan point of presence.

For reference here’s what traffic looked like the first week in January.

A familiar pattern with peak traffic typically in the evening. Here’s traffic for March 5 to 12.

Traffic has grown by more than 30% with Internet usage up at all hours of the day and night. Another change that’s a little harder to see is that traffic is ramping up earlier in the morning than in early January. In early January traffic started rising rapidly at 0900 UTC and reach the daytime plateaus you see above around 1400 UTC. In March, we see the traffic jump up more rapidly at 0900 UTC and reach a first plateau before tending to jump up again.

Drilling into the types of domains that Italians are accessing we see changes in how people are using the Internet. Online chat systems are up 1.3x to 3x of normal usage. Video streaming appears to have roughly doubled. People are accessing news and information websites about 30% to 60% more and online gaming is up about 20%.

One final look at northern Italy. Here’s the period that covers the introduction of the first cordon sanitaire in communes in the north.

The big spike of traffic is the evening of Monday, February 24 when the first cordons sanitaire came into full effect.

Here’s the normal traffic pattern in Seoul, South Korea using the first week of January as an example of what traffic looked like before the outbreak of COVID-19:

And here’s March 5 to 12 for comparison:

There’s no huge change in traffic patterns other than that Internet traffic seen by Cloudflare is up about 5%.

Digging into the websites and APIs that people are accessing in South Korea shows some significant changes: traffic to websites offering anime streaming up over 2x, online chat up 1.2x to 1.8x and online gaming up about 30%.

In both northern Italy and South Korea traffic associated with fitness trackers is down, perhaps reflecting that people are unable to take part in their usual exercise, sports and fitness activities.

Cloudflare is watching carefully as Internet traffic patterns around the world alter as people alter their daily lives through home-working, cordon sanitaire, and social distancing. None of these traffic changes raise any concern for us. Cloudflare's network is well provisioned to handle significant spikes in traffic. We have not seen, and do not anticipate, any impact on our network's performance, reliability, or security globally.