When it comes to managing Internet properties, the difference between a small technical hiccup and major incident is often a matter of speed. Proactive alerting plays a crucial role, which is why we were excited when we released HTTP Error Rate notifications — giving administrators visibility into when end users are experiencing errors.

But what if there are issues that don't show up as errors, like a sudden drop in traffic, or a spike?

Today, we're excited to announce Traffic Anomalies notifications, available to enterprise customers. These notifications trigger when Cloudflare detects unexpected changes in traffic, giving another valuable perspective into the health of your systems.

Unexpected changes in traffic could be indicative of many things. If you run an ecommerce site and see a spike in traffic that could be great news — maybe customers are flocking to your sale, or you just had an ad run on a popular TV show. However, it could also mean that something is going wrong: maybe someone accidentally turned off a firewall rule, and now you’re seeing more malicious traffic. Either way, you might want to know that something has changed.

Similarly, a sudden drop in traffic could mean many things. Perhaps it’s Friday afternoon and all of your employees are signing off and no longer accessing your company website. Or maybe a link to your site is broken, and now potential customers aren’t able to access it. You could be losing potential revenue every minute that traffic is low, so you’d want to know as soon as possible to investigate.

Calculating anomalies in time series datasets is difficult to do well. The simplest way to do it is to use basic thresholds. However, as we’ve previously blogged about, simple thresholds aren’t very accurate when trying to determine when things are actually going wrong. There are too many edge cases for them to work effectively.

Calculating anomalies in HTTP errors is relatively easy. We know that in general there should be a very low number of errors, so any spike is bad and therefore alertable. That’s why we use Service Level Objectives (SLOs) to calculate anomalies for our HTTP Error Rate notifications.

However, analyzing overall HTTP traffic behaves more similarly to Cloudflare Security Events: there’s some general baseline of events that is computed from historical trends. Any deviation from that baseline is alertable. Because of those similarities, we decided to use the same calculations for Traffic Anomalies notifications as we have previously used for Security Event notifications: z-scores. This involves comparing the current value to the average over a period of time. However, many standard deviations away from the average the current value is, is the z-score.

Plot of HTTP traffic against z-scores. The blue is the HTTP traffic, purple is the positive z-score bound of the traffic, and green is the negative z-score bound of the traffic

For Traffic Anomalies notifications, we’re comparing the traffic over the past 5 minutes (the short window) to the average of the traffic over the past 4 hours (the long window). Positive z-scores indicate a spike, and negative z-scores indicate a drop. If the current value is more than 3.5 standard deviations away from the average, we alert. We measure every 5 minutes, so we can alert on any traffic spike or drop in a timely fashion.

Green bucket is the long window and the red bucket is the short window

While our Security Event notifications only trigger when there is a spike in security events (a drop is almost always a good thing), in the case of Traffic Anomalies we send notifications for both spikes and drops. This is because a drop of HTTP traffic is likely indicative of a problem, and a surge could be good or bad.

As with Security Events, Traffic Anomalies notifications support minimum thresholds. This means that, even if we determine that an event is outside 3.5 standard deviations, if the number of events is insignificant, we don’t alert. A spike must be at least 200 requests, and a drop must fall by at least 200 requests. This makes the notifications less noisy, since we aren’t alerting on small spikes and drops.

Cloudflare stores sampled statistics on requests that go through its network in Clickhouse. Every minute, we take HTTP traffic from Clickhouse and store it in an instance of VictoriaMetrics, a time-series data storage solution. VictoriaMetrics gives us out-of-the-box algorithmic functions for free, and it has been a good fit for our use case. We chose VictoriaMetrics for a few reasons.

Firstly, it's easy to configure and operate. As a team, we want to optimize for low operational burden and VictoriaMetrics has been great thus far. Secondly, VictoriaMetrics has the ability to scale horizontally, meaning we can run it in a highly available mode. For a system such as this where we want something reliable to compute time sensitive information for our customers, the high availability requirement is essential. Finally, in our tests, we found that VictoriaMetrics used around ⅓ of the memory that Prometheus, a similar alternative product, did for the same use case.

Once we have data in VictoriaMetrics, we can run queries against it and determine whether we need to alert our customers or not, based on notification configurations they have created ahead of time. To do this we leverage our existing Alert Notification System, which we blogged about initially in 2019. We know we can count on our current notification system for the last mile to deliver these critical notifications to our customers.

Data flow from HTTP request to notification

Setting up the Notification

To configure this notification, navigate to the “Notifications” tab of the dashboard. Select Traffic Anomalies as your notification type. As with all Cloudflare notifications, you’re able to name and describe your notification, and choose how you want to be notified.

Traffic Anomalies notification in the Dashboard

You can choose which domains you want monitored for Traffic Anomalies, whether you want to include traffic that’s already been mitigated by Cloudflare DoS or WAF products, and whether there are specific status codes you want included or excluded. You can also choose whether you want to be alerted on traffic spikes, drops, or both.

We’re excited to use this system to help serve our Enterprise customers with invaluable notifications regarding the overall health of their systems. Head over to the Notifications tab in the dash to check this new notification out now!

Whether you’re a software engineer deploying a new feature, network engineer updating routes, or a security engineer configuring a new firewall rule: You need visibility to know if your system is behaving as intended — and if it’s not, to know how to fix it.

Cloudflare is committed to helping our customers get visibility into the services they have protected behind Cloudflare. Being a single pane of glass for all network activity has always been one of Cloudflare’s goals. Today, we’re outlining the future vision for Cloudflare observability.

Observability means gaining visibility into the internal state of a system. It’s used to give users the tools to figure out what’s happening, where it’s happening, and why.

At Cloudflare, we believe that observability has three core components: monitoring, analytics, and forensics. Monitoring measures the health of a system - it tells you when something is going wrong. Analytics give you the tools to visualize data to identify patterns and insights. Forensics helps you answer very specific questions about an event.

Observability becomes particularly important in the context of security to validate that any mitigating actions performed by our security products, such as Firewall or Bot Management, are not false positives. Was that request correctly classified as malicious? And if it wasn’t, which detection system classified it as such?

Cloudflare, additionally, has products to improve performance of applications and corporate networks and allow developers to write lightning fast code that runs on our global network. We want to be able to provide our customers with insights into every request, packet, and fetch that goes through Cloudflare’s network.

Analytics are fantastic for summarizing data, but how do you know when to look at them? No one wants to sit on the dashboard clicking refresh over and over again just in case something looks off. That’s where notifications come in.

When we talk about something “looking off” on an analytics page, what we really mean is that there’s a significant change in your traffic or network which is reflected by spikes or drops in our analytics. Availability and performance directly affect end users, and our goal is to monitor and notify our customers as soon as we see things going wrong.

Today, we have many different types of notifications from Origin Error Rates, Security Events, and Advanced Security Events to Usage Based Billing and Health Checks. We’re continuously adding more notification types to have them correspond with our awesome analytics. As our analytics get more customizable, our notifications will as well.

There’s tons of different algorithms that can be used to detect spikes, including using burn rates and z-scores. We’re continuing to iterate on the algorithms that we use for detections to offer more variations, make them smarter, and make sure that our notifications are both accurate and not too noisy.

So, you’ve received an alert from Cloudflare. What comes next?

Analytics can be used to get a birds eye view of traffic or focus on specific types of events by adding filters and time ranges. After you receive an alert, we want to show you exactly what’s been triggered through graphs, high level metrics, and top Ns on the Cloudflare dashboard.

Whether you’re a developer, security analyst, or network engineer, the Cloudflare dashboard should be the spot for you to see everything you need. We want to make the dashboard more customizable to serve the diverse use cases of our customers. Analyze data by specifying a timeframe and filter through dropdowns on the dashboard, or build your own metrics and graphs that work alongside the raw logs to give you a clear picture of what's happening.

Focusing on security, we believe analytics are the best tool to build confidence before deploying security policies. Moving forward, we plan to layer all of our security related detection signals on top of HTTP analytics so you can use the dashboard to answer questions such as: if I were to block all requests that the WAF identifies as an XSS attack, what would I block?

Customers using our enterprise Bot Management may already be familiar with this experience, and as we improve it and build upon it further, all of our other security products will follow.

Analytics are a powerful tool to see high level patterns and identify anomalies that indicate that something unusual is happening. We’re working on new dashboards, customizations, and features that widen the use cases for our customers. Stay tuned!

Logs are used when you want to examine specific details about an event. They consist of a timestamp and fields that describe the event and are used to get visibility on a granular level when you need a play-by-play.

In each of our datasets, an event measures something different. For example, in HTTP request logs, an event is when an end user requests content from or sends content to a server. For Firewall logs, an event occurs when the Firewall takes an action on an HTTP request. There can be multiple Firewall events for each HTTP request.

Today, our customers access logs using Logpull, Logpush, or Instant Logs. Logpull and Logpush are great for customers that want to send their logs to third parties (like our Analytics Partners) to store, analyze, and correlate with other data sources. With Instant Logs, our customers can monitor and troubleshoot their traffic in real-time straight from the dashboard or CLI. We’re planning on building out more capabilities to dig into logs on Cloudflare. We’re hard at work on building log storage on R2 - but what’s next?

We’ve heard from customers that the activity log on the Firewall analytics dashboard is incredibly useful. We want to continue to bring the power of logs to the dashboard by adding the same functionality across our products. For customers that will store their logs on Cloudflare R2, this means that we can minimize the use of sampled data.

If you’re looking for something very specific, querying logs is also important, which is where forensics comes in. The goal is to let you investigate from high level analytics all the way down to individual logs lines that make them up. Given a unique identifier, such as the ray ID, you should be able to look up a single request, and then correlate it with all other related activity. Find out the client IP of that ray ID and from there, use cases are plentiful: what other requests from this IP are malicious? What paths did the client follow?

Logs are really useful, but they don’t capture the context around a request. Traces show the end-to-end life cycle of a request from when a user requests a resource to each of the systems that are involved in its delivery. They’re another way of applying forensics to help you find something very specific.

These are used to differentiate each part of the application to identify where errors or bottlenecks are occurring. Let's say that you have a Worker that performs a fetch event to your origin and a third party API. Analytics can show you average execution times and error rates for your Worker, but it doesn’t give you visibility into each of these operations.

Using wrangler dev and console.log statements are really helpful ways to test and debug your code. They bring some of the visibility that’s needed, but it can be tedious to instrument your code like this.

As a developer, you should have the tools to understand what’s going on in your applications so you can deliver the best experience to your end users. We can help you answer questions like: Where is my Worker execution failing? Which operation is causing a spike in latency in my application?

Notifications, analytics, logs, and tracing each have their distinct use cases, but together, these are powerful tools to provide analysts and developers visibility. Looking forward, we’re excited to bring more and more of these capabilities on the Cloudflare dashboard.

We would love to hear from you as we build these features out. If you’re interested in sharing use cases and helping shape our roadmap, contact your account team!

Back in 2019, we blogged about our brand new Notification center as a centralized hub for configuring notifications on your account. Since then, we’ve talked a lot about new types of notifications you can set up, but not as much about updates to the notification platform itself. So what’s new with Notifications?

We know that notifications are incredibly important to our customers. Cloudflare sits in between your Internet property and the rest of the world. When something goes wrong, you want to know right away because it could have a huge impact on your end users. However, you don’t want to have to sit on the Cloudflare Dashboard all day, pressing refresh on analytics pages over and over just to make sure that you don’t miss anything important. This is where Notifications come in. Instead of requiring you to actively monitor your Internet properties, you want Cloudflare to be able to directly inform you when something might be going wrong.

Cloudflare has many different notification types to ensure that you don’t miss anything important. We have notifications to inform you that you’ve been DDoS’d, or that the Firewall is blocking more requests than normal, or that your origin is seeing high levels of 5xx errors, or even that your Workers script’s CPU usage is above average. We’re constantly adding new notifications, so make sure to check our Cloudflare Development Docs to see what’s new!

So we have all of these super great notifications, but how do we actually inform you of an event? The classic answer is “we email you.” All of our customers have the ability to configure notifications to send to the email addresses of their choosing.

However, email isn’t always the optimal choice. What happens when an email gets sent to spam, or filtered out into another folder that you rarely check? What if you’re a person who never cleans out their inbox and has four thousand unread emails that can drown out new important emails that come in? You want a way for notifications to go directly to the messaging platform that you check the most, whether that’s Slack or Microsoft Teams or Discord or something else entirely. For customers on our Professional, Business, and Enterprise plans, this is where webhooks come in.

Webhooks are incredibly powerful! They’re a type of API with a simple, standardized behavior. They allow one service (Cloudflare) to send events directly to another service. This destination service can be nearly anything: messaging platforms, data management systems, workflow automation systems, or even your own internal APIs.

While Cloudflare has had first class support for webhooking into Slack, Microsoft Teams, Google Chat, and customer’s own APIs for a while, we’ve recently added support for DataDog, Discord, OpsGenie, and Splunk as well. You can read about how to set up each of those types of webhooks in our Cloudflare Development Docs.

Because webhooks are so versatile, more and more customers are using them! The number of webhooks configured within Cloudflare’s notification system doubles, on average, every three months. Customers can configure webhooks in the Notifications tab in the dashboard.

Webhooks are cool, but they still leave room for error. What happens when you receive a notification but accidentally delete it? Or when someone new starts at your company, but you forget to update the notification settings to send to the new employee?

Before now, Cloudflare notifications were entirely point in time. We sent you a notification via your preferred method, and we no longer had any visibility into that notification. If that notification gets lost on your end, we don’t have any way to help recover the information it contained.

Notification history fixes that exact issue. Users are able to see a log of the notifications that were sent, when they were sent, and who they were sent to. Customers on Free, Professional, or Business plans are able to see notification history for the past 30 days. Customers on Enterprise plans are able to see notification history for the past 90 days.

Right now, notification history is only available via API, but stay tuned for updates about viewing directly in the Cloudflare Dashboard!

Our core application security features such as the WAF, firewall rules and rate limiting help keep millions of Internet properties safe. They all do so quietly without generating any notifications when attack traffic is blocked, as our focus has always been to stop malicious requests first and foremost.

Today, we are happy to announce a big step in that direction. Business and Enterprise customers can now set up proactive alerts whenever we observe a spike in firewall related events indicating a likely ongoing attack.

Alerts can be configured via email, PagerDuty or webhooks, allowing for flexible integrations across many systems.

You can find and set up the new alert types under the notifications tab in your Cloudflare account.

Two new notification types have been added to the platform.

This notification can be set up on Business and Enterprise zones, and will alert on any spike of firewall related events across all products and services. You will receive the alert within two hours of the attack being mitigated.

This notification can be set up on Enterprise zones only. It allows you to filter on the exact security service you are interested in monitoring and different notifications can be set up for different services as necessary. The alert will fire within five minutes of the attack being mitigated.

We’ve previously blogged about how accurately calculating anomalies in time series data sets is hard. Simple threshold alerting — “notify me if there are more than X events” — doesn’t work well. It takes a lot of work to tune the specific thresholds to be accurate, and even then you’re still likely to end up with false positives or missed events.

For Origin Error Rate notifications, we leaned on the methodology outlined in the Google SRE Handbook for alerting based on Service Level Objectives (SLOs). However, SLO alerting assumes that there is an established baseline. We know exactly what percentage of responses from your origin are “allowed” to be errors before something is definitely wrong. We don’t know what that percentage is for Firewall events. For example, Internet properties with many Firewall rules are more likely to have more Firewall events than Internet properties with few Firewall rules.

Instead of using SLO based alerting for Security Event notifications, we’re using Z-score calculations. The z-score methodology calculates how many standard deviations away from the mean a certain data point is. For Security Event notifications we can take the mean number of Firewall events for each distinct Internet property as the effective “baseline”, and compare the current number of Firewall events to see if there is a significant spike.

In this first iteration, a z-score threshold of 3.5 has been configured in the system and will be adjusted based on customer feedback. You can read more about the system in our WAF developer docs.

To configure these notifications, navigate to the Notifications tab of the dashboard and click “Add”. Select Security Events Alert or Advanced Security Events Alert.

As with all Cloudflare notifications, you’re able to name and describe your notification, and choose how you want to be notified. From there, you can select which domains you want to monitor.

For Advanced Security Event notifications, you can also select which services the notification should monitor. The log value in Firewall Event logs for each relevant service is also displayed in the event you are integrating directly with Cloudflare logs and wish to filter relevant events in your existing SIEMs.

Once the notifications have been set up, you can rely on Cloudflare to warn you whenever an anomaly is detected. An example email notification is shown below:

The alert provides details on the service detecting the events (in this case the WAF), the timestamp and the affected zone. A link is provided that will direct you to the Firewall Events dashboard filtered on the correct service and time range.

We are looking forward to customers setting up their notifications, so they can stay on top of any malicious activity affecting their applications.

This is just the first step of many towards building a much more comprehensive suite of notifications and incident management systems directly embedded in the Cloudflare dashboard. We look forward to posting feature improvements to our application security alert system in the near future.

Today, as part of Cloudflare’s Impact Week, we’re excited to announce a new tool to help you understand the environmental impact of operating your websites, applications, and networks. Your Carbon Impact Report, available today for all Cloudflare accounts, will outline the carbon savings of operating your Internet properties on Cloudflare’s network.

Everyone has a role to play in reducing carbon impact and reversing climate change. We shared today how we’re approaching this, by committing to power our network with 100% renewable energy. But we’ve also heard from customers that want more visibility into the impact of the tools they use (also referred to as “Scope 3” emissions) — and we want to help!

We’ve previously blogged about how Internet infrastructure affects the environment. At a high level, powering hardware (like servers) uses energy. Depending on its source, producing this energy may involve emitting carbon into the atmosphere, which contributes to climate change.

When you use Cloudflare, we use energy to power hardware to deliver content for you. But how does that energy we use compare to the energy it would take to deliver content without Cloudflare? As of today, you can go to the Cloudflare dashboard to see the (approximate) carbon savings from your usage of Cloudflare services versus Internet averages for your usage volume.

Most of the energy that Cloudflare uses comes from powering the servers at our edge to serve your content. We’ve outlined how we quantify the carbon impact of this energy in our emissions report. To determine the percentage of this impact derived from your Cloudflare usage specifically, we’ve used the following method:

When you use Cloudflare, data from requests destined to your Internet property goes through our edge. Data transfer for your Internet properties roughly represents a fraction of the energy consumed at Cloudflare’s edge. If we sum up the data transfer for your Internet properties and multiply that number by the energy it takes to power each request (derived from our emissions report and overall usage data), we can approximate the total carbon impact of powering your Internet properties with Cloudflare.

We already knew that delivering content takes some energy and therefore has some carbon impact. So how much energy does Cloudflare actually save you? To determine what your usage would look like without Cloudflare, we’ve used the following method:

Using public information on average data center energy usage and the International Energy Agency’s global average emissions for energy usage, we can calculate the carbon cost of data transfer through average (non-Cloudflare) networks. We can then compare these numbers to arrive at your carbon savings from using Cloudflare.

With our new Carbon Impact Report, available for all plans/users, we’ve given you this value for your account. It represents the carbon dioxide equivalent (CO2e) that you’ve saved as a result of using Cloudflare to serve requests to your Internet properties in 2020.

This raw number is great, but it isn’t the easiest to understand. What does a gram of carbon dioxide equivalent actually mean in practice? It’s not a unit of measurement most of us are used to seeing in our day-to-day lives. To make this number a little easier to digest, we’ve also provided a comparison to light bulbs.

Standard light bulbs are 60 watts, so we know that turning on a light bulb for an hour uses 0.06 kilowatt-hours of energy. According to the EPA, that’s about 42 grams of carbon dioxide equivalent. That means that if your carbon dioxide equivalent saving is 126 grams, that’s approximately the same impact as turning off a light bulb for three hours.

As explained in more detail here, Cloudflare purchases Renewable Energy Credits to account for the energy used by our network. This means that your use of Cloudflare’s services is powered by renewable energy.

Additionally, using Cloudflare helps you reduce your overall carbon footprint. Using Cloudflare’s cloud security and performance services such as WAF, Network Firewall, and DDoS mitigation allow you to decommission specialized hardware and transfer those functions to software running efficiently at our edge. This reduces your carbon footprint by significantly decreasing the energy used to operate your network stack, and improves your security, performance, and reliability along the way.

Optimizing your website also reduces your carbon footprint by requiring less energy for your end users to load a page. Using Cloudflare’s Image Resizing for visual content on your site to properly resize images reduces the energy it takes each of your end users to load a page, thus reducing downstream carbon emissions.

Lastly, since Cloudflare is a certified green host, any content you host on Pages or Workers KV is hosted green and certified powered by renewable energy.

This dashboard is just a first step in giving our customers transparent information on their carbon use, savings, and ideas for improvement with Cloudflare. Right now, you can view data on your carbon savings from 2020 (aligned with our 2020 emissions report). As we continue to iterate on how we measure carbon impact, we’re working toward providing dynamic information on carbon savings at a quarterly or even monthly granularity.

Have other ideas on what we can provide to help you understand and reduce the carbon impact of your Internet properties? Please reach out to us in the comments on this post or on social media!

We hope that this data helps you with your sustainability goals, and we’re excited to keep providing you with transparent information for 2021 and beyond.

Today we’re excited to announce Origin Error Rate notifications: a new, sophisticated way to detect and notify you when Cloudflare sees elevated levels of 5xx errors from your origin.

In 2019, we announced Passive Origin Monitoring alerts to notify you when your origin goes down. Passive Origin Monitoring is great — it tells you if every request to your origin is returning a 521 error (web server down) for a full five minutes. But sometimes that’s not enough. You don’t want to wait for all of your users to have issues; you want to be notified when more users than normal are having issues before it becomes a big problem.

No service is perfect — we know that a very small percentage of your origin responses are likely to be 5xx errors. Most of the time, these issues are one-offs and nothing actually needs to be done. However, for Internet properties with very high traffic, even a very small percentage could potentially be a large number. If we alerted you for every one of these errors, you would never stop getting useless notifications. When it comes to notifying, the question isn’t whether there are any errors, but how many errors need to exist before it’s a problem.

So how do we actually tell if something is a problem? As humans, it’s relatively easy for us to look at a graph, identify a spike, and think “hmm, that’s not supposed to be there.” For a computer it gets a little more complicated. Unlike humans, who have intuition and can exercise judgement in grey areas, a computer needs an exact set of criteria to tell whether something is out of the ordinary.

The simplest way to detect abnormalities in a time series dataset is to set a single threshold — for example, “notify me whenever more than 5% of the requests to my Internet properties result in errors.” Unfortunately, it’s really hard to pick an effective threshold. Too high and you never actually get notified; too low, and you’re drowning in notifications:

Even when we find that happy medium, we can still miss issues that burn “low and slow”. This is where there’s no obvious, dramatic spike, but something has been going a little wrong for a long time:

We can try layering on multiple thresholds. For example: notify you if the error rate is ever over 10%, or if it’s over 5% for more than five minutes, or if it’s over 2% for more than 10 minutes. Not only does this quickly become complicated, but it also doesn’t account for periodic issues, such as kubernetes pods restarting or deployments going out at a regular interval. What if the error rate is over 5% for only four minutes, but it happens every five minutes? We know that a lot of your end users are being affected, but even the long set of rules listed above wouldn’t catch it.

So thresholds probably aren’t sophisticated enough to detect origin issues. Instead, we turn to the Google SRE Handbook for alerting based on Service Level Objectives (SLOs). An SLO is a part of an agreement between a customer and a service provider. It’s a defined metric and value that both parties agree on. One of the most common types of SLOs is availability, or “the service will be available for a certain percentage of the time”. In this case, the service is your origin and the agreement is between you and your end users. Your end users expect your origin to be available a certain percent of the time.

If we revisit our original concept, we know that you’re comfortable with your origin returning a certain number of errors. We define that number as your SLO. An SLO of 99.9 means that you’re OK with 0.01% of all of your requests over a month being errors. Therefore, 0.01% of all the requests that reach your origin is your total error budget — you can have this many errors in a month and never be notified, because you’ve defined that as acceptable.

What you really want to know is when you’re burning through that error budget too quickly — this probably means that something is actually going wrong (instead of a one-time occurrence). We can measure a burn rate to gauge how quickly you’re burning through your error budget, given the rate of errors that we’re currently seeing. A burn rate of one means that the entirety of the error budget will be used up exactly within the set time period — an ideal scenario. A burn rate of zero is even better since we’re not seeing any errors at all, but ultimately is pretty unrealistic. A burn rate of 10 is most likely a problem — if that rate keeps up for the full month, you’ll have had 10x the number of errors than you originally said was acceptable.

Even when using burn rates instead of thresholds, we still want to have multiple criteria. We want to measure a short time period with a high burn rate (a short indicator). This covers your need to “alert me quickly when something dramatic is happening.” But we also want to have a longer time period with a lower burn rate (a long indicator), in order to cover your need to “don’t alert me on issues that are over quickly.” This way we can ensure that we alert quickly without sending too many false positives.

Let’s take a look at the life of an incident using this methodology. In our first measurement, the short indicator tells us it looks like something is starting to go wrong. However, the long indicator is still within reasonable bounds. We’re not going to sound the alarm yet.

When we measure next, we see that the problem is worse. Now we’re at the point where there are enough errors that not only is the short indicator telling us there’s something wrong, but the long indicator has been impacted too. We feel confident that there’s a problem, and it’s time to notify you.

A couple cycles later, the incident is over. The long indicator is still telling us that something is wrong, because the incident is still within the long time period. However, the short indicator shows that nothing is currently concerning. Since we don’t have both indicators telling us that something is wrong, we won’t notify you. This keeps us from sending notifications for incidents that are already over.

This methodology is cool because of how well it responds to different incidents. If the original spike had been more dramatic, it would have triggered both the long and short indicators immediately. The more errors we’re seeing, the more confident we are that there’s an issue and the sooner we can notify you.

Even with this methodology, we know that different services behave differently. So for this notification, you can choose the Service Level Objective (SLO) you want to use to monitor your Internet property: 99.9% (high sensitivity), 99.8% (medium sensitivity), or 99.7% (low sensitivity). You can also choose which Internet properties you want to monitor — no need to be notified for test properties or lower priority domains.

HTTP Origin Error Rate notifications can be configured in the Notifications tab of the dashboard. Select Origin Error Rate Alert as your alert type. As with all Cloudflare notifications, you’re able to name and describe your notification, and choose how you want to be notified. From there, you can select which domains you want to monitor, and at what SLO.

This notification is available to all Enterprise customers. If you’re interested in monitoring your origin, we encourage you to give it a try.

Our team is hiring in Austin, Lisbon and London.