Cloudflare incident on June 20, 2024
2024-06-26
A new DDoS rule resulted in an increase in error responses and latency for Cloudflare customers. Here’s how it went wrong, and what we’ve learned...
Continue reading »
Major data center power failure (again): Cloudflare Code Orange tested
2024-04-08
Just four months after a complete power outage at a critical data center we were hit with the exact same scenario. Here’s how we did this time, and what’s next...
Post mortem on the Cloudflare Control Plane and Analytics Outage
2023-11-04
Beginning on Thursday, November 2, 2023, at 11:43 UTC Cloudflare's control plane and analytics services experienced an outage. Here are the details...
Cloudflare incident on October 30, 2023
2023-11-01
Multiple Cloudflare services were unavailable for 37 minutes on October 30, 2023, due to the misconfiguration of a deployment tool used by Workers KV....
How Cloudflare mitigated yet another Okta compromise
2023-10-20
On Wednesday, October 18, 2023, we discovered attacks on our system that we were able to trace back to Okta. We have verified that no Cloudflare customer information or systems were impacted by this event because of our rapid response. ...
MORE POSTS
October 04, 2023 7:40 PM
1.1.1.1 lookup failures on October 4, 2023
On 4 October 2023, Cloudflare experienced DNS resolution problems. Some users may have received SERVFAIL DNS responses to valid queries. In this blog, we’re going to talk about what the failure was, why it occurred, and what we’re doing to make sure this doesn’t happen again...
- By
August 02, 2023 1:05 PM
Hardening Workers KV
A deep dive into the recent incidents relating to Workers KV, and how we’re going to fix them...
- By
July 11, 2023 8:48 AM
Connection errors in Asia Pacific region on July 9, 2023
On July 9, 2023, users in the Asia Pacific region experienced connection errors due to origin DNS resolution failures to .com and .net TLD nameservers...
- By
January 25, 2023 3:47 AM
Cloudflare incident on January 24, 2023
Several Cloudflare services became unavailable for 121 minutes on January 24th, 2023 due to an error releasing code that manages service tokens. The incident degraded a wide range of Cloudflare products...
- By
October 26, 2022 3:02 AM
Partial Cloudflare outage on October 25, 2022
Today, a change to our Tiered Cache system caused some requests to fail for users with status code 530...
- By
August 09, 2022 3:56 PM
The mechanics of a sophisticated phishing scam and how we stopped it
Yesterday, August 8, 2022, Twilio shared that they’d been compromised by a targeted phishing attack. Around the same time as Twilio was attacked, we saw an attack with very similar characteristics also targeting Cloudflare’s employees...
- By
June 21, 2022 12:38 PM
Cloudflare outage on June 21, 2022
Today, June 21, 2022, Cloudflare suffered an outage that affected traffic in 19 of our data centers...
- By
April 05, 2022 12:57 PM
PIPEFAIL: How a missing shell option slowed Cloudflare down
This post tells the story of how a missing shell option called “pipefail” slowed Cloudflare down....
- By
March 22, 2022 4:57 PM
Cloudflare’s investigation of the January 2022 Okta compromise
Today at 03:30 UTC we learnt of a compromise of Okta. We use Okta internally for employee identity as part of our authentication stack. We have investigated this compromise carefully and do not believe we have been compromised as a result...
- By
January 26, 2022 6:34 PM
Incorrect proxying of 24 hostnames on January 24, 2022
On January 24, 2022, as a result of an internal product migration, 24 hostnames (including www.cloudflare.com) that were actively proxied through the Cloudflare global network were mistakenly redirected to the wrong origin...
- By
November 27, 2020 12:00 PM
A Byzantine failure in the real world
When we review design documents at Cloudflare, we are always on the lookout for Single Points of Failure (SPOFs). In this post, we present a timeline of a real-world incident, and how an interesting failure mode known as a Byzantine fault played a role in a cascading series of ev...
- By
August 30, 2020 8:55 PM
August 30th 2020: Analysis of CenturyLink/Level(3) outage
Today CenturyLink/Level(3), a major ISP and Internet bandwidth provider, experienced a significant outage that impacted some of Cloudflare’s customers as well as a significant number of other services and providers across the Internet. ...
- By
July 18, 2020 1:22 AM
Cloudflare outage on July 17, 2020
Today a configuration error in our backbone network caused an outage for Internet properties and Cloudflare services that lasted 27 minutes. We saw traffic drop by about 50% across our network....
- By
July 12, 2019 3:45 PM
Details of the Cloudflare outage on July 2, 2019
Almost nine years ago, Cloudflare was a tiny company and I was a customer not an employee. Cloudflare had launched a month earlier and one day alerting told me that my little site, jgc.org, didn’t seem to have working DNS any more. ...
- By
July 02, 2019 3:50 PM
Cloudflare outage caused by bad software deploy (updated)
Starting at 1342 UTC today we experienced a global outage across our network that resulted in visitors to Cloudflare-proxied domains being shown 502 errors (“Bad Gateway”). The cause of this outage was deployment of a single misconfigured rule within the Cloudflare Web Applicatio...
- By