Post Mortem

June 21, 2022 1:38PM

Today, June 21, 2022, Cloudflare suffered an outage that affected traffic in 19 of our data centers...

By 
• Tom Strickx
• ,  Jeremy Hartman

April 05, 2022 1:57PM

This post tells the story of how a missing shell option called “pipefail” slowed Cloudflare down....

By 
• Alex Forster

March 22, 2022 4:57PM

Today at 03:30 UTC we learnt of a compromise of Okta. We use Okta internally for employee identity as part of our authentication stack. We have investigated this compromise carefully and do not believe we have been compromised as a result...

By 
• John Graham-Cumming
• ,  Lucas Ferreira
• ,  Daniel Stinson-Diess

January 26, 2022 6:34PM

On January 24, 2022, as a result of an internal product migration, 24 hostnames (including www.cloudflare.com) that were actively proxied through the Cloudflare global network were mistakenly redirected to the wrong origin...

By 
• Jeremy Hartman

November 27, 2020 12:00PM

When we review design documents at Cloudflare, we are always on the lookout for Single Points of Failure (SPOFs). In this post, we present a timeline of a real-world incident, and how an interesting failure mode known as a Byzantine fault played a role in a cascading series of events....

By 
• Tom Lianza
• ,  Chris Snook

August 30, 2020 9:55PM

Today CenturyLink/Level(3), a major ISP and Internet bandwidth provider, experienced a significant outage that impacted some of Cloudflare’s customers as well as a significant number of other services and providers across the Internet....

By 
• Matthew Prince

July 18, 2020 2:22AM

Today a configuration error in our backbone network caused an outage for Internet properties and Cloudflare services that lasted 27 minutes. We saw traffic drop by about 50% across our network....

By 
• John Graham-Cumming

July 12, 2019 4:45PM

Almost nine years ago, Cloudflare was a tiny company and I was a customer not an employee. Cloudflare had launched a month earlier and one day alerting told me that my little site, jgc.org, didn’t seem to have working DNS any more....

By 
• John Graham-Cumming

July 02, 2019 4:50PM

Starting at 1342 UTC today we experienced a global outage across our network that resulted in visitors to Cloudflare-proxied domains being shown 502 errors (“Bad Gateway”). The cause of this outage was deployment of a single misconfigured rule within the Cloudflare Web Application Firewall (WAF)...

By 
• John Graham-Cumming