Today, shortly after 21:00 UTC, on our internal operations chat there was a scary message from one of our senior support staff: "getting DNS resolution errors on support.cloudflare.com", at the same time as automated monitoring indicated a problem.
Details Behind Today's Internet Hacks
At 1:19pm (PDT) today, a researcher noticed that the New York Times' website wasn't loading. We know the New York Times tech team, so we sent an email to check in. A few minutes later, the CTO of the NYT called us back.
Today's Network Issue
Today at 16:13 UTC a large amount of traffic began hitting our Los Angeles data center. We have an in-house team that monitors our network 24x7x365 and immediately all their alarms went off.
Today's Outage Post Mortem
This morning at 09:47 UTC CloudFlare effectively dropped off the Internet. The outage affected all of CloudFlare's services including DNS and any services that rely on our web proxy.
Post Mortem: What Yesterday's Network Outage Looked Like
Yesterday, around 16:36 GMT, we had an interruption to our network services. The interruption was caused by a combination of factors. First, we had an upstream bandwidth provider with some network issues that primarily affected our European data centers.