The Four Critical Security Flaws that Resulted in Last Friday's Hack

A core value CloudFlare is that security information should be shared between organizations to make the entire Internet safer. That is how CloudFlare's systems work: if one site is attacked, data about that attack is immediately shared with the rest of the network so other sites can be safe. We believe that same core value should apply when we are the victim of the attack. That is why we immediately posted an incident report and have continue to update it as we learn more.

Writing that report wasn't fun, but I believe it is important to share the details of the event so others who may be affected can learn from the events that transpired last Friday. This is not the usual way for the security industry, but we believe it's the way the security industry should be. To that end, here's what we know about the hack.

There were four key security flaws that allowed the hack to happen:

1. AT&T was tricked into redirecting my voicemail to a fraudulent voicemail box;

2. Google's account recovery process was tricked by the fraudulent voicemail box and left an account recovery PIN code that allowed my personal Gmail account to be reset;

3. A flaw in Google's Enterprise Apps account recovery process allowed the hacker to bypass two-factor authentication on my CloudFlare.com address; and

4. CloudFlare BCCing transactional emails to some administrative accounts allowed the hacker to reset the password of a customer once the hacker had gained access to the administrative email account.

We are following up with AT&T to understand more about how the voicemail was redirected. That remains unsettling, but it is not surprising that a phone company's voicemail security procedures are lax. It is also unsettling that Gmail's account recovery process appears to still be vulnerable to the voicemail hack. That is troubling since it means if a hacker knows your phone number then your Gmail account may, at best, only be as secure as your voicemail PIN.

You can mitigate these risk if you are a user by enabling two-factor authentication, ideally relying on Google's Authenticator App rather than anything that passes through the phone company's network. While Google is advising otherwise, I have removed my phone number from all my Google accounts.

Google has publicly stated that the flaw in the Google Enterprise App account recovery process has been patched and you can no longer use it get around two-factor authentication. Again, since any security system is only as strong as its weakest link, we would recommend using an out-of-band authentication that doesn't rely on the phone company's network (e.g., Google Authenticator App, not SMS or voice verification).

Finally, CloudFlare has stopped BCCing password reset and other transactional messages to administrative accounts, closing that attack vector if an administrator's email account is compromised in the future. If you're doing that at your company, and a troubling number of companies do use email as a poor man's logs, you should stop. This incident is why.

The event, from start to finish, lasted less than 2 hours. The hackers were in my personal Gmail account for about 1 hour 35 minutes. They were in CloudFlare's email accounts for about 28 minutes, although likely interrupted several times as our ops team reset passwords and sessions. To better understand the hack, we put together the visual timeline below which is our best understanding of the events as they transpired. As we learn more, we'll update the information here and on the official incident report.