Keeping our users safe

To everyone in Cloudflare, account security is one of our most important tasks. We recognize that to every customer on our platform, we are critical infrastructure. We also know that the simplest attacks often lead to the most devastating of outcomes. Most people think that if they are going to get hacked it will be by some clever ”zero day”. The reality couldn’t be farther from the truth.

Attackers are smart and they have realized that even in 2018, the human is still the weakest link in the chain. The 2017 Verizon breach report identified that 81% of hacking related breaches occurred as a result of weak credentials or credential theft, an increase from the 63% reported in 2016’s breach report.

Source: Verizon 2017 data breach report

Your credentials are as important as your house or car keys. If someone copies or steals them, the repercussions can be catastrophic. If you suspect someone has access to your house keys you change your locks. If you aren’t fast enough, someone might break in.

Likewise if you realize that someone might have access to your password, the remedy is to change it. Too often, as with house keys, we are slow to change our passwords. This is why we see so many account compromises happen after a big public breach like Yahoo. As soon as the attacker gets their hands on a cache of credentials from a breach they immediately try them against every other account the user has. They know that a very high percentage of users still reuse the same credentials across multiple sites. So their chances of success are high - sometimes years after a breach has happened.

This is especially problematic with API keys because they often leak without anyone ever knowing it’s happened. Common examples include attackers who steal API keys by reversing client software, or using malicious code inside the browser to navigate and steal secrets - often long after the user has logged out. Some banking trojans such as the infamous Zeus even continue to browse after you have logged out, and display fake balance information to hide withdrawals it makes. This is why many banks now force you to re-authenticate often with 2FA when you log in and again when you perform any kind of financial transaction.

The Trojan Horse. (It’s not a new attack)

Attackers realize this too and they have been steadily upping their game to catch unsuspecting users. Not content to just sit and wait, attackers constantly try to grab credentials through a combination of trickery and sophisticated software attacks. The one thing that links all of this? It’s you that they are attacking, not some server sitting in a datacenter.

Phishing emails remain the most popular method but they have now evolved and range from the mundane badly spelled email asking for your password or offering you a link to some unknown site...

Common Cloudflare phishing email

A more sophisticated Cloudflare phishing email

Stepping up from phishing emails, we now have to face malicious software hidden inside innocuous everyday packages from browser extensions to free games. The most popular and difficult to detect of these by far are the malicious browser extensions. Often these start out as legitimate browser tools, then either they get hacked, or an attacker simply buys a forgotten extension project and injects malicious code into it. Now, anything your browser can see, the malicious extension can also see.

The most sophisticated of these are often targeted at a particular institution and know how to silently navigate to the credentials they wish to steal. Over the past couple of years we have seen several of these targeted specifically at Cloudflare customers. Below is an example of one such campaign from late last year involving a very popular Chrome extension called “Web Developer”.

Web Developer for Chrome breach alert

The malicious code injected into the extension was designed to be as stealthy and as resilient to attempts to stop it as possible.

• First it checks to make sure its been installed for at least 10 minutes.

• If it determines the coast is clear, it connects to a machine generated domain name, also called a DGA or “Domain Generation Algorithm” domain. These are random seeming domain names that are actually made by an algorithm so that they are harder to find and the attacker can automatically change to a new domain if the old one gets shut down, without changing any code. On August 2nd 2017 the DGA domain for this malicious extension was “wd7bdb20e4d622f6569f3e8503138c859d[.]win”. By August 3rd it had changed to “wd8a2b7d68f1c7c7f34381dc1a198465b4[.]win” as you can imagine this makes predicting new domains very hard unless you break the code behind the DGA algorithm.

• If the connection is successful it downloads a fresh payload, ga.js over HTTPS which is meant to fool anyone that sees it into thinking that it is downloading Google analytics.

Example code downloaded by the malicious version of Web Developer

The Cloudflare API Key stealing payload downloaded by the malicious extension.

Web Developer for Chrome wasn’t the only extension compromised in that particular campaign. Other extensions compromised included

• Chrometana – Version 1.1.3

• Infinity New Tab – Version 3.12.3

• CopyFish – Version 2.8.5

• Web Paint – Version 1.2.1

• Social Fixer 20.1.1 affected

• TouchVPN

• Betternet VPN

All these extensions have since been updated, but anyone with an older version should consider themselves at risk. How did this happen? Phishing of course:

Phishing email sent to Chrome developer a9t9.com

If you want to read more about this, our friends over at proofpoint have an in depth teardown of all the other aspects of this particular campaign.

System security

We have built our systems to be secure by design. In some cases, this is as simple as ensuring that sensitive data is restricted, or made completely unobtainable. In other cases this has meant building systems in a way that makes them secure against a wide range of common attacks.

Passwords We store user passwords in a secure database using a complex, salted hash that uses the blowfish based bcrypt() hashing algorithm.

API Keys We ensure that API keys are unique by generating them using a combination of AES (Rijndael 256), SHA256 and plenty of entropy. Once generated through these hashing algorithms, API Keys are also stored in a secure database that only a handful of people have access to.

At the back-end

All calls to these sensitive databases are audited, and stored in logs that go back to the very beginning of Cloudflare. In one recent audit exercise we were able to review, and determine the exact time an API Key was generated for a customer in 2013. Access to both the audit logs and critical systems, like our databases, is restricted to a handful of senior production engineers and security staff. All staff access is also logged, and stored securely for audit purposes.

Finally, all programmatic calls to these databases are made through stored procedures linked to dedicated accounts. Dynamic SQL is not permitted.

In Transit

All connections are made over HTTPS and sensitive tokens or credentials are never exposed in transit.

In the User InterfaceTo access your dashboard you need your account email, your password, and assuming you enabled it, (you really should if you haven’t) your 2FA code. If all of this is correct, and the IP you are using is one you are known to use, you are logged in. If the IP address isn’t known we send an email alert to your registered email address alerting you to the event.

Screenshot of an IP alert generated by accessing the dashboard

If your account is Pro, Business or Enterprise, that email also contains a “Multi Factor Authentication” or MFA code. Until this code is typed in, attempts to log-in are blocked.

MFA Alert generated by accessing the dashboard

Make sure the email registered to your Cloudflare account is correct and that it is one whose emails you can quickly see for maximum benefit. The faster you react to an alert the better!

API Keys are very sensitive things. They have as much power as your password but benefit from relatively few of the protections built into a modern browser. This is why we are deepening our investment to make our API even more secure.

Our first improvement, released last week, was to protect the API key against malicious software - such as a malicious browser extension - by adding a CAPTCHA to the “View API Key” feature. Below is a screenshot of the challenge you will now see when attempting to access your API Key. This change means that even if malicious software manages to steal your password, it cannot easily request and harvest your API Key.

Updated “View API Key” user experience

Next, we are looking at looked at scoped API keys. These are keys which can either be restricted to authorized IP addresses or limited in terms of what they can do. At the same time we will be adding the ability to turn the API off completely for your account. Finally, looking to the future, we are exploring options like other technical frameworks and token types, so that you have even better tools to build an API architecture that that is secure by design.

• Turn on 2FA and ensure your email address is correct The sooner you see an alert, the faster you can take action to lock your account down.Handle your credentials with care, NEVER enter them into a site other than Cloudflare.com if in doubt check the website certificate fingerprints:

SHA 256 - 12 C4 A5 74 7E D5 6E 37 2C 87 89 02 25 E4 CD 51 89 6D 8E AD 7D 55 CF 76 BF D1 9B 6B 74 6C 70 D0

SHA1 - D4 AD AB 1B 95 72 8D 3D 6E 26 4A 70 70 B1 1E 88 2F CA 71 67

• Check your browser extensions regularly. If you see any you don’t recognize remove them immediately. Remember that like all software, regular updates to browser extensions are important too.

• Design your client or application to protect your API credentials. If your API design is weak - for example it doesn’t do proper certificate validation, or even worse transmit any of the data in plaintext at any point in the connection - then you are risking disaster.

• Change your API key regularly, especially if you have any concern that it may have been exposed.

• Do not store your API key (or any credentials for that matter) in public repositories. One way to ensure this is by making sure you don’t store your API key in your application source tree. A significant number of accidental leaks to GitHub happen because this gets overlooked.

• Be very careful when embedding keys or credentials in clients that you will expose publically. Do not expose your account API key this way. Reverse engineering is not hard and many organizations have learnt this the hard way.

• Review your code before you release it. Use a tool that’s part of your CI or Build processes to automatically check for accidental key leakage. How many times have we seen API Keys for major institutions leak because they were accidentally checked into GitHub? Don’t be that person.

• IBM has some great additional security guidance for organizations building with APIs here,

• Finally, it may seem obvious but take care when clicking on links in emails. Even experienced Chrome developers get tricked sometimes!