Cloudflare’s customers recognize that they need to protect the confidentiality and integrity of communications with their web visitors. The widely accepted solution to this problem is to use the SSL/TLS protocol to establish an encrypted HTTPS session, over which secure requests can then be sent. Eavesdropping is protected against as only those who have access to the “private key” can legitimately identify themselves to browsers and decrypt encrypted requests.
Today, more than half of all traffic on the web uses HTTPS—but this was not always the case. In the early days of SSL, the protocol was viewed as slow as each encrypted request required two round trips between the user’s browser and web server. Companies like Cloudflare solved this problem by putting web servers close to end users and utilizing session resumption to eliminate those round trips for all but the very first request.
Expanding footprint meets geopolitical concerns
As Internet adoption grew around the world, with companies increasingly serving global and more remote audiences, providers like Cloudflare had to continue expanding their physical footprint to keep up with demand. As of the date this blog post was published, Cloudflare has data centers in over 55 countries, and we continue to accelerate the build out of our global network.
While this expansion makes our customers’ sites faster and better protected against DDoS wherever visitors (or attackers) are connecting from, it presents a few related security challenges.
First, as we put equipment (and SSL private keys) in more and more countries around the world, it’s likely that local governments will possess differing—and often evolving—views on privacy and the sanctity of encrypted communications. In the past, there have been reports that certain governments have attempted to compel organizations to turn key material to them. (To be unambiguously clear, Cloudflare—as documented in our Transparency Report—has never turned over our SSL keys or our customers’ SSL keys to anyone.)
Even if local governments are to be trusted, organizations may have strong geopolitical-based opinions on security or mandates to adhere to certain regulatory frameworks. That, or they simply may understand there are only so many data centers in the world that can meet our most stringent physical security requirements and controls. As Cloudflare’s network grows, it’s inevitable that we will exhaust these facilities. We recognize that our customers may want to control distribution of their private key material based on an assessment of physical controls.
Private key restrictions to match your security posture
As Cloudflare serves larger, more multinational and geographically distributed customers, the frequency of requests we’ve heard to restrict private key distribution by physical location has increased.
We’ve listened carefully to those requests, and today we’re excited to introduce a new feature that hands this geography-based control over to you. We call it Geo Key Manager.
On upload of a new Custom Certificate, you now have the ability to indicate which subset of our data centers should have local access to your private key for the purpose of establishing HTTPS sessions.
Connections to data centers that you’ve told us aren’t allowed to hold your key will still be fully encrypted and accessible via HTTPS, using identical encryption algorithms, but will be done so using our Keyless SSL technology rather than locally accessing the key.
Initial options available for use
When we thought about which options to include in the dropdown box pictured above, we reached out to human rights and civil liberties organizations, leading financial and banking institutions, and other groups that place a high premium on security. These discussions resulted in the options you see today, and over the coming weeks we’ll be having additional conversations. Our goal in these talks is identifying what other designations make sense for our customers, so we can provide easily usable defaults.
Everywhere
By default, we won’t apply any restrictions; securely storing your keys in all of our data centers allows us to optimize performance for your visitors across the globe.
U.S. Only or E.U. Only
One common theme we heard from customers was the desire to restrict the distribution of private SSL keys to either the U.S. only—typically requested by U.S. government agencies or businesses that serve exclusively U.S. customers—or the E.U. only, for comparable reasons.
Selecting either of these options in the dropdown will ensure that your keys do not leave this geopolitical designation. As of the time this post was written, Cloudflare has 31 data centers in 25 U.S. cities and 30 data centers across 21 E.U. countries.
Highest Security Data Centers
Cloudflare has a checklist of security requirements that we require before locating our equipment somewhere. With every installation we ensure that our deployment is in compliance with internationally recognized security standards, including PCI DSS 3.2 and SSAE-18 SOC 2. This security review is re-evaluated quarterly by our internal security staff as well as annually by independent, third-party auditors.
As part of this assessment we group our data centers into security tiers, using this knowledge to help shape our deployment and operational decision making and strategies.
Baseline Requirements
At minimum, all of our data centers offer the following physical controls:
External and Internal Security
The datacenter must be physically secure with perimeter controls that meet industry accepted minimums (e.g., locked doors, high walls, man traps, alarm systems, etc.).
Restricted Facility Access
Access is restricted to Cloudflare employees and named authorized individuals only. Every access request must be authorized and the requester’s access confirmed with government issued ID.
All access requests are logged and those logs are reviewed on a regular basis.
Any attempts to physically access, modify or remove our equipment generates an alert to security and operational teams.
24x7 Onsite Security
Security guards are onsite and present 24 hours a day, 7 days a week.
CCTV
Facilities are monitored by closed-circuit television cameras.
Secured Server Cabinet
Access to Cloudflare’s server cabinets are controlled through physical security locks and other mechanisms.
Physical tamper detection on all systems.
All hardware deployed within cabinets are equipped with physical tamper detection systems that have been enabled and configured to report any tampering attempts.
Additional "High Security" Requirements
While we ensure that every Cloudflare data center meets minimum industry security standards, our top tier of data centers are held to a much higher standard.
These all have the baseline security requirements plus:
Pre-Scheduled, Biometric Controlled Facility Access
Employees of Cloudflare permitted to access the facility must have previously scheduled a visit before access will be granted.
Access to the entrance of the facility is controlled through the use of a biometric hand reader combined with an assigned access code.
Private Cages with Biometric Readers
All equipment is in private cages with physical access controlled via biometrics and recorded in audit logs.
Entrants have to pass through 5 separate readers before the cage can be accessed.
Exterior Security Controls and Monitoring
All points of ingress/egress are monitored by an intrusion detection system (IDS), with authorized users and access events archived for historical review.
Interior Security Controls and Monitoring
Interior points of ingress/egress are controlled by the access control subsystem, with entry routed through a mantrap. All areas are monitored and recorded with closed-circuit television, with data kept for a minimum of 30 days.
Exterior walls are airtight and may incorporate additional security measure such as reinforced concrete, Kevlar bullet board, vapor barriers, or bullet proof front doors.
We designate facilities that meet the above our "Highest Security" data centers.
What's the catch?
As mentioned previously, the very first HTTPS request made to a data center that does not hold your private key requires a bit of overhead while a symmetric encryption key for the session is negotiated.
By “overhead” we mean that a bit of network latency is introduced while the remote facility is contacted for assistance establishing the secure channel. The further (physically) this key-holding data center is from where the HTTPS request is being established, the longer the wait.
For example, the first request to Moscow for a resource that's protected by a key held only in our “Highest Security” facilities takes an additional three one-hundredths of a second (30ms) to go to Stockholm and back. Or four one-hundredths of a second (40ms) to go from Moscow to Frankfurt and back.
Cloudflare will pick the fastest route to a nearby data center for this first request that requires an extra hop. However, all subsequent requests after the initial one will be just as fast as if the key were local thanks to SSL/TLS session resumption.
This page brought to you by Geo Key Manager
To demonstrate the performance impact is minimal, we set blog.cloudflare.com to use only our Highest Security data centers. If you click around on the blog you can see that, while the pages are encrypted, there's no noticeable delay in surfing them.
We estimate that the average user, based on the geographic distribution of traffic to our blog, will see a ~65ms average increase in latency for their first request. After that, subsequent requests will be as fast as if keys were distributed to all our data centers.
Looking forward
Product-offering wise, we plan to extend Geo Key Manager to work with Dedicated Certificates. Once this is available, we will communicate the news on the blog or in a knowledge-base article.
Additionally, we plan to soon allow Enterprise users to specify a precise list of which of our data centers they do or do not want their keys held in; for example, customers who have concerns about specific countries or regions will be able to use this feature to gain fine-grained control over their private key distribution. If you’re on an Enterprise plan and would like early access to this feature today, please contact your Customer Success Manager.
Beyond granular controls, we aim to fundamentally change the dynamics of geo key selection by allowing you to outsource the decision making and, potentially, legal challenge process. Imagine, for instance, if you could specify that a civil society organization like the Electronic Frontier Foundation be the custodian of your keys. Different organizations that you trust could develop expertise on fighting overly broad legal requests that may someday come.
We’re just getting started having conversations with these types of organizations, so if you have one in mind, be sure to let us know in the comment section below!