CloudFlare is an engineering-driven company. This is a story we're proud of because it embodies the essence of who we are: when faced with a problem, we found a novel solution. Technical details to follow but, until then, welcome to the no hardware world. (Update: The post with the technical details is now online.)
Fall in San Francisco
The story begins on a Saturday morning, in the Fall of 2012, almost exactly two years ago. I got a call on my cell phone that woke me. It was a man who introduced himself as the Chief Information Security Officer (CISO) at one of the world's largest banks.
"I got your number from a reporter," he said. "We have an incident. Could you and some of your team be in New York Monday morning? We'd value your advice." We were a small startup. Of course we were going to drop everything and fly across the country to see if we could help.
I called John Roberts and Sri Rao, two members of CloudFlare's team. John had an air of calm about him and owned more khaki pants than any of the rest of us. Sri was a senior member of our technical operations team and could, already at that point, justifiably claim he'd essentially "seen it all" in the two years he'd spent keeping CloudFlare's network online.
Sunday night we packed into a plane to New York. En route I made Sri promise he wouldn't wear cargo shorts to the meeting with the bank executives the next day. And he didn't. Instead, we all showed up in ill-fitting suits like the out-of-place engineers that we were.
Rock and the Hard Place
At the meeting the bankers explained the rock and the hard place they were between. On one side they were under attack. As the New York Times and other publications have subsequently reported, in the Fall of 2012 allegedly Iranian hackers systematically launched DDoS attacks that crippled major US financial institutions.
The bankers related that the attacks, which were between 60 - 80Gbps (far shy of the 500Gbps+ attacks we regularly see today), were sufficient to cripple their on-premise network hardware solutions. The multiple banks that we visited that day told us the same story. Whether it was their load balancer, their firewall, their router, or their switch, under attack, something had become saturated and was unable to keep up with the traffic. It didn't matter how clever the software on the device was, in every case they were dead at Layer 3.
If that was the rock, what was the hard place? The bankers all acknowledged what they needed was a cloud-based solution that could scale to meet the challenges they faced. Unfortunately, since they needed to support encrypted connections, that meant the cloud-based solution needed to terminate SSL connections. And there was the rub.
The Key is the (SSL) Key
An SSL key is the data that allows an organization to establish a secure connection with the customers that connect to it. It is also the data that lets an organization establish its identity. If you have an organization private SSL key, you can authenticate as if you were it. You can spoof identity and intercept traffic.
If, say, a media organization loses an SSL key, it's a very bad day. If a financial institution loses one, it's a nightmare. In addition to the public embarrassment and loss of trust, in the United States, the bankers we met with that Fall day in 2012 told us, if an SSL key is lost it's a critical security event that must be reported to the Federal Reserve.
Other vendors have tried to deal with this by, what several of the bankers we met with, termed: "security theater." They show you pictures of big, locked racks of servers with electronic combination locks.
We came away from that day of meetings in New York with one conclusion: the only way organizations that had the highest standards of SSL security could ever adopt the benefits of the cloud is if we never took possession of their SSL keys.
Sri, John, and I returned to San Francisco somewhat disheartened. I relayed what we'd learned to our engineering team. Everyone was bummed for a bit. Then, Sebastien Pahl, one of our engineers who had previously helped build DotCloud and Docker, said, "Do we really need to have physical access to the private key?"
That spawned a late evening in the office in front of several white boards. We'd speculated previously that there was a way to split session signing, the only part of the SSL handshake that requires the private key, from rest of the process. Sebastien pulled up the documentation on his phone and was convinced that there was a way to do it. Over the course of the night, he convinced the rest of us.
Creative Engineers FTW
Sebastien is the kind of engineer that, when he's transfixed with a problem, can't sleep. It's a trait we hire for at CloudFlare. He showed up the next morning looking both exhausted and excited. "I've proven it's possible," he said. "It's crude. It won't scale. It probably has security vulnerabilities galore, but I've proved we can terminate SSL connections even if we don't have physical access to the private SSL key."
Tomorrow, we'll publish a full post on the nitty, gritty techical details of how, what has come to be called Keyless SSL™, works. (Update: The post with the technical details is now online.) For now, I'll just tell you about what Sebastien had built. It was a dramatic demo. A simple agent ran on a Raspberry Pi. A web server, running on a remote server on CloudFlare's network, received HTTPS connections. When the Raspberry Pi was plugged in, the connections went through from a browser as they would normally. The lock appeared and the connection was secured, end-to-end. The minute the Raspberry Pi's power was disconnected, HTTPS access terminated.
Sebastien had proven that the solution to what the banks needed was possible: you could have SSL keys remote from the actual server terminating the connection. If that worked, there was no need to ever have limited on-premise network hardware again. Provide the functionality in the infinitely scalable environment of the cloud, but keep the keys on-premise so there's no risk they are ever misappropriated.
A prototype made in an evening is one thing, having something production ready is another. Sebastien turned the project over to John Graham-Cumming, Piotr Sikora, and Nick Sullivan, three of the lead engineers on our team. They worked with the banks that had originally contacted us to build a system that worked in high availability environments.
To make it work, we needed to hold connections open between CloudFlare's network and agents running on our customers' infrastructure. Moreover, we needed to share data about crytographic sessions setup for a visitor between all the machines that could serve that visitor. Making it work was one thing, making it fast was another. And, today, Keyless SSL clients are experiencing 3x+ faster SSL termination globally using the service than they were when they were relying only on on-premise solutions.
Tomorrow Nick Sullivan will spend time going through the details of how Keyless SSL works. For now know this: private clouds are an oxymoron. Keeping your network behind on-premise hardware you control is a recipe for disaster. Over time, the network edge needs the infinite scalability and elastacity that only a service like CloudFlare can provide. And, now, with Keyless SSL, anyone can get that flexibility without having to turn over their most guarded secrets: their private SSL keys.
Here's what people are saying about Keyless SSL:
Security
World-renowned security experts Jon Callas and Phil Zimmermann support CloudFlare's latest announcement sharing, “One of the core principles of computer security is to limit access to cryptographic keys to as few parties as possible, ideally only the endpoints. Application such as PGP, Silent Circle, and now Keyless SSL implement this principle and are correspondingly more secure.”
A spokesperson from NCC Group’s Cryptography Services practice commented: “We’ve seen how private keys can be stolen, and investing in techniques to limit their exposure makes the Internet a safer place. Our review of Keyless SSL indicates the keys themselves do not leave your infrastructure, and a secure channel with CloudFlare both protects the communication and reduces the attack surface for your key.”
"Because this system keeps your long-lived SSL private keys on-premise, it provides the same protection to those keys as conventional on-premise SSL solutions. This provides the security and performance benefits of managing SSL traffic in the cloud." explained Jian Jiang, Independent Academic Researcher at UC Berkeley
Enterprise
Senior Director of Trust at EMC Corporation, Davi Ottenheimer believes Keyless SSL is a fundamental innovation in security. “Everyone should be increasingly aware and concerned about the risks of handing their private keys over to service providers. The trade-offs between control and services are being solved by innovation in key management. Keyless solutions, where customers retain control, clearly improve security while maintaining the best service offerings. As we move to a more interconnected world with more localized-access to global providers, our trust has to be based on security controls that remain relevant within the latest advances of content delivery networks. Keeping control of your own private key, yet enabling a service provider to serve your customers with the same level of trust, is a real breakthrough in content delivery security.”
“Recent incidents like the APT exploit of Heartbleed to breach Community Health Systems and the Mask operation show that attacks on keys and certificates that establish trust are on the rise. If security teams don’t protect their keys and certificates they undermine their critical threat protection and existing security controls,” said Kevin Boeck, vice president of security strategy & threat intelligence at Venafi. “With our partner CloudFlare, Venafi supports the development of Keyless SSL technology to help further protect our Venafi Trust Protection Platform customers and secure their use of cloud services.”
Financial
“At Coinbase, we take security very seriously. To be successful in the Bitcoin ecosystem we prioritize security highly,” said Ryan McGeehan, director of security at Coinbase. “Technology that improves the security of our critical infrastructure, like our SSL keys, is always welcomed."
“As a private-cloud file-sync and share startup working with many financial organizations worldwide, we are always looking for the best security technologies that help keep important data safe, secure, and behind the firewall while maintaining the scale benefits of the cloud,” said Yuri Sagalov, co-founder and CEO of AeroFS. “Keyless SSL lets companies get the best of both worlds: Companies get to keep their private keys behind the corporate firewall where they belong, while still providing edge-level encryption for their customers accessing their services.