A quick followup to our last blog post on our decision to reissue and revoke all of CloudFlare's customers' SSL certificates. One question we've received is why we didn't just reissue and revoke all SSL certificates as soon as we got word about the Heartbleed vulnerability? The answer is that the revocation process for SSL certificates is far from perfect and imposes a significant cost on the Internet's infrastructure.
Today, after having done a mass reissuance and revocation, we have a tangible sense of that cost. To understand it, you need to understand a bit about how your browser checks if an SSL certificate has been revoked.
OCSP & CRL
When most browsers visit web pages over HTTPS they perform a check using one of two certificate revocation methods: Online Certificate Status Protocol (OCSP) or Certificate Revocation List (CRL). For OCSP, the browser pings the certificate authority and asks whether a particular site's certificate has been revoked. For CRL, the browser pings the certificate authority (CA) and downloads a complete list of all the certificates that have been revoked by that CA.
There are pluses and minuses to both systems. OCSP imposes a lighter bandwidth cost, but a higher number of requests and backend lookups. CRL doesn't generate as many requests, but, as the CRL becomes large, can impose a significant bandwidth burden. These costs are borne by visitors to websites, whose experience will be slower as a result, but even more so by the CAs who need significant resources in place to handle these requests.
Technical Costs of Revocation
Yesterday, CloudFlare completed the process of reissuing all the SSL certificates we manage for our customers. Once that was complete, we revoked all previously used certificates. You can see the spike in global CRL activity we generated:
What you can't see is the spike in bandwidth that imposed. Globalsign, who is CloudFlare's primary CA partner, saw their CRL grow to approximately 4.7MB in size from approximately 22KB on Monday. The activity of browsers downloading the Globalsign CRL generated around 40Gbps of net new traffic across the Internet. If you assume that the global average price for bandwidth is around \$10/Mbps, just supporting the traffic to deliver the CRL would have added \$400,000USD to Globalsign's monthly bandwidth bill.
Lest you think that’s an overestimate, to make the total costs more accurate, we ran the numbers using AWS’s CloudFront price calculator using a mix of traffic across regions that approximates what we see at CloudFlare. The total cost to Globalsign if they were using AWS’s infrastructure, would be at least \$952,992.40/month. Undoubtedly they’d give some additional discounts above the pricing they list publicly, but any way you slice it the costs are significant.
Beyond the cost, many CAs are not setup to be able to handle this increased load. Revoking SSL certificates threatens to create a sort of denial of service attack on their own infrastructures. Thankfully, CloudFlare helps power Globalsign's OCSP and CRL infrastructure. We were able to bear the brunt of the load, allowing us to move forward with revocation as quickly as we did. And, no, we didn’t charge them anything extra.
So, if you're wondering why some people are dragging their feet on mass certificate revocation, now you know why — it imposes a real cost. And if you're a CA who's wondering what you're going to do when you inevitably have to revoke all the certs you've issued over the last year, we're happy to help.