Participating in the Internet democracy occasionally means that ideas and technologies that were once popular or ubiquitous on the net lose their utility as newer technologies emerge. SPDY is one such technology. Several years ago, Google drafted a proprietary and experimental new protocol called SPDY. SPDY offered many performance improvements over the aging HTTP/1.1 standard and these improvements resulted in significantly faster page load times for real-world websites. Stemming from its success, SPDY became the starting point for HTTP/2 and, when the new HTTP standard was finalized, the SPDY experiment came to an end where it gradually fell into disuse.

As a result, we're announcing our intention to deprecate the use of SPDY for connections made to Cloudflare's edge by February 21st, 2018.

Five and a half years ago, when the majority of the web was unencrypted and web developers were resorting to creative tricks to improve performance (such as domain sharding to download resources in parallel) to get around the limitations of the then thirteen-year-old HTTP/1.1 standard, Cloudflare launched support for an exciting new protocol called SPDY.

CC BY-SA 4.0 by Maximilien Brice

SPDY aimed to remove many of the bottlenecks present in HTTP/1.1 by changing the way HTTP requests were sent over the wire. Using header compression, request prioritization, and multiplexing, SPDY was able to provide significant performance gains while remaining compatible with the existing HTTP/1.1 standard. This meant that server operators could place a SPDY layer, such as Cloudflare, in front of their web application and gain the performance benefits of SPDY without modifying any of their existing code. SPDY effectively became a fast tunnel for HTTP traffic.

As the HTTPbis Working Group submitted SPDY for standardization, the protocol underwent some changes (e.g.— Using HPACK for compression), but retained its core performance optimizations and came to be known as HTTP/2 in May 2015.

With the standardization of HTTP/2, Google announced that they would cease supporting SPDY with the public release of Chrome 51 in May 2016. This signaled to other software providers that they too should abandon support for the experimental SPDY protocol in favor of the newly standardized HTTP/2. Mozilla did so with the release of Firefox 51 in January 2017 and NGINX built their HTTP/2 module so that either it or the SPDY module could be used to terminate HTTPS connections, but not both.

Cloudflare announced support for HTTP/2 in December of 2015. It was at this point that we deviated from our peers in the industry. We knew that adoption of this new standard and the migration away from SPDY would take longer than the 1-2 year timeline put forth by Google and others, so we created our own patch for NGINX so that we could support terminating both SPDY and HTTP/2. This allowed Cloudflare customers who had yet to upgrade their clients to continue to receive the performance benefits of SPDY until they were able to support HTTP/2.

When we made this decision, SPDY was used for 53.59% of TLS connections to our edge and HTTP/2 for 26.79%. Had we only adopted the standard NGINX HTTP/2 module, we would've made the internet around 20% slower for more than half of all visitors to sites on Cloudflare— definitely not a performance compromise we were willing to make!

Two years after we began supporting HTTP/2 (and nearly three years after standardization), the majority of web browsers now support HTTP/2 where the notable exceptions are the UC Browser for Android and Opera Mini. This results in SPDY being used for only 3.83% of TLS connections to Cloudflare's edge whereas HTTP/2 accounts for 66.88%. At this point, and for several reasons, now is the time to stop supporting SPDY.

CC0 by JanBaby

Looking closer at the low percentage of clients connecting with SPDY in 2018, 65% of these connections are made by older iOS and macOS apps compiled against HTTP and TLS libraries which only support SPDY and not HTTP/2. This means that these app developers need to publish an update with HTTP/2 support to enable support for this protocol. We worked closely with Apple to assess the impact of deprecating SPDY for these clients and determined that the impact of deprecation is minimal.

We mentioned earlier that we applied our own patch to NGINX in order to be able to continue to support both SPDY and HTTP/2 for TLS connections. What we didn't mention was the engineering cost associated with maintaining this patch. Every time we want to update NGINX, we also need to update and test our patch, which makes each upgrade more difficult. Further, no active development is being done to SPDY so in the event that security issues arise, we would incur the cost of developing our own security patches.

Finally, when we do disable SPDY this February, the less than 4% of traffic that currently uses SPDY will still be able to connect to Cloudflare's edge by gracefully falling back to using HTTP/1.1.

Part of being an innovator is knowing when it is time to move forward and put older innovations in the rear view mirror. By seeing 10% of HTTP requests made on the internet, Cloudflare is in a unique position to analyze overall adoption trends of new technologies, allowing us to make informed decisions on when to launch new features or deprecate legacy ones.

SPDY has been extraordinarily beneficial to clients connecting to Cloudflare over the years, but now that the protocol is largely abandoned and superseded by newer technologies, we recognize that 2018 is time to say goodbye to an aging legacy protocol.

We began supporting the DNS Certification Authority Authorization (CAA) Resource Record in June behind a beta flag. Our goal in doing so was to see how the presence of these records would affect SSL certificate issuance by publicly-trusted certification authorities. We also wanted to do so in advance of the 8 September 2017 enforcement date for mandatory CAA checking at certificate issuance time, without introducing a new and externally unproven behavior to millions of Cloudflare customers at once. This beta period has provided invaluable insight as to how CAA records have changed and will continue to change the commercial public-key infrastructure (PKI) ecosystem.

As of today, we’ve removed this beta flag and all users are welcome to add CAA records as they see fit—without having to first contact support. Note that if you’ve got Universal SSL enabled, we’ll automatically augment your CAA records to allow issuance from our CA partners; if you’d like to disable Universal SSL and provide your own certificates, you’re welcome to do that too.Below are some additional details on CAA, the purpose of this record type, and how its use has evolved since it was first introduced. If you’d rather just jump to the details of our implementation, click here and we’ll take you to the relevant section of the post.

Before diving into CAA it’s helpful to understand the purpose of a public key infrastructure (PKI). Quite simply, PKI is a framework that’s used to secure communications between parties over an insecure public network. In “web PKI”, the PKI system that’s used to secure communications between your web browser and this blog (for example), the TLS protocol is used with SSL certificates and private keys to protect against eavesdropping and tampering.

While TLS handles the sanctity of the connection, ensuring that nobody can snoop on or mess with HTTPS requests, how does your browser know it’s talking to the actual owner of blog.cloudflare.com and not some imposter? Anyone with access to OpenSSL or a similar tool can generate a certificate purporting to be valid for this hostname but fortunately your browser only trusts certificates issued (or “signed by”) by certain well-known parties.

These well-known parties are known as certification authorities (CAs). The private and public key that form the certificate for blog.cloudflare.com were generated on Cloudflare hardware, but the stamp of approval—the signature—was placed on the certificate by a CA. When your browser receives this “leaf” certificate, it follows the issuer all the way to a “root” that it trusts, validating the signatures along the way and deciding whether to accept the certificate as valid for the requested hostname.

Before placing this stamp of approval, CAs are supposed to take steps to ensure that the certificate requester can demonstrate control over the hostname. (In some cases, as you’ll learn below, this is not always the case, and is one of the reasons that CAA was introduced.)

Given that people are imperfect beings and prone to making mistakes or poor judgement calls, it should come to the surprise of no one that the PKI ecosystem has a fairly blemished track-record when it comes to maintaining trust. Clients, CAs, servers, and certificate requesters are all created or operated by people who have made mistakes.

Jurassic Park. 1993, Stephen Spielberg [Film] Universal Pictures.

Client providers have been known to add compromising certificates to the local trust store or install software to intercept secure connections; servers have been demonstrated to leak private keys or be unable to properly rotate session ticket keys; CAs have knowingly mis-issued certificates or failed to validate hostname ownership or control reliably; and individuals requesting certificates include phishers creating convincing imposter versions of popular domains and obtaining valid and trusted certificates. Every party in this ecosystem is not completely without blame in contributing to a diminished sense of trust.

Of these many problems (most of which have already been addressed or are in the process of being resolved), perhaps the most unsettling is the willful mis-issuance of certificates by trusted CAs. Knowingly issuing certificates to parties who haven't demonstrated ownership, or issuance outside the parameters defined by the CA/Browser Forum (a voluntary and democratic governing body for publicly-trusted certificates) by CAs who have certificates present in trust stores severely undermines the value of that trust store, all certificates issued by that CA, and the publicly-trusted PKI ecosystem as a whole.

To help prevent future mis-issuance by publicly trusted CAs, a new DNS resource record was proposed by those CAs to help reduce the risk of certificate mis-issuance: The Certification Authority Authorization (CAA) Resource Record.

The general idea is that the owner of any given domain (e.g., example.com) would add CAA records at their authoritative DNS provider, specifying one or more CAs who are authorized to issue certificates for their domain.

RFC6844 currently specifies three property tags for CAA records: issue, issuewild, and iodef.

• The issue property tag specifies CAs who are authorized to issue certificates for a domain. For example, the record example.com. CAA 0 issue "certification-authority.net" allows the "Certification Authority" CA to issue certificates for example.com.

• The issuewild property tag specifies CAs that are only allowed to issue certificates that specify a wildcard domain. E.g., the record example.com. CAA 0 issuewild "certification-authority.net" only allows the "Certification Authority" CA to issue certificates containing wildcard domains, such as *.example.com.

• The iodef property tag specifies a means of reporting certificate issue requests or cases of certificate issuance for the corresponding domain that violate the security policy of the issuer or the domain name holder. E.g., the record example.com. CAA 0 iodef "mailto:example@example.com" instructs the issuing CA to send violation reports via email to the address provided at the attempted time of issuance.

CAA records with the issue and issuewild tags are additive; if more than one are returned as the response for a DNS query for a given hostname, the CAs specified in both records are both considered authorized.

If the authoritative DNS provider does not yet support CAA records or none are present in the zone file, the issuing CA is still authorized to issue when no records are present, largely preserving the issuance behavior before CAA records were an adopted standard.

As of 8 September 2017, all publicly-trusted CAs are now required to check CAA at issuance time for all certificates issued, thereby enabling certificate requestors (domain owners) to dictate which CAs can issue certificates for their domain.

RFC6844 specifies a very curious CAA record processing algorithm:

The search for a CAA record climbs the DNS name tree from the
   specified label up to but not including the DNS root '.'.

   Given a request for a specific domain X, or a request for a wildcard
   domain *.X, the relevant record set R(X) is determined as follows:

   Let CAA(X) be the record set returned in response to performing a CAA
   record query on the label X, P(X) be the DNS label immediately above
   X in the DNS hierarchy, and A(X) be the target of a CNAME or DNAME
   alias record specified at the label X.

   o  If CAA(X) is not empty, R(X) = CAA(X), otherwise

   o  If A(X) is not null, and R(A(X)) is not empty, then R(X) =
      R(A(X)), otherwise

   o  If X is not a top-level domain, then R(X) = R(P(X)), otherwise

   o  R(X) is empty.

While the above algorithm is not easily understood at first, the example immediately following it is much easier to comprehend:

For example, if a certificate is requested for X.Y.Z the issuer will
   search for the relevant CAA record set in the following order:

      X.Y.Z

      Alias (X.Y.Z)

      Y.Z

      Alias (Y.Z)

      Z

      Alias (Z)

      Return Empty

In plain English, this means that if the owner of example.com requests a certificate for test.blog.example.com, the issuing CA must

1. Query for a CAA record at test.blog.example.com.. If a CAA record exists for this hostname, the issuing CA stops checking for CAA records and issues accordingly. If no CAA record exists for this hostname and this hostname exists as an A or AAAA record, the CA then moves up the DNS tree to the next highest label.

2. Query for a CAA record at blog.example.com.. Just like the first check, if no CAA record exists for this hostname and this hostname exists as an A or AAAA record, the CA then continues traversing the DNS tree.

3. Query for a CAA record at example.com.

4. Query for a CAA record at com.

At the end of the last step, the issuing CA has climbed the entire DNS tree (excluding the root) checking for CAA records. This functionality allows a domain owner to create CAA records at the root of their domain and have those records apply to any and all subdomains.

However, the CAA record processing algorithm has an additional check if the hostname exists as a CNAME (or DNAME) record. In this case, the issuing CA must also check the target of the CNAME record. Revisiting the example above for test.blog.example.com. where this hostname exists as a CNAME record, the issuing CA must

1. Query for a CAA record at test.blog.example.com.. Since test.blog.example.com. exists as the CNAME test.blog.example.com. CNAME test.blog-provider.net., the issuing CA must next check the target for the presence of a CAA record before climbing the DNS tree.

2. Query for a CAA record at test.blog-provider.net.

The issuing CA in this example is only at step two in the CAA processing algorithm and it has already come across two separate issues.

First, the issuing CA has checked the hostname requested on the certificate (test.blog.example.com.) and since that hostname exists as a CNAME record, it has also checked the target of that record (test.blog-provider.net.). However, if test.blog-provider.net. itself is also a CNAME record, the the CAA record processing algorithm states that the issuing CA must check the target of that CNAME as well.

In this case it is fairly simple to create a CNAME loop (or very long CNAME chain) either via an accidental misconfiguration or with malicious intent to prevent the issuing CA from completing the CAA check.

Second, example.com and blog-provider.net might not be owned and operated by the same entity or even exist in the same network. The RFC authors appear to be operating under the assumption that CNAME records are still used as they were in November 1987:

A CNAME RR identifies its owner name as an alias, and specifies the corresponding canonical name in the RDATA section of the RR.

This behavior may have been true thirty years ago, but consider the number and prevalence of SaaS providers in 2017; who truly is authoritative for the content addressed by a DNS record, the content creator (and likely domain name owner who is using a CNAME) or the SaaS provider to whom the content creator subscribes?

Many if not all major service or application providers include clauses in their respective terms of service with regards to content added by the user to their account for the service. Intellectual property must be user-generated, properly licensed, or otherwise lawfully obtained. As content uploaded by users is difficult to control, liability limitations, indemnification, and service termination are all commonly used when the intellectual property added to the service is owned by another party, is unlawful, or is outside the definition of acceptable use. From a content perspective within the scope of a service provider's terms of service, it’s difficult to consider the service provider canonical for the content hosted there.

Using a real world example, there are currently 401,716 CNAME records in the zone files for domains on Cloudflare whose target is ghs.google.com. This hostname is given to subscribers of Google's Blogger service to use as the target of a CNAME so that a Blogger subscriber may use their own domain name in front of Google's service. With one CAA record, Google could dictate that nearly half a million blogs with a vanity domain name can only have one CA or no CAs issue certificates unless each of those hostnames created their own CAA records to allow issuance.

Even without following CNAMEs to their targets, the DNS climbing algorithm is not unproblematic. Operators of top-level domains may decide that only certain issuing CAs are trustworthy or certain CAs are advantageous for business and create CAA records only allowing those CAs to issue. We already see this in action today with the pseudo-top level domain nl.eu.org which has the CAA record nl.eu.org. CAA 0 issue "letsencrypt.org", which only allows issuance through Let's Encrypt without CAA records being present at the subdomains of nl.eu.org.

The authors of RFC6844 were also unwilling to secure their record and its use from any potential on-path attacks—

Use of DNSSEC to authenticate CAA RRs is strongly RECOMMENDED but not required.

Without DNSSEC, DNS responses are returned to the requestor in plain-text. Anyone in a privileged network position (a recursive DNS provider or ISP) could alter the response to a CAA query to allow or deny issuance as desired. As only about 820,000 .com domain names out of the more than 130 million registered .com domain names are secured with DNSSEC, perhaps the low adoption rates influenced the decision to not make DNSSEC with CAA mandatory.

Moving beyond the RFC, the CA/Browser Forum's Baseline Requirements (BR) attempt to clarify aspects regarding the behavior a CA should take when checking for CAA.

CAs are permitted to treat a record lookup failure as permission to issue if:

• the failure is outside the CA's infrastructure;

• the lookup has been retried at least once; and

• the domain's zone does not have a DNSSEC validation chain to the ICANN root

Effectively, this means that if the CAA response is either a SERVFAIL, REFUSED, or the query times out, regardless of whether or not a CAA record exists, the CA is permitted to issue if this query fails more than once while attempting to issue a certificate. However, multiple CAs have told us that DNS lookup failures will prevent issuance regardless of the above conditions in the BR. In this case, something as benign as a transient network error could result in a denial of issuance or worse, any recursor that doesn’t understand and a query for CAA record will prevent issuance. We actually saw this with Comcast’s resolvers and reported the bug to their DNS provider.

There’s an additional security gap in that neither the RFC nor the BR indicate where the issuing CA should query for CAA records. It is acceptable within the current standards to query any DNS recursor for these records as well as the authoritative DNS provider for a domain. For example, an issuing CA could query Google’s Public DNS or a DNS recursor provided by their ISP for these responses. This enables compromised DNS recursors or one run by a rogue operator to alter these responses, either denying issuance or allowing issuance by a CA not approved by the domain owner. The RFC and BR should be amended so that an issuing CA must always query these records at the authoritative provider to close this gap.

As of today, CAA records are no longer in beta and all customers are able to add CAA records for their zones. This can be done in the DNS tab of the Cloudflare dashboard or via our API.

When creating CAA records in the dashboard, an additional modal appears to help clarify the different CAA tag options and format the record correctly as an incorrectly formatted record would result in every CA not being able to issue a certificate.

Cloudflare is in a unique position to be able to complete SSL validation for a domain and have certificates issued on the domain owner's behalf. This is how Cloudflare is able to automatically provision and have our Universal SSL certificates issued for every domain active on Cloudflare for free.

Cloudflare partners with multiple CAs to issue certificates for our managed SSL products: Universal SSL, Dedicated Certificates, and SSL for SaaS. Since CAA record checking is now mandatory for publicly-trusted CAs, Cloudflare automatically adds the requisite CAA records to the zone when a user adds one or more CAA records in the Cloudflare dashboard for their domain in order to allow our partner CAs to continue to be able to issue certificates for each of our SSL products.

Some site owners may want to manage their own SSL certificates in order to be compliant with their own standard operating procedures or policies. Alternately, domain owners may only want to trust specific CAs that are not the CAs Cloudflare currently partners with to issue Universal SSL certificates. In the latter case, users now have the ability to disable Universal SSL.

When Universal SSL is disabled, the CAA records added to allow our partner CAs to issue certificates are deleted and all Universal SSL certificates available for the zone are removed from our edge. Dedicated certificates, custom certificates, as well as SSL for SaaS certificates are all individually managed by each customer and can be added or removed as needed.

CAA as it exists today does very little to reduce the attack surface around certificate issuance while making it more difficult for well-intended parties to participate. That said, CAA is a young standard recently adopted by the web PKI community with many involved individuals as well as active parties working toward addressing some of the gaps present in the current RFC and committed to making the overall CAA experience better for both the certificate requesters and CAs.

To start, an errata report exists to clarify the CAA record processing algorithm and to reduce the degree in which the targets of CNAME records should be checked. Similarly, the DNS tree-climbing behavior of the CAA record processing algorithm is still up for debate. There are also active discussions around implementation issues, such as the recognition that some authoritative DNS resolvers incorrectly sign empty responses to CAA queries when DNSSEC is enabled for a zone and how to handle these cases in a way that would still allow CAs to issue. Proposals exist suggesting new tags or property fields in CAA records, such as a CA requiring an account number in the property value for issue tags, or only allowing specific validation types (e.g., DV, OV, or EV). The Electronic Frontier Foundation (EFF) has expressed interest in hardening CAs against non-cryptographic attacks, particularly with a focus on the domain validation process for obtaining certificates. While not directly pertaining to CAA, any ideas or proposed hardening solutions might increase reliance on CAA or completely obviate the need for it altogether.

CC BY-NC 2.5 by xkcd.com

As with all internet standards, none are perfect and all are far from permanent—CAA included. Being in the position to implement new standards at scale, seeing what effect adoption of those standards has, and working with the Internet community to address any issues or gaps is a privilege and allows us to live up to our mission of building a better Internet.

We're thrilled to be involved in efforts to make CAA (and any other standard) better for anyone and everyone.