We welcome the CMA’s recognition that publishers need a fairer deal and believe the proposed rules are a step into the right direction. Publishers should be entitled to have access to tools that enable them to control the inclusion of their content in generative AI services, and AI companies should have a level playing field on which to compete. 

But we believe the CMA has not gone far enough and should do more to safeguard the UK’s creative sector and foster healthy competition in the market for generative and agentic AI. 

In January 2025, the UK’s regulatory landscape underwent a significant legal shift with the implementation of the Digital Markets, Competition and Consumers Act 2024 (DMCC). Rather than relying on antitrust investigations to address risks to competition, the CMA can now designate firms as having Strategic Market Status (SMS) when they hold substantial, entrenched market power. This designation allows for targeted CMA interventions in digital markets, such as imposing detailed conduct requirements, to improve competition. 

In October 2025, the CMA designated Google as having SMS in general search and search advertising, given its 90 percent share of the search market in the UK. Crucially, this designation encompasses AI Overviews and AI Mode, with the CMA now having the authority to impose conduct requirements on Google’s search ecosystem. Final requirements imposed by the CMA are not merely suggestions but legally enforceable rules that can relate specifically to AI crawling with significant sanctions to ensure Google operates fairly. 

The CMA’s designation could not be more timely. As we’ve said before, we are indisputably in a time when the Internet needs clear “rules of the road” for AI crawling behavior. 

As the CMA rightly states, “publishers have no realistic option but to allow their content to be crawled for Google’s general search because of the market power Google holds in general search. However, Google currently uses that content in both its search generative AI features and in its broader generative AI services.” 

In other words: the same content that Google scrapes for search indexing is also used for inference/grounding purposes, like AI Overviews and AI Mode, which rely on fetching live information from the Internet in response to real-time user queries. And that creates a big problem for publishers—and for competition.

Because publishers cannot afford to disallow or block Googlebot, Google’s search crawler, on their website, they have to accept that their content will be used in generative AI applications such as AI Overviews and AI Mode within Google Search that return very little, if any, traffic to their websites. This undermines the ad-supported business models that have sustained digital publishing for decades, given the critical role of Google Search in driving human traffic to online advertising. It also means that Google’s generative AI applications enter into direct competition with publishers by reproducing their content, most often without attribution or compensation. 

Publishers’ reluctance to block Google because of its dominance in search gives Google an unfair competitive advantage in the market for generative and agentic AI. Unlike other AI bot operators, Google can use its search crawler to gather data for a variety of AI functions with little fear that its access will be restricted. It has minimal incentive to pay publishers for that data, which it is already getting for free. 

This prevents the emergence of a well-functioning marketplace where AI developers negotiate fair value for content. Instead, other AI companies are disincentivized from coming to the table, as they are structurally disadvantaged by a system that allows one dominant player to bypass compensation entirely. As the CMA itself recognizes, "[b]y not providing sufficient control over how this content is used, Google can limit the ability of publishers to monetise their content, while accessing content for AI-generated results in a way that its competitors cannot match”. 

Cloudflare data validates the concern about Google’s competitive advantage. Based on our data, Googlebot sees significantly more Internet content than its closest peers. 

Over an observed period of two months, Googlebot successfully accessed individual pages almost two times more than ClaudeBot and GPTBot, three times more than Meta-ExternalAgent, and more than three times more than Bingbot. The difference was even more extreme for other popular AI crawlers: for instance, Googlebot saw 167 times more unique pages than PerplexityBot. Out of the sampled unique URLs using our network that we observed over the last two months, Googlebot crawled roughly 8%.

In rounded multiple terms, Googlebot sees:

• vs. ~1.70x the amount of unique URLs seen by ClaudeBot;

• vs. ~1.76x the amount of unique URLs seen by GPTBot;

• vs. ~2.99x the amount of unique URLs by Meta-ExternalAgent;

• vs. ~3.26x the amount of unique URLs seen by Bingbot;

• vs. ~5.09x the amount of unique URLs seen by Amazonbot;

• vs. ~14.87x the amount of unique URLs seen by Applebot;

• vs. ~23.73x the amount of unique URLs seen by Bytespider;

• vs. ~166.98x the amount of unique URLs seen by PerplexityBot;

• vs. ~714.48x the amount of unique URLs seen by CCBot; and

• vs: ~1801.97x the amount of unique URLs seen by archive.org_bot.

Googlebot also stands out in other Cloudflare datasets.  

Even though it ranks as the most active bot by overall traffic, publishers are far less likely to disallow or block Googlebot in their robots.txt file compared to other crawlers. This is likely due to its importance in driving human traffic to their content—and, as a result, ad revenue—through search. 

As shown below, almost no website explicitly disallows the dual-purpose Googlebot in full, reflecting how important this bot is to driving traffic via search referrals. (Note that partial disallows often impact certain parts of a website that are irrelevant for search engine optimization, or SEO, such as login endpoints.)

Robots.txt merely allows the expression of crawling preferences; it is not an enforcement mechanism. Publishers rely on “good bots” to comply. To manage crawler access to their sites more effectively—and independently of a given bot’s compliance—publishers can set up a Web Application Firewall (WAF) with specific rules, technically preventing undesired crawlers from accessing their sites. Following the same logic as with robots.txt above, we would expect websites to block mostly other AI crawlers but not Googlebot. 

Indeed, when comparing the numbers for customers using AI Crawl Control, Cloudflare’s own AI crawler blocking tool that is integrated in our Application Security suite, between July 2025 and January 2026, one can see that the number of websites actively blocking other popular AI crawlers (e.g., GPTBot, Claudebot), was nearly seven times as high as the number of websites that blocked Googlebot and Bingbot. (Like Googlebot, Bingbot combines search and AI crawling and drives traffic to these sites, but given its small market share in search, its impact is less significant.)

So we agree with the CMA on the problem statement. But how can publishers be enabled to effectively opt out of Google using their content for its generative AI applications? We share the CMA’s conclusion that “in order to be able to make meaningful decisions about how Google uses their Search Content, (...) publishers need the ability effectively to opt their Search Content out of both Google’s search generative AI features and Google’s broader generative AI services.” 

But we’re concerned that the CMA’s proposal is insufficient.

On January 28, 2026, the CMA published four sets of proposed conduct requirements for Google, including conduct requirements related to publishers. According to the CMA, the proposed publisher rules are designed to address concerns that publishers (1) lack sufficient choice over how Google uses their content in its AI-generated responses, (2) have limited transparency into Google’s use of that content, and (3) do not get effective attribution for Google’s use of their content. The CMA recognized the importance of these concerns because of the role that Google search plays in finding content online. 

The conduct requirements would mandate Google grant publishers "meaningful and effective" control over whether their content is used for AI features, like AI Overviews. Google would be prohibited from taking any action that negatively impacts the effectiveness of those control options, such as intentionally downranking the content in search. 

To support informed decisionmaking, the CMA proposal also requires Google to increase transparency, by publishing clear documentation on how it uses crawled content for generative AI and on exactly what its various publisher controls cover in practice. Finally, the proposal would require Google to ensure effective attribution of publisher content and to provide publishers with detailed, disaggregated engagement data—including specific metrics for impressions, clicks, and "click quality"—to help them evaluate the commercial value of allowing their content to be used in AI-generated search summaries.

Although we support the CMA’s efforts to improve options for publishers, we are concerned that the proposed requirements do not solve the underlying issue of promoting fair, transparent choice over how their content is used by Google. Publishers are effectively forced to use Google’s proprietary opt-out mechanisms, tied specifically to the Google platform and under the conditions set by Google, rather than granting them direct, autonomous control. A framework where the platform dictates the rules, manages the technical controls, and defines the scope of application does not offer “effective control” to content creators or encourage competitive innovation in the market. Instead, it reinforces a state of permanent dependency.  

Such a framework also reduces choice for publishers. Creating new opt-out controls makes it impossible for publishers to choose to use external tools to block Googlebot from accessing their content without jeopardizing their appearance in Search results. Instead, under the current proposal, content creators will still have to allow Googlebot to scrape their websites, with no enforcement mechanisms to deploy and limited visibility available if Google does not respect their signalled preferences. Enforcement of these requirements by the CMA, if done properly, will be very onerous, without guarantee that publishers will trust the solution.

In fact, Cloudflare has received feedback from its customers that Google’s current proprietary opt-out mechanisms, including Google-Extended and ‘nosnippet’, have failed to prevent content from being utilized in ways that publishers cannot control. These opt-out tools also do not enable mechanisms for fair compensation for publishers. 

More broadly, as reflected in our proposed responsible AI bot principles, we believe that all AI bots should have one distinct purpose and declare it, so that website owners can make clear decisions over who can access their content and why. Unlike its leading competitors, such as OpenAI and Anthropic, Google does not comply with this principle for Googlebot, which is used for multiple purposes (search indexing, AI training, and inference/grounding). Simply requiring Google to develop a new opt-out mechanism would not allow publishers to achieve meaningful control over the use of their content. 

The most effective way to give publishers that necessary control is to require Googlebot to be split up into separate crawlers. That way, publishers could allow crawling for traditional search indexing, which they need to attract traffic to their sites, but block access for unwanted use of their content in generative AI services and features.

To ensure a fair digital ecosystem, the CMA must instead empower content owners to prevent Google from accessing their data for particular purposes in the first place, rather than relying on Google-managed workarounds after the crawler has already accessed the content for other purposes. That approach also enables creators to set conditions for access to their content. 

Although the CMA described crawler separation as an “equally effective intervention”, it ultimately rejected mandating separation based on Google’s input that it would be too onerous. We disagree.

Requiring Google to split up Googlebot by purpose — just like Google already does for its nearly 20 other crawlers — is not only technically feasible, but also a necessary and proportionate remedy that empowers website operators to have the granular control they currently lack, without increasing traffic load from crawlers to their websites (and in fact, perhaps even decreasing it, should they choose to block AI crawling).

To be clear, a crawler separation remedy benefits AI companies, by leveling the playing field between them and Google, in addition to giving UK-based publishers more control over their content. (There has been widespread public support for a crawler separation remedy by Daily Mail Group, the Guardian and the News Media Association.) Mandatory crawler separation is not a disadvantage to Google, nor does it undermine investment in AI. On the contrary, it is a pro-competitive safeguard that prevents Google from leveraging its search monopoly to gain an unfair advantage in the AI market. By decoupling these functions, we ensure that AI development is driven by fair-market competition rather than the exploitation of a single hyperscaler’s dominance.

******

The UK has a unique chance to lead the world in protecting the value of original and high-quality content on the Internet. However, we worry that the current proposals fall short. We would encourage rules that ensure that Google operates under the same conditions for content access as other AI developers, meaningfully restoring agency to publishers and paving the way for new business models promoting content monetization.

Cloudflare remains committed to engaging with the CMA and other partners during upcoming consultations to provide evidence-based data to help shape a final decision on conduct requirements that are targeted, proportional, and effective. The CMA still has an opportunity to ensure that the Internet becomes a fair marketplace for content creators and smaller AI players—not just a select few tech giants.

To mitigate this threat, Cloudflare is helping to migrate the Internet to Post-Quantum (PQ) cryptography. Today, about 50% of traffic to Cloudflare's edge network is protected against the most urgent threat: an attacker who can intercept and store encrypted traffic today and then decrypt it in the future with the help of a quantum computer. This is referred to as the harvest now, decrypt later threat.

However, this is just one of the threats we need to address. A quantum computer can also be used to crack a server's TLS certificate, allowing an attacker to impersonate the server to unsuspecting clients. The good news is that we already have PQ algorithms we can use for quantum-safe authentication. The bad news is that adoption of these algorithms in TLS will require significant changes to one of the most complex and security-critical systems on the Internet: the Web Public-Key Infrastructure (WebPKI).

The central problem is the sheer size of these new algorithms: signatures for ML-DSA-44, one of the most performant PQ algorithms standardized by NIST, are 2,420 bytes long, compared to just 64 bytes for ECDSA-P256, the most popular non-PQ signature in use today; and its public keys are 1,312 bytes long, compared to just 64 bytes for ECDSA. That's a roughly 20-fold increase in size. Worse yet, the average TLS handshake includes a number of public keys and signatures, adding up to 10s of kilobytes of overhead per handshake. This is enough to have a noticeable impact on the performance of TLS.

That makes drop-in PQ certificates a tough sell to enable today: they don’t bring any security benefit before Q-day — the day a cryptographically relevant quantum computer arrives — but they do degrade performance. We could sit and wait until Q-day is a year away, but that’s playing with fire. Migrations always take longer than expected, and by waiting we risk the security and privacy of the Internet, which is dear to us.

It's clear that we must find a way to make post-quantum certificates cheap enough to deploy today by default for everyone — not just those that can afford it. In this post, we'll introduce you to the plan we’ve brought together with industry partners to the IETF to redesign the WebPKI in order to allow a smooth transition to PQ authentication with no performance impact (and perhaps a performance improvement!). We'll provide an overview of one concrete proposal, called Merkle Tree Certificates (MTCs), whose goal is to whittle down the number of public keys and signatures in the TLS handshake to the bare minimum required.

But talk is cheap. We know from experience that, as with any change to the Internet, it's crucial to test early and often. Today we're announcing our intent to deploy MTCs on an experimental basis in collaboration with Chrome Security. In this post, we'll describe the scope of this experiment, what we hope to learn from it, and how we'll make sure it's done safely.

Why does the TLS handshake have so many public keys and signatures?

Let's start with Cryptography 101. When your browser connects to a website, it asks the server to authenticate itself to make sure it's talking to the real server and not an impersonator. This is usually achieved with a cryptographic primitive known as a digital signature scheme (e.g., ECDSA or ML-DSA). In TLS, the server signs the messages exchanged between the client and server using its secret key, and the client verifies the signature using the server's public key. In this way, the server confirms to the client that they've had the same conversation, since only the server could have produced a valid signature.

If the client already knows the server's public key, then only 1 signature is required to authenticate the server. In practice, however, this is not really an option. The web today is made up of around a billion TLS servers, so it would be unrealistic to provision every client with the public key of every server. What's more, the set of public keys will change over time as new servers come online and existing ones rotate their keys, so we would need some way of pushing these changes to clients.

This scaling problem is at the heart of the design of all PKIs.

Instead of expecting the client to know the server's public key in advance, the server might just send its public key during the TLS handshake. But how does the client know that the public key actually belongs to the server? This is the job of a certificate.

A certificate binds a public key to the identity of the server — usually its DNS name, e.g., cloudflareresearch.com. The certificate is signed by a Certification Authority (CA) whose public key is known to the client. In addition to verifying the server's handshake signature, the client verifies the signature of this certificate. This establishes a chain of trust: by accepting the certificate, the client is trusting that the CA verified that the public key actually belongs to the server with that identity.

Clients are typically configured to trust many CAs and must be provisioned with a public key for each. Things are much easier however, since there are only 100s of CAs instead of billions. In addition, new certificates can be created without having to update clients.

These efficiencies come at a relatively low cost: for those counting at home, that's +1 signature and +1 public key, for a total of 2 signatures and 1 public key per TLS handshake.

That's not the end of the story, however. As the WebPKI has evolved, so have these chains of trust grown a bit longer. These days it's common for a chain to consist of two or more certificates rather than just one. This is because CAs sometimes need to rotate their keys, just as servers do. But before they can start using the new key, they must distribute the corresponding public key to clients. This takes time, since it requires billions of clients to update their trust stores. To bridge the gap, the CA will sometimes use the old key to issue a certificate for the new one and append this certificate to the end of the chain.

That's +1 signature and +1 public key, which brings us to 3 signatures and 2 public keys. And we still have a little ways to go.

The main job of a CA is to verify that a server has control over the domain for which it’s requesting a certificate. This process has evolved over the years from a high-touch, CA-specific process to a standardized, mostly automated process used for issuing most certificates on the web. (Not all CAs fully support automation, however.) This evolution is marked by a number of security incidents in which a certificate was mis-issued to a party other than the server, allowing that party to impersonate the server to any client that trusts the CA.

Automation helps, but attacks are still possible, and mistakes are almost inevitable. Earlier this year, several certificates for Cloudflare's encrypted 1.1.1.1 resolver were issued without our involvement or authorization. This apparently occurred by accident, but it nonetheless put users of 1.1.1.1 at risk. (The mis-issued certificates have since been revoked.)

Ensuring mis-issuance is detectable is the job of the Certificate Transparency (CT) ecosystem. The basic idea is that each certificate issued by a CA gets added to a public log. Servers can audit these logs for certificates issued in their name. If ever a certificate is issued that they didn't request itself, the server operator can prove the issuance happened, and the PKI ecosystem can take action to prevent the certificate from being trusted by clients.

Major browsers, including Firefox and Chrome and its derivatives, require certificates to be logged before they can be trusted. For example, Chrome, Safari, and Firefox will only accept the server's certificate if it appears in at least two logs the browser is configured to trust. This policy is easy to state, but tricky to implement in practice:

1. Operating a CT log has historically been fairly expensive. Logs ingest billions of certificates over their lifetimes: when an incident happens, or even just under high load, it can take some time for a log to make a new entry available for auditors.

2. Clients can't really audit logs themselves, since this would expose their browsing history (i.e., the servers they wanted to connect to) to the log operators.

The solution to both problems is to include a signature from the CT log along with the certificate. The signature is produced immediately in response to a request to log a certificate, and attests to the log's intent to include the certificate in the log within 24 hours.

Per browser policy, certificate transparency adds +2 signatures to the TLS handshake, one for each log. This brings us to a total of 5 signatures and 2 public keys in a typical handshake on the public web.

The WebPKI is a living, breathing, and highly distributed system. We've had to patch it a number of times over the years to keep it going, but on balance it has served our needs quite well — until now.

Previously, whenever we needed to update something in the WebPKI, we would tack on another signature. This strategy has worked because conventional cryptography is so cheap. But 5 signatures and 2 public keys on average for each TLS handshake is simply too much to cope with for the larger PQ signatures that are coming.

The good news is that by moving what we already have around in clever ways, we can drastically reduce the number of signatures we need.

Merkle Tree Certificates (MTCs) is a proposal for the next generation of the WebPKI that we are implementing and plan to deploy on an experimental basis. Its key features are as follows:

1. All the information a client needs to validate a Merkle Tree Certificate can be disseminated out-of-band. If the client is sufficiently up-to-date, then the TLS handshake needs just 1 signature, 1 public key, and 1 Merkle tree inclusion proof. This is quite small, even if we use post-quantum algorithms.

2. The MTC specification makes certificate transparency a first class feature of the PKI by having each CA run its own log of exactly the certificates they issue.

Let's poke our head under the hood a little. Below we have an MTC generated by one of our internal tests. This would be transmitted from the server to the client in the TLS handshake:

-----BEGIN CERTIFICATE-----
MIICSzCCAUGgAwIBAgICAhMwDAYKKwYBBAGC2ksvADAcMRowGAYKKwYBBAGC2ksv
AQwKNDQzNjMuNDguMzAeFw0yNTEwMjExNTMzMjZaFw0yNTEwMjgxNTMzMjZaMCEx
HzAdBgNVBAMTFmNsb3VkZmxhcmVyZXNlYXJjaC5jb20wWTATBgcqhkjOPQIBBggq
hkjOPQMBBwNCAARw7eGWh7Qi7/vcqc2cXO8enqsbbdcRdHt2yDyhX5Q3RZnYgONc
JE8oRrW/hGDY/OuCWsROM5DHszZRDJJtv4gno2wwajAOBgNVHQ8BAf8EBAMCB4Aw
EwYDVR0lBAwwCgYIKwYBBQUHAwEwQwYDVR0RBDwwOoIWY2xvdWRmbGFyZXJlc2Vh
cmNoLmNvbYIgc3RhdGljLWN0LmNsb3VkZmxhcmVyZXNlYXJjaC5jb20wDAYKKwYB
BAGC2ksvAAOB9QAAAAAAAAACAAAAAAAAAAJYAOBEvgOlvWq38p45d0wWTPgG5eFV
wJMhxnmDPN1b5leJwHWzTOx1igtToMocBwwakt3HfKIjXYMO5CNDOK9DIKhmRDSV
h+or8A8WUrvqZ2ceiTZPkNQFVYlG8be2aITTVzGuK8N5MYaFnSTtzyWkXP2P9nYU
Vd1nLt/WjCUNUkjI4/75fOalMFKltcc6iaXB9ktble9wuJH8YQ9tFt456aBZSSs0
cXwqFtrHr973AZQQxGLR9QCHveii9N87NXknDvzMQ+dgWt/fBujTfuuzv3slQw80
mibA021dDCi8h1hYFQAA
-----END CERTIFICATE-----

Looks like your average PEM encoded certificate. Let's decode it and look at the parameters:

$ openssl x509 -in merkle-tree-cert.pem -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 531 (0x213)
        Signature Algorithm: 1.3.6.1.4.1.44363.47.0
        Issuer: 1.3.6.1.4.1.44363.47.1=44363.48.3
        Validity
            Not Before: Oct 21 15:33:26 2025 GMT
            Not After : Oct 28 15:33:26 2025 GMT
        Subject: CN=cloudflareresearch.com
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:70:ed:e1:96:87:b4:22:ef:fb:dc:a9:cd:9c:5c:
                    ef:1e:9e:ab:1b:6d:d7:11:74:7b:76:c8:3c:a1:5f:
                    94:37:45:99:d8:80:e3:5c:24:4f:28:46:b5:bf:84:
                    60:d8:fc:eb:82:5a:c4:4e:33:90:c7:b3:36:51:0c:
                    92:6d:bf:88:27
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Subject Alternative Name:
                DNS:cloudflareresearch.com, DNS:static-ct.cloudflareresearch.com
    Signature Algorithm: 1.3.6.1.4.1.44363.47.0
    Signature Value:
        00:00:00:00:00:00:02:00:00:00:00:00:00:00:02:58:00:e0:
        44:be:03:a5:bd:6a:b7:f2:9e:39:77:4c:16:4c:f8:06:e5:e1:
        55:c0:93:21:c6:79:83:3c:dd:5b:e6:57:89:c0:75:b3:4c:ec:
        75:8a:0b:53:a0:ca:1c:07:0c:1a:92:dd:c7:7c:a2:23:5d:83:
        0e:e4:23:43:38:af:43:20:a8:66:44:34:95:87:ea:2b:f0:0f:
        16:52:bb:ea:67:67:1e:89:36:4f:90:d4:05:55:89:46:f1:b7:
        b6:68:84:d3:57:31:ae:2b:c3:79:31:86:85:9d:24:ed:cf:25:
        a4:5c:fd:8f:f6:76:14:55:dd:67:2e:df:d6:8c:25:0d:52:48:
        c8:e3:fe:f9:7c:e6:a5:30:52:a5:b5:c7:3a:89:a5:c1:f6:4b:
        5b:95:ef:70:b8:91:fc:61:0f:6d:16:de:39:e9:a0:59:49:2b:
        34:71:7c:2a:16:da:c7:af:de:f7:01:94:10:c4:62:d1:f5:00:
        87:bd:e8:a2:f4:df:3b:35:79:27:0e:fc:cc:43:e7:60:5a:df:
        df:06:e8:d3:7e:eb:b3:bf:7b:25:43:0f:34:9a:26:c0:d3:6d:
        5d:0c:28:bc:87:58:58:15:00:00

While some of the parameters probably look familiar, others will look unusual. On the familiar side, the subject and public key are exactly what we might expect: the DNS name is cloudflareresearch.com and the public key is for a familiar signature algorithm, ECDSA-P256. This algorithm is not PQ, of course — in the future we would put ML-DSA-44 there instead.

On the unusual side, OpenSSL appears to not recognize the signature algorithm of the issuer and just prints the raw OID and bytes of the signature. There's a good reason for this: the MTC does not have a signature in it at all! So what exactly are we looking at?

The trick to leave out signatures is that a Merkle Tree Certification Authority (MTCA) produces its signatureless certificates in batches rather than individually. In place of a signature, the certificate has an inclusion proof of the certificate in a batch of certificates signed by the MTCA.

To understand how inclusion proofs work, let's think about a slightly simplified version of the MTC specification. To issue a batch, the MTCA arranges the unsigned certificates into a data structure called a Merkle tree that looks like this:

Each leaf of the tree corresponds to a certificate, and each inner node is equal to the hash of its children. To sign the batch, the MTCA uses its secret key to sign the head of the tree. The structure of the tree guarantees that each certificate in the batch was signed by the MTCA: if we tried to tweak the bits of any one of the certificates, the treehead would end up having a different value, which would cause the signature to fail.

An inclusion proof for a certificate consists of the hash of each sibling node along the path from the certificate to the treehead:

Given a validated treehead, this sequence of hashes is sufficient to prove inclusion of the certificate in the tree. This means that, in order to validate an MTC, the client also needs to obtain the signed treehead from the MTCA.

This is the key to MTC's efficiency:

1. Signed treeheads can be disseminated to clients out-of-band and validated offline. Each validated treehead can then be used to validate any certificate in the corresponding batch, eliminating the need to obtain a signature for each server certificate.

2. During the TLS handshake, the client tells the server which treeheads it has. If the server has a signatureless certificate covered by one of those treeheads, then it can use that certificate to authenticate itself. That's 1 signature,1 public key and 1 inclusion proof per handshake, both for the server being authenticated.

Now, that's the simplified version. MTC proper has some more bells and whistles. To start, it doesn’t create a separate Merkle tree for each batch, but it grows a single large tree, which is used for better transparency. As this tree grows, periodically (sub)tree heads are selected to be shipped to browsers, which we call landmarks. In the common case browsers will be able to fetch the most recent landmarks, and servers can wait for batch issuance, but we need a fallback: MTC also supports certificates that can be issued immediately and don’t require landmarks to be validated, but these are not as small. A server would provision both types of Merkle tree certificates, so that the common case is fast, and the exceptional case is slow, but at least it’ll work.

Ever since early designs for MTCs emerged, we’ve been eager to experiment with the idea. In line with the IETF principle of “running code”, it often takes implementing a protocol to work out kinks in the design. At the same time, we cannot risk the security of users. In this section, we describe our approach to experimenting with aspects of the Merkle Tree Certificates design without changing any trust relationships.

Let’s start with what we hope to learn. We have lots of questions whose answers can help to either validate the approach, or uncover pitfalls that require reshaping the protocol — in fact, an implementation of an early MTC draft by Maximilian Pohl and Mia Celeste did exactly this. We’d like to know:

What breaks? Protocol ossification (the tendency of implementation bugs to make it harder to change a protocol) is an ever-present issue with deploying protocol changes. For TLS in particular, despite having built-in flexibility, time after time we’ve found that if that flexibility is not regularly used, there will be buggy implementations and middleboxes that break when they see things they don’t recognize. TLS 1.3 deployment took years longer than we hoped for this very reason. And more recently, the rollout of PQ key exchange in TLS caused the Client Hello to be split over multiple TCP packets, something that many middleboxes weren't ready for.

What is the performance impact? In fact, we expect MTCs to reduce the size of the handshake, even compared to today's non-PQ certificates. They will also reduce CPU cost: ML-DSA signature verification is about as fast as ECDSA, and there will be far fewer signatures to verify. We therefore expect to see a reduction in latency. We would like to see if there is a measurable performance improvement.

What fraction of clients will stay up to date? Getting the performance benefit of MTCs requires the clients and servers to be roughly in sync with one another. We expect MTCs to have fairly short lifetimes, a week or so. This means that if the client's latest landmark is older than a week, the server would have to fallback to a larger certificate. Knowing how often this fallback happens will help us tune the parameters of the protocol to make fallbacks less likely.

In order to answer these questions, we are implementing MTC support in our TLS stack and in our certificate issuance infrastructure. For their part, Chrome is implementing MTC support in their own TLS stack and will stand up infrastructure to disseminate landmarks to their users.

As we've done in past experiments, we plan to enable MTCs for a subset of our free customers with enough traffic that we will be able to get useful measurements. Chrome will control the experimental rollout: they can ramp up slowly, measuring as they go and rolling back if and when bugs are found.

Which leaves us with one last question: who will run the Merkle Tree CA?

Standing up a proper CA is no small task: it takes years to be trusted by major browsers. That’s why Cloudflare isn’t going to become a “real” CA for this experiment, and Chrome isn’t going to trust us directly.

Instead, to make progress on a reasonable timeframe, without sacrificing due diligence, we plan to "mock" the role of the MTCA. We will run an MTCA (on Workers based on our StaticCT logs), but for each MTC we issue, we also publish an existing certificate from a trusted CA that agrees with it. We call this the bootstrap certificate. When Chrome’s infrastructure pulls updates from our MTCA log, they will also pull these bootstrap certificates, and check whether they agree. Only if they do, they’ll proceed to push the corresponding landmarks to Chrome clients. In other words, Cloudflare is effectively just “re-encoding” an existing certificate (with domain validation performed by a trusted CA) as an MTC, and Chrome is using certificate transparency to keep us honest.

With almost 50% of our traffic already protected by post-quantum encryption, we’re halfway to a fully post-quantum secure Internet. The second part of our journey, post-quantum certificates, is the hardest yet though. A simple drop-in upgrade has a noticeable performance impact and no security benefit before Q-day. This means it’s a hard sell to enable today by default. But here we are playing with fire: migrations always take longer than expected. If we want to keep an ubiquitously private and secure Internet, we need a post-quantum solution that’s performant enough to be enabled by default today.

Merkle Tree Certificates (MTCs) solves this problem by reducing the number of signatures and public keys to the bare minimum while maintaining the WebPKI's essential properties. We plan to roll out MTCs to a fraction of free accounts by early next year. This does not affect any visitors that are not part of the Chrome experiment. For those that are, thanks to the bootstrap certificates, there is no impact on security.

We’re excited to keep the Internet fast and secure, and will report back soon on the results of this experiment: watch this space! MTC is evolving as we speak, if you want to get involved, please join the IETF PLANTS mailing list.

Historically, the common way to deploy a Google tag meant serving the JavaScript payload directly from Google’s domain. This can work quite well, but can sometimes impact performance and accurate data measurement. That’s why Google developed a way to deploy a Google tag using your own first-party infrastructure using server-side tagging. However, this server-side tagging required deploying and maintaining a separate server, which comes with a cost and requires maintenance.

That’s why we’re excited to be Google’s launch partner and announce our direct integration of Google tag gateway for advertisers, providing many of the same performance and accuracy benefits of server-side tagging without the overhead of maintaining a separate server.  

Any domain proxied through Cloudflare can now serve your Google tags directly from that domain. This allows you to get better measurement signals for your website and can enhance your campaign performance, with early testers seeing on average an 11% uplift in data signals. The setup only requires a few clicks — if you already have a Google tag snippet on the page, no changes to that tag are required.

Oh, did we mention it’s free? We’ve heard great feedback from customers who participated in a closed beta, and we are excited to open it up to all customers on any Cloudflare plan today.      

Google Tag Manager is the most used tag management solution: it makes a complex tagging ecosystem easy to use and requires less effort from web developers. That’s why we’re collaborating with the Ads measurement and analytics teams at Google to make the integration with Google tag gateway for advertisers as seamless and accessible as possible.

Site owners will have two options of where to enable this feature: in the Google tag console, or via the Cloudflare dashboard. When logging into the Google tag console, you’ll see an option to enable Google tag gateway for advertisers in the Admin settings tab. 

Alternatively, if you already know your tag ID and have admin access to your site’s Cloudflare account, you can enable the feature, edit the measurement ID and path directly from the Cloudflare dashboard: 

Before, if site owners wanted to serve first-party tags from their own domain, they had to set up a complex configuration: create a CNAME entry for a new subdomain, create an Origin Rule to forward requests, and a Transform Rule to include geolocation information.

This new integration dramatically simplifies the setup and makes it a one-click integration by leveraging Cloudflare's position as a reverse proxy for your domain. 

In Google Tag Manager’s Admin settings, you can now connect your Cloudflare account and configure your measurement ID directly in Google, and it will push your config to Cloudflare. 

When you enable the Google tag gateway for advertisers, specific calls to Google’s measurement servers from your website are intercepted and re-routed through your domain. The result: instead of the browser directly requesting the tag script from a Google domain (e.g. www.googletagmanager.com), the request is routed seamlessly through your own domain (e.g. www.example.com/metrics).

Cloudflare acts as an intermediary for these requests. It first securely fetches the necessary Google tag JavaScript files from Google's servers in the background, then serves these scripts back to the end user's browser from your domain. This makes the request appear as a first-party request.

A bit more on how this works: When a browser requests https://example.com/gtag/js?id=G-XXXX, Cloudflare intercepts and rewrites the path into the original Google endpoint, preserving all query-string parameters and normalizing the Origin and Referer headers to match Google’s expectations. It then fetches the script on your behalf, and routes all subsequent measurement payloads through the same first-party proxy to the appropriate Google collection endpoints.

This setup also impacts how cookies are stored from your domain. A cookie is a small text file that a website asks your browser to store on your computer. When you visit other pages on that same website, or return later, your browser sends that cookie back to the website's server. This allows the site to remember information about you or your preferences, like whether a user is logged in, items in a shopping cart, or, in the case of analytics and advertising, an identifier to recognize your browser across visits.

With Cloudflare’s integration with Google tag gateway for advertisers, the tag script itself is delivered from your own domain. When this script instructs the browser to set a cookie, the cookie is created and stored under your website's domain. 

Detailed instructions to get started can be found here. You can also log in to your Cloudflare Dashboard, navigate to the Engagement Tab, and select Google tag gateway in the navigation to set it up directly in the Cloudflare dashboard.

One of the great things about container technology is that it delivers the same experience and functionality across different platforms. This frees you as a developer from having to rewrite or update your application to deploy it on a new cloud provider—or lets you run it across multiple cloud providers. With a containerized application running on multiple clouds, you can avoid lock-in, run your application on the cloud for which it’s best suited, and lower your overall costs.

If you’re using Kubernetes, you probably manage traffic to clusters and services across multiple nodes using internal load-balancing services, which is the most common and practical approach. But if you’re running an application on multiple clouds, it can be hard to distribute traffic intelligently among them. In this blog post, we show you how to use Cloudflare Load Balancer in conjunction with Kubernetes so you can start to achieve the benefits of a multi-cloud configuration.

To continue reading follow the Google Cloud blog here or if you are ready to get started we created a guide on how to deploy an application using Kubernetes on GCP and AWS along with our Cloudflare Load Balancer.

When Cloudflare started, our company needed two things: an initial group of users, and the finances to fund our development. We know most developers face the same issues. The Cloudflare Apps Platform solves the first problem by allowing third parties to develop applications that can be delivered across Cloudflare's edge network to any of the six million sites powered by Cloudflare. The Cloudflare Developer Fund alleviates the second by giving developers the financial support they need to fund their company. Today, we are excited to announce another initiative that will make it possible for developers to make their app dreams a reality.

Cloudflare and Google Cloud are working together to offer developers the resources needed to quickly launch and scale Cloudflare Apps. This partnership will give any Cloudflare Apps developer the chance to access a wide range of benefits including $3k - $100k of Google Cloud Platform (GCP) for one year at no cost. Some startups will also be eligible for 24/7 technical support, and access to GCP’s technical solutions team. This supports a core belief of the Cloudflare Apps initiative: we want developers to focus on building great Apps, not worry about paying for infrastructure. Hundreds of startups have already built successful applications on Cloudflare Apps and those applications have grown to serve hundreds of thousands of users. This program with Google Cloud significantly decreases the friction to getting up and running on Cloudflare Apps, allowing the next generation of developers and startups to make their living by building Apps.

$100k for Exceptional Apps: After an approval process your App could be awarded $20k in Cloud Credits, extendable to $100k based on usage in the first year.

Up to $3,000 for early stage startups: If you are an early-stage startup you are entitled to a $3,000 Google Cloud credit. Even if you aren't quite a startup yet, you are entitled to $500 if you are a first-time Google Cloud Platform user, and $200 if you are an existing user.

[Collect your credits now!](Offer discontinued)

Come learn about Cloudflare’s recent partnership with Google Cloud Platform (GCP) through their CDN Interconnect Program. Cloudflare offers performance and security to over 25,000 Google Cloud Platform customers. The CDN Interconnect program allows Cloudflare’s servers to establish high-speed interconnections with Google Cloud Platform at various locations around the world, accelerating dynamic content while reducing bandwidth and egress billing costs.

We’ll be at booth C7 discussing the benefits of Cloudflare, our partnership with Google Cloud Platform, and handing out Cloudflare SWAG. In addition, our Co-Founder, Michelle Zatlyn, will be presenting “A Cloud Networking Blueprint for Securing Your Workloads” on Thursday, March 9th from 11:20 AM to 12:20 PM at Moscone West, Room 2005.

Google Cloud Platform’s CDN Interconnect program allows select CDN providers to establish direct interconnect links with Google’s edge network at various locations. Customers egressing network traffic from Google Cloud Platform through one of these links will benefit from the direct connectivity to the CDN providers and will be billed according to the lower Google Cloud Interconnect pricing.

Joint customers of Cloudflare and Google Cloud Platform can expect bandwidth savings of up to 75% and receive discounted egress pricing. Egress traffic is traffic flowing from Google Cloud Platform servers to Cloudflare’s servers. The high-speed interconnections between GCP and Cloudflare speed up the delivery of dynamic content for visitors.

As part of this program, 41 Cloudflare data centers are directly connected to Google Cloud Platform’s infrastructure. When one of these Cloudflare data centers requests content from a Google Cloud Platform origin, it’s routed through a high-performance interconnect instead of the public Internet. This dramatically reduces latency for origin requests, and it also enables discounted Google Cloud Platform egress pricing in the US, Europe and Asia regions.

Quizlet and Discord, two prominent joint customers of Cloudflare and Google Cloud Platform, have shared their performance, security, and cost-savings stories.

Discord is a free voice and text chat app designed specifically for gaming. In one year, Discord grew from 25,000 concurrent users to 2.4 million, a 9,000 percent growth. Discord’s 25 million registered users send 100 million messages per day across the platform, requiring a global presence with tremendous amounts of network throughput. As Discord experiences explosive growth, they're thankful Cloudflare helps keep bandwidth & hardware costs down and web performance high.

• Saving $100,000 on annual hardware costs

• Saving $100,000 monthly on Google Cloud Network Egress bill

• Secure traffic even with spikes of websockets events up to 2 million/second

Learn more about Discord’s use of Cloudflare on Google Cloud Platform: https://www.cloudflare.com/case-studies/discord

Quizlet is the world’s largest student and teacher online learning community. Every month, over 20 million active learners from 130 countries practice and master more than 140 million study sets of content on every conceivable subject and topic. Quizlet’s Alexa ranking is 588 globally, and 104 in the United States, ranking it as one of the most highly-trafficked websites.

Quizlet receives performance and security benefits, while saving more than 50 percent on their Google Cloud networking egress bill by using Cloudflare.

• Save 50% on monthly Google Cloud Network Egress Bill

• Reduced daily bandwidth use by 76 percent reduction (or over 10 Tb)

Learn more about Quizlet’s use of Cloudflare on Google Cloud Platform:https://www.cloudflare.com/case-studies/quizlet

Cloudflare’s Co-Founder, Michelle Zatlyn, will be presenting alongside Google and Palo Alto Networks, in a talk titled “A Cloud Networking Blueprint for Securing Your Workloads”.

Thursday, March 9th | 11:20 AM - 12:20 PM | Moscone West, Room 2005

Securing your workloads in the cloud requires shifting away from the traditional “perimeter” security to a “pervasive, hierarchical, scalable” security model. In this session, we discuss cloud networking best practices for securing enterprise and cloud-native workloads on Google Cloud Platform. We describe a network security blueprint that covers securing your virtual networks (VPCs), DDoS protection, using third-party security appliances and services, and visibility and analytics for your deployments. We also highlight Google’s experiences in delivering its own services securely and future trends in cloud network security.

Since the very beginning, Cloudflare has offered two-factor authentication with Authy, and starting today we are expanding your options to keep your account safe with Google Authenticator and any Time-based One Time Password (TOTP) app of your choice.

If you want to get started right away, visit your account settings. Setting up Two-Factor with Google Authenticator or with any TOTP app is easy - just use the app to scan the barcode you see in the Cloudflare dashboard, enter the code the app returns, and you’re good to go.

Often when you hear that an account was ‘hacked’, it really means that the password was stolen.

If the media stopped saying 'hacking' and instead said 'figured out their password', people would take password security more seriously.

— Khalil Sehnaoui (@sehnaoui) January 5, 2017

Two-Factor authentication is sometimes thought of as something that should be used to protect important accounts, but the best practice is to always enable it when it is available. Without a second factor, any mishap involving your password can lead to a compromise. Journalist Mat Honan’s high profile compromise in 2012 is a great example of the importance of two-factor authentication. When he later wrote about the incident he said, "Had I used two-factor authentication for my Google account, it’s possible that none of this would have happened."

TOTP (Time-based One Time Password) is the mechanism that Google Authenticator, Authy and other two-factor authentication apps use to generate short-lived authentication codes. We’ve written previously on the blog about how TOTP works.

We didn’t want to limit you to only using two-factor providers that we'd built integrations with, so we built an open TOTP integration in the Cloudflare dashboard, allowing you to set up two-factor with any app that implements TOTP. That means you can choose from a wide array of apps for logging into Cloudflare securely with two-factor such as Symantec, Duo Mobile and 1Password.

If you want to enable Two-Factor Authentication with Google Authenticator or any other TOTP provider, visit your account settings here. It’s easy to set up and the best way to secure your account. We also have step by step instructions for you in our knowledge base.

In October 2015, the team at Google announced Accelerated Mobile Pages (AMP), a new, open technology to make the mobile web as fast as native apps. Since then, a large number of publishers have adopted AMP. Today, 600 million pages across 700,000 different domains are available in the AMP format.

The majority of traffic to this AMP content comes from people running searches on Google.com. If a visitor finds content through some source other than a Google search, even if the content can be served from AMP, it typically won't be. As a result, the mobile web continues to be slower than it needs to be.

Cloudflare's Accelerated Mobile Links helps solve this problem, making content, regardless of how it's discovered, app-quick. Once enabled, Accelerated Mobile Links automatically identifies links on a Cloudflare customer's site to content with an AMP version available. If a link is clicked from a mobile device, the AMP content will be loaded nearly instantly.

To see how it works, try viewing this post from your mobile device and clicking any of these links:

• TechCrunch: Cloudflare explains how FBI gag order impacted business

• ZDNet: CloudFlare figured out how to make the Web one second faster

• The Register: CloudFlare apologizes for Telia screwing you over

• AMP 8 Ball: Ask The Amp Magic 8-Ball A Yes Or No Question.

  

One of the benefits of Accelerated Mobile Links is that AMP content is loaded in a viewer directly on the site that linked to the content. As a result, when a reader is done consuming the AMP content closing the viewer returns them to the original source of the link. In that way, every Cloudflare customers' site can be more like a native mobile app, with the corresponding increase in user engagement.

For large publishers that want an even more branded experience, Cloudflare will offer the ability to customize the domain of the viewer to match the publisher's domain. This, for the first time, provides a seamless experience where AMP content can be consumed without having to send visitors to a Google owned domain. If you're a large publisher interested in customizing the Accelerated Mobile Links viewer, you can contact Cloudflare's team.

While Google was the initial champion of AMP, the technologies involved are open. We worked closely with the Google team in developing Cloudflare's Accelerated Mobile Links as well as our own AMP cache. Malte Ubl, the technical lead for the AMP Project at Google said of our collaboration:

"Working with Cloudflare on its AMP caching solution was as seamless as open-source development can be. Cloudflare has become a regular contributor on the project and made the code base better for all users of AMP. It is always a big step for a software project to go from supporting specific caches to many, and it is awesome to see Cloudflare’s elegant solution for this."

Cloudflare now powers the only compliant non-Google AMP cache with all the same performance and security benefits as Google.

In the spirit of open source, we're working to help develop updates to the project to address some of publishers' and end users' concerns. Specifically, here are some features we're developing to address concerns that have been expressed about AMP:

• Easier ways to share AMP content using publisher's original domains

• Automatically redirecting desktop visitors from the AMP version back to the original version of the content

• A way for end users who would prefer not to be redirected to the AMP version of content to opt out

• The ability for publishers to brand the AMP viewer and serve it from their own domain

Cloudflare is committed to the AMP project. Accelerated Mobile Links is the first AMP feature we're releasing, but we'll be doing more over the months to come. As of today, Accelerated Mobile Links is available to all Cloudflare customers for free. You can enable it in your Cloudflare Performance dashboard. Stay tuned for more AMP features that will continue to increase the speed of the mobile web.

But there are some non-browser tools that come in handy when working with HTTP/2. This blog post starts with a useful browser add-on, and then delves into command-line tools, load testing, conformance verification, development libraries and packet decoding for HTTP/2.

If you know of something that I've missed please write a comment.

For Google Chrome there's a handy HTTP/2 and SPDY Indicator extension that adds a colored lightning bolt to the browser bar showing the protocol being used when a web page is viewed.

The blue lightning bolt shown here indicates that the CloudFlare home page was served using HTTP/2:

A green lightning bolt indicates the site was served using SPDY and gives the SPDY version number. In this case SPDY/3.1:

A grey lightning bolt indicates that neither HTTP/2 no SPDY were used. Here the web page was served using HTTP/1.1.

There's a similar extension for Firefox.

There's also a handy online tool to check any individual web site.

CloudFlare also has a Google Chrome extension called Claire that gives information about how a web page was loaded. For example here's the information that Claire shows for a site using CloudFlare that uses IPv6, Railgun, and HTTP/2.

There's a handy command-line tool called is-http which is installed using npm as follows:

npm install -g is-http2-cli

Once installed you can check the HTTP/2 status of a web on the command-line:

$ is-http2 www.cloudflare.com
✓ HTTP/2 supported by www.cloudflare.com
Supported protocols: h2 spdy/3.1 http/1.1

$ is-http2 www.amazon.com
× HTTP/2 not supported by www.amazon.com
Supported protocols: http/1.1

The is-http tool is also useful because it gives you a list of the protocols advertised by the server. As you can see www.cloudflare.com supports HTTP/2, HTTP/1.1 and SPDY/3.1.

In version 7.43.0 the venerable curl tool got HTTP/2 support when it's linked with the nghttp library. To build curl from sources you'll need OpenSSL, zlib, nghttp2 and libev. I used the following sequence of commands.

$ curl -LO http://dist.schmorp.de/libev/libev-4.20.tar.gz
$ tar zvxf libev-4.20.tar.gz
$ cd libev-4.20
$ ./configure
$ make
$ sudo make install

$ curl -LO https://www.openssl.org/source/openssl-1.0.2d.tar.gz
$ tar zxvf openssl-1.0.2d.tar.gz
$ cd openssl-1.0.2d
$ ./config shared zlib-dynamic
$ make && make test
$ sudo make install

$ curl -LO http://zlib.net/zlib-1.2.8.tar.gz
$ tar zxvf zlib-1.2.8.tar.gz
$ cd zlib-1.2.8
$ ./configure
$ make && make test
$ sudo make install

$ curl -LO https://github.com/tatsuhiro-t/nghttp2/releases/download/v1.5.0/nghttp2-1.5.0.tar.gz
$ tar zxvf nghttp2-1.5.0.tar.gz
$ cd nghttp2-1.5.0
$ OPENSSL_CFLAGS="-I/usr/local/ssl/include" OPENSSL_LIBS="-L/usr/local/ssl/lib -lssl -lcrypto -ldl" ./configure
$ make
$ sudo make install

$ curl -LO http://curl.haxx.se/download/curl-7.46.0.tar.gz
$ tar zxvf curl-7.46.0.tar.gz
$ cd curl-7.46.0
$ ./configure
$ make && make test
$ sudo make install
$ sudo ldconfig

Once installed curl has a new --http2 option that causes it to use HTTP/2 (if it can). The -v verbose option will show information about the use of HTTP/2:

$ curl -vso /dev/null --http2 https://www.cloudflare.com/
[...]
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* TCP_NODELAY set
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0xc3dba0)
[...]

If you built curl using my instructions above you will have built and installed some tools that come with the nghttp2 library. One of those is a command-line client called nghttp. It can be used like curl to download from the web using HTTP/2 but it also has a handy verbose option that shows that actual HTTP/2 frames sent and received.

By running it with -nv you get HTTP/2 and throw away the actual downloaded page. Here's its output when downloading www.cloudflare.com using HTTP/2. On a terminal with color support it uses coloring to highlight different parts of the log.

Another curl-like command-line tool for HTTP/2 is h2c. It also enables dumping of HTTP/2 frames and has a nice feature that it runs in the background and keeps connections to servers alive and has a useful 'wiretap' feature for intercepting an HTTP/2 for debugging.

If you have Go 1.5.1 installed then you can download it as follows:

$ export GO15VENDOREXPERIMENT=1
$ go get github.com/fstab/h2c

You start h2c running by doing h2c start & which sets it running in the background. You can then communicate with a web server like this:

 $ h2c connect www.cloudflare.com
 $ h2c get /
 $ h2c disconnect

And it will perform the HTTP request. To see detailed output at the HTTP/2 you start h2c with the --dump parameter:

 $ h2c start --dump

You will then get detailed output dumped by that process, in color, of the HTTP/2 frames being used.

Details of the wiretap feature are in [this blog post](http://unrestful.io/2015/08/28/wiretap.html).

If you just want to find out what protocols a web site supports OpenSSL's s_client can be used. If you specify an empty -nextprotoneg option OpenSSL sends an empty TLS option asking for negotiation of the next protocol and the server responds with a complete list of protocols it supports.

$ openssl s_client -connect www.cloudflare.com:443 -nextprotoneg ''
CONNECTED(00000003)
Protocols advertised by server: h2, spdy/3.1, http/1.1

There you can see that www.cloudflare.com support HTTP/2 (the h2), SPDY/3.1 and HTTP/1.1.

If you want to do low level HTTP/2 debugging there's an interactive client call h2i. Once again it requires that you have Go installed. To get it, run

$ go get github.com/golang/net/http2/h2i

You can then use h2i to connect to a site that uses HTTP/2 and send it individual HTTP/2 frames. For example, here's the start of session connecting to www.cloudflare.com and requesting the home page using the headers command which allows you to type in a standard HTTP/1.1 request.

$ h2i www.cloudflare.com
Connecting to www.cloudflare.com:443 ...
Connected to 198.41.214.163:443
Negotiated protocol "h2"
[FrameHeader SETTINGS len=18]
 [MAX_CONCURRENT_STREAMS = 128]
 [INITIAL_WINDOW_SIZE = 2147483647]
 [MAX_FRAME_SIZE = 16777215]
[FrameHeader WINDOW_UPDATE len=4]
  Window-Increment = 2147418112

h2i> headers
 (as HTTP/1.1)> GET / HTTP/1.1
 (as HTTP/1.1)> Host: www.cloudflare.com
 (as HTTP/1.1)>
  Opening Stream-ID 1:
  :authority = www.cloudflare.com
  :method = GET
  :path = /
  :scheme = https
 [FrameHeader HEADERS flags=END_HEADERS stream=1 len=819]
   :status = "200"
   server = "cloudflare-nginx"
   date = "Fri, 04 Dec 2015 10:36:15 GMT"
   content-type = "text/html"
   last-modified = "Thu, 03 Dec 2015 22:27:41 G MT" 
   strict-transport-security = "max-age=31536000"
   x-content-type-options = "nosniff"
   x-frame-options = "SAMEORIGIN"
   cf-cache-status = "HIT"
   expires = "Fri, 04 Dec 2015 14:36:15 GMT"
   cache-control = "public, max-age=14400"
 [FrameHeader DATA stream=1 len=7261]
   "<!DOCTYPE html>\n<html>\n<head>\n<!

The nghttp2 library also includes a load testing tool called h2load which can be used a little like ab. There's a useful HOWTO on using the tool.

I ran it against the CloudFlare test server like this:

If you are testing an HTTP/2 implementation there's a useful tool called h2spec which runs through a conformance test against a real HTTP/2 server. To use it first install Go 1.5.1 and then do

$ go get github.com/summerwind/h2spec/cmd/h2spec

I only recommend it for testing locally running HTTP/2 servers. If you built nghttp2 above while building curl you will also have the nghttpd HTTP/2 server available. You can run h2spec against it like this:

$ nghttpd --no-tls -D -d /tmp 8888
$ h2spec -p 8888

h2spec will run a battery of tests against the server and output conformance information with a reference to the relevant part of RFC7540.

If you need to test a number of web sites to see which support HTTP/2 there's a tool called h2scan. Feed it a list of web sites and it will check to see if they support HTTPS, SPDY/3.1 and HTTP/2.

$ cat examples
www.cloudflare.com
www.amazon.com
www.yahoo.com
$ h2scan --fields < examples
name,resolves,port443Open,tlsWorks,httpsWorks,cloudflare,spdyAnnounced,http2Announced,spdyWorks,http2Works,npn
www.cloudflare.com,t,t,t,t,t,t,t,t,t,h2 spdy/3.1 http/1.1
www.amazon.com,t,t,t,t,f,f,f,-,-,http/1.1
www.yahoo.com,t,t,t,t,f,t,f,t,-,spdy/3.1 spdy/3 http/1.1 http/1.0

If you are working in C and need to add HTTP/2 support to a program there's the nghttp2 library that is full implementation of HTTP/2 with a simple interface. Their HTTP/2 client tutorial explains how to use the library to add HTTP/2 client capabilities. nghttp2 can also be used for servers.

Go programmers will find full HTTP/2 support will arrive with Go 1.6. If you can't wait until then there's an extension package for HTTP/2 golang.org/x/net/http2 . Details here.

There's a pure Ruby implementation of HTTP/2 available from Ilya Grigorik.

Haskell programmers can use the http2 hackage.

The popular Wireshark packet analyzer added decoding on HTTP/2 in version 1.12.0 and fully decodes HTTP/2 frames. Unfortunately most HTTP/2 is sent over TLS which means that, by default, Wireshark will not be able to decrypt the packets to be able to get to the HTTP/2 for decoding.

Fortunately, there is a workaround if you are using Google Chrome for testing. It is possible to get Chrome to save the symmetric cryptographic key used for TLS connections to a file and Wireshark is able to read that file to decode TLS connections.

This is done by setting the SSLKEYLOGFILE environment variable before running Chrome. I'm running on Mac OS X and use Google Chrome Canary for testing so I run:

% export SSLKEYLOGFILE=`pwd`/sslkey.log
% /Applications/Google\ Chrome\ Canary.app/Contents/MacOS/Google\ Chrome\ Canary

Google Chrome will then write session keys to that file. In Wireshark I configure it to read the file by going to Preferences, expanding the Protocols list.

Then I find SSL and set the Pre-Main-Secret log filename to point to the same file.

Then Wireshark can decode the TLS connections made by that browser. Here's the beginning of a connection between Google Chrome Canary and the experimental server https://http2.cloudflare.com/.

Finally, if you are looking at the performance of your own web site it can be handy to understand which parts of the page were downloaded using HTTP/2 and which were not. You can do that pretty easily using the Developer view in Google Chrome. Here's a shot of the CloudFlare blog loaded in Chrome. There's an additional Protocol fields available on the pop-up menu.

Once added you can sort by protocol to see which parts were HTTP/2 (the h2) and which were other protocols.

1. Short introduction to HTTP/2 from CloudFlare.

2. http2 explained from the creator of curl.

3. The home page of the HTTP/2 working group with lots of information plus a list of useful tools.

The past few years have been marked by tremendous growth for CloudFlare. At the time of our last fundraising in December 2012, CloudFlare was a team of 37 operating a network in 23 cities and 15 countries—today we number over 200 with a presence in 62 cities and 33 countries. We’ve grown from delivering 85 billion page views per month for 500 thousand customers to nearly 1 trillion each month across 4 million Internet properties, all the while protecting our customers from hundreds of billions of cyber threats. The growth and resonance of our service since CloudFlare’s founding 5 years ago is beyond our wildest of expectations, but it is only in the coming years that our scale and efforts to build a better Internet will become visible.

In 2016 alone we will more than double our global presence, increase the size of our network by an order of magnitude, and with that allow millions of new businesses and online publishers to accelerate and secure their online applications and harness the growing power of the Internet economy. Our service is built on the simple premise that any individual or business should be able to quickly and easily ensure the global performance and availability of their websites and mobile apps, and withstand withering and sophisticated cyber attacks—all without a single piece of hardware, or the need to build a global network and the team of engineers to manage it all.

To sustain this level of growth requires investment. In this pursuit we’re pleased to announce that we have raised $110 million from Fidelity, Google, Microsoft, Baidu, Qualcomm and our existing investors. Beyond the additional capital raised, the broad, strategic participation in the round from many of the world’s leading technology firms validates the opportunity ahead of us, and positions us for the next chapter of our growth.

But before we write this new chapter, a bit about how we got here.

Seismic shifts are the sorts of groundbreaking developments that fundamentally alter the ways in which organizations and industries operate. We came upon a few such shifts as CloudFlare was founded in 2009. Although these shifts were already well underway, their strength has increased logarithmically each year.

These shifts are the massive movement of commerce and communication online, and the rapidly evolving means by which organizations deploy applications, increasingly in the cloud. Had one started a business in 2009, their first call might very well have been to Dell or HP to purchase servers (ours was to HP!), followed by calls to Oracle and Microsoft for the databases and software necessary to deliver the applications across the final tier of Internet hierarchy, the network edge. It is in this last tier that companies like Cisco capitalized on the last seismic shift at the network edge—the need to process and manage an explosion of network traffic directed towards the applications, and storage and compute platforms beneath.

Excerpt from our fundraising presentation titled "Scalability and elasticity of the cloud will prevail"

Today it is almost inconceivable that an upstart business would purchase a single piece of hardware, much less call anyone to procure it. In less time, with less cost, and more elasticity, an organization can turn up any number of storage and compute instances on cloud based platforms. The on-premise, perpetual license-based software and databases of yesteryear are now also available as services. Even complex applications such as customer relationship management (CRM) and enterprise resource management (ERP) systems are now largely dominated by software as a service (SaaS) offerings. CloudFlare is now pioneering this same shift at the network edge. In other words, it’s time to say farewell to hardware appliances.

The network edge can be broadly described as the “control plane” for all traffic directed to the layers below. This control plane ensures that traffic to the layers below consistently reaches its destination, that authorized users receive appropriate access, and that hackers are kept at bay. Each of these appliances (e.g., firewalls, load balancers, DDoS mitigation appliances, WAN optimizers, malware scanners, authentication devices, among many others) look for patterns in traffic, apply stored rules against the patterns, and make network I/O decisions to block, re-route, and compress, among many other functions. Now, CloudFlare is able to perform each of these functions without a single piece of customer hardware, and across a global network with immensely greater scale and elasticity, and a small fraction of the latency and cost.

Excerpt from our fundraising presentation titled "The network edge now expands beyond the data center"

This shift at the network edge is further driven by its expansion, as well as the blurring of its perimeter (security jargon for the line delineating assets to be protected). This expansion is occurring across multiple vectors. Organizations now deploy their applications across multiple geographies to reduce latency, across disparate physical and cloud environments for redundancy and elasticity, and adopt third party SaaS applications to replace internal business systems. What used to be solved by hardware appliances is now far better solved by cloud services.

CloudFlare is the new control plane for the network edge. With a simple DNS change, traffic to an organization’s applications and store and compute environment are directed through CloudFlare’s global network no matter where they reside geographically, logically or virtually. With one of the largest edge networks globally, we can perform each of the aforementioned edge functions to accelerate, secure and ensure the availability of the applications behind us with blazing fast speed, absolute elasticity and an enormous return on investment compared to hardware based solutions.

This is our vision, and one that is now shared with many of the world’s leading technology firms (and investors). Each of Fidelity, Microsoft, Baidu, Qualcomm and Google are uniquely positioned to support CloudFlare’s evolution in a rapidly evolving environment, and address key questions. What is our mobile strategy? What is our China strategy? How are we going to succeed with large enterprises? Will other giants partner or compete with you? How does CloudFlare become a public company?

Our choice of investors should hopefully provide some insight into our answers to each of these questions. Moreover, each of the participants in our round agreed to invest in the company without any special control provisions, board representation, economic treatment or even information rights. It was important for us as a company to continue to execute against our vision without distraction.

We don’t precisely know, but if the past is any indicator, it may continue to be beyond our expectations. In a fun bit of CloudFlare history, we unearthed an e-mail between two of CloudFlare’s founders from May 2009.

What might have sounded delusional to many five years ago (and they probably would have been right at the time), is a reality today. Five years later, and built upon the contributions of a talented team, our steadfast customers, and an unparalleled group of investors, we are well on the path to building something big and unique.

We’re always looking for feedback. You asked for an intuitive, easy-to-use interface, and we delivered. Encryption and SSL certificates? Now free, and included with all plans. A larger network to serve your global visitor base? We’re 62 deep, and not stopping. This feedback has helped tremendously over the past 5 years, so please keep it coming as we tackle the next five.

We’re looking for new team members. Great people who are passionate about building a better Internet, and willing to tackle big problems. We are building a service that allows anyone to run a website or mobile app as fast and secure as the Internet giants, improving the surfing experience for over 2 billion Internet users, and empowering millions of businesses to securely transact online each day. If that sounds interesting, check out our job postings here.

Thanks for your support over the past five years.

—The CloudFlare team

We're inviting these hosted customers: don't wait...migrate your site to CloudFlare for global acceleration (and more) right now.

CC BY 2.0 image by RogerAs TechCrunch wrote: "In many ways, PageSpeed Service is similar to what CloudFlare does but without the focus on security."

PageSpeed started as — and continues — as a Google-created software module for the Apache webserver to rewrite webpages to reduce latency and bandwidth, to help make the web faster.

Google introduced their hosted PageSpeed Service in July 2011, to save webmasters the hassle of installing the software module.

It's the hosted service that is being discontinued.

CloudFlare provides similar capabilities to PageSpeed, such as minification, image compression, and asynchronous loading.

Additionally, CloudFlare offers more performance gains through a global network footprint, Railgun for dynamic content acceleration, built-in SPDY support, and more.

PageSpeed Service customers care about performance, and CloudFlare delivers. CloudFlare also includes security, SSL, DNS, and more, at all plans, including Free.

There's no need to install any software or change your host. CloudFlare works at the network level, with a five minute sign up process.

Here's simple instructions for migrating from the hosted PageSpeed Service to CloudFlare.

We’re excited to announce that CloudFlare has just been named a Google Cloud Platform Technology Partner. So what does this mean? Now, Google Cloud Platform customers can experience the best of both worlds—the power and protection of the CloudFlare community along with the flexibility and scalability of Google’s infrastructure.

We share many mutual customers with Google, and this collaboration makes it even easier for Google Cloud Platform customers to get started with CloudFlare.

When CloudFlare is enabled, Google Cloud Platform customers have their infrastructure extended directly to the network edge, giving them faster content delivery as well as heightened optimization and security.

• 2x Web Performance Speed - CloudFlare uses advanced caching and the SPDY protocol to double web content transfer speeds, making web content transfer times significantly faster.

• Datacenters at Your Customer’s Doorstep - CloudFlare’s global edge network caches static files close to their destination, meaning that content always loads fast no matter where customers are located. Also, CloudFlare peers with Google in strategic locations globally, improving response times for Google Cloud Platform services.

• Protection Against DDoS and SQL Injection Attacks - Because CloudFlare sits on the edge, customers are protected from malicious traffic before it reaches Google's Cloud Platform. This keeps websites up and running without compromising page load times or performance.

• ...and more! - Check out the full list of features here.

To get started and learn more about the partnership please visit: https://www.cloudflare.com/google.

_{CC BY SA 2.0 image by Quinn Dombrowski}

With the right dictionary for the right page the savings can be spectacular, even 70% smaller than gzip alone. In theory, a whole file can be replaced by a single token.

The drawbacks of the method are twofold: first - the dictionary that is created is fairly large and must be distributed as a separate file, in fact the dictionary is often larger than the individual pages it compresses; second - the dictionary is usually absolutely useless for another set of pages.

For large domains that are visited repeatedly the advantage is huge: at a cost of single dictionary download, all the following page views can be compressed with much higher efficiency. Currently we aware of Google and LinkedIn compressing content with SDCH.

Here at CloudFlare our task is to support millions of domains, which have little in common, and creating a single SDCH dictionary is very difficult. Nevertheless better compression is important, because it produces smaller payloads, which result in content being delivered faster to our clients. That is why we set out to find that little something that is common to all the pages and to see if we could compress them further.

Besides SDCH (which is only supported by the Chrome browser), the common compression methods over HTTP are gzip and DEFLATE. Some do not know it but the compression they perform is identical. The two formats differ in the content headers they use, with gzip having slightly larger headers, and the error detection function - gzip uses CRC32, whereas DEFLATE uses Adler32.

Usually the servers opt to compress with gzip, however its cousin DEFLATE supports a neat feature called "Preset Dictionary". This dictionary is not like the dictionary used by SDCH, in fact it is not a real dictionary. To understand how this "dictionary" can be used to our advantage, it is important to understand how the DEFLATE algorithm works.

_{CC BY 2.0 image by Caleb Roenigk}

The DEFLATE algorithm consists of two stages, first it performs the LZ77 algorithm, where it simply goes over the input, and replaces occurrences of previously encountered strings with (short) "directions" where the same string can be found in the previous input. The directions are a tuple of (length, distance), where distance tells how far back in the input the string was and length tells how many bytes were matched. The minimal length deflate will match is 3 (4 in the highly optimized implementation CloudFlare uses), the maximal length is 258, and the farthest distance back is 32KB.

This is an illustration of the LZ77 algorithm:

Input:

Output (length tokens are blue, distance tokens are red):

DEFLATE managed to reduce the original text from 251 characters, to just 152 tokens! Those tokens are later compressed further by Huffman encoding, which is the second stage.

How long and devotedly the algorithm searches for a string before it stops is defined by the compression level. For example with compression level 4 the algorithm will be happy to find a match of 16 bytes, whereas with level 9 it will attempt to look for the maximal 258 byte match. If a match was not found the algorithm outputs the input as is, uncompressed.

Clearly at the beginning of the input, there can be no references to previous strings, and it is always uncompressed. Similarly the first occurrence of any string in the input will never be compressed. For example almost all HTML files start with the string "<html ", however in this string only the second HTML will be replaced with a match, and the rest of the string will remain uncompressed.

To solve this problem the deflate dictionary effectively acts as an initial back reference for possible matches. So if we add the aforementioned string "<html " to the dictionary, the algorithm will be able to match it from the start, improving the compression ratio. And there are many more such strings that are used in any HTML page, which we can put in the dictionary to improve compression ratio. In fact the SPDY protocol uses this technique for HTTP header compression.

To illustrate, lets compress the children’s song with the help of a 42 byte dictionary, containing the following: Little bunny Foo hopping forest Good Fairy. The compressed output will then be:

Now, even strings at the very beginning of the input are compressed and strings that only appear once in the file are compressed as well. With the help of the dictionary we are down to 115 tokens. That means roughly 25% better compression rate.

We wanted to see if we could make a dictionary that would benefit ALL the HTML pages we serve, and not just a specific domain. To that end we scanned over 20,000 publicly available random HTML pages that passed through our servers on a random sunny day, we took the first 16KB of each page, and used them to prepare two dictionaries, one of 16KB and the other of 32KB. Using a larger dictionary is useless, because then it would be larger than the LZ77 window used by deflate.

To build a dictionary, I made a little go program that takes a set of files and performs "pseudo" LZ77 over them, finding strings that DEFLATE would not compress in the first 16KB of each input file. It then performs a frequency count of the individual strings, and scores them according to their length and frequency. In the end the highest scoring strings are saved into the dictionary file.

Our benchmark consists of another set of pages obtained in similar manner. The number of benchmarked files was about 19,000 with total size of 563MB.

We can see from the results that the compression we gain for level 4 is almost 5% better than without the dictionary, which is even greater than the compression gained by using level 9 compression, while being substantially faster. For level 9, the gain is greater than 5% without a significant performance hit.

The results highly depend on the dataset used for the dictionary and on the compressed pages. For example when making a dicitonary aimed at a specific web site, the compression rate for that site increased by up to 30%.

For very small pages, such as error pages, with size less than 1KB, a DEFLATE dictionary was able to gain compression rates of up to 50% smaller than DEFLATE alone.

Of course different dictionaries may be used for different file types. In fact we think that it would make sense to create a standard set of dictionaries that could be used accross the web.

The utility to make a dictionary for deflate can be found at https://github.com/vkrasnov/dictator.The optimized version of zlib used by CloudFlare can be found at https://github.com/cloudflare/zlib

CC BY 2.0 by JD Hancock

Last Monday we announced our SSL for Free plan users called Universal SSL. Universal SSL means that any site running on CloudFlare gets a free SSL certificate, and is automatically secured over HTTPS.

Using SSL for a web site helps make the site more secure, but there's another benefit: it can also make the site faster. That's because the SPDY protocol, created by Google to speed up the web, actually requires SSL and only web sites that support HTTPS can use SPDY.

CloudFlare has long supported SPDY, and kept up to date with improvements in the protocol. We currently support the most recent version of SPDY: 3.1.

CloudFlare's mission to bring the tools of the Internet giants to everyone is two fold: security and performance. As part of the Universal SSL launch, we also rolled out SPDY for everyone. Many of the web's largest sites use SPDY; now all sites that use CloudFlare are in the same league.

If your site is on CloudFlare, and you use a modern browser that supports SPDY, you'll find that the HTTPS version of your site is now served over SPDY. SPDY allows the browser to query for multiple objects in one request, and for the objects to be sent down the wire as they are ready to prevent hold-ups—that can be a big performance win.

Our goal is a faster, safer, more secure Internet. Universal SSL and SPDY for everyone are two big steps in that direction.

CC BY 2.0 by Gregg Tavares

First, Google just announced that they will begin taking into account whether a site supports HTTPS connections in their ranking algorithm. This means that if you care about SEO then ensuring your site supports HTTPS should be a top priority. Kudos to Google to giving webmasters a big incentive to add SSL to their sites.

Second, at CloudFlare we've cleared one of the last major technical hurdle before making SSL available for every one of our customers -- even free customers. One of the challenges we had was ensuring we still had the flexibility to move traffic to sites dynamically between the servers that make up our network. While we can do this easily when traffic is over an HTTP connection, when a connection uses HTTPS we need to ensure that the correct certificates are in place and loaded into memory before requests are processed by a server.

To accomplish this, we needed to redesign how certificates are loaded into a server's memory. Previously, we'd load certificates into memory before traffic was directed to a server. That creates challenges when dealing with millions of domains and when shifting traffic to help isolate or mitigate an attack.

Last week we pushed new code that allows us to "lazy load" SSL certificates on demand. This means that a certificate only needs to be in a data center, not on a particular server, before HTTPS traffic needing the certificate is directed to that server. When a request is received, the server can now dynamically retrieve the correct certificate even if it hasn't been previously loaded into memory. This allows us to continue to shift traffic to manage our network even if we are managing SSL certificates for millions of domains.

We're on track to roll out SSL for all CloudFlare customers by mid-October. When we do, the number of sites that support HTTPS on the Internet will more than double. That they'll also rank a bit higher is pretty cool too.

In the meantime, if you want a quick way to boost your Google ranking, upgrading to any paid CloudFlare account will enable HTTPS by default. Even before we make it free, it's already the fastest, easiest way to get HTTPS support on any site.

Go running on the Linux kernel has been used to build many important systems like RRDNS (our DNS server) at CloudFlare. Accurately, precisely and efficiently determining the time is an important part of many of the these systems.

To see why time is important, consider that humans have had some trouble convincing computers to keep time for them in the recent past. It been a bit more than a decade since we had to dust off our best COBOL programmers to tackle Y2K.

More recently, a bug in the handling of a leap second propagated through the Network Time Protocol (NTP) also took many systems off-line. As we've seen in recent days, NTP is very useful for synchronizing computer clocks and/or DDoSing them. The leap second bug received extensive coverage. Google was ready but many other popular sites were taken offline.

We also have the Year 2038 problem to look forward to. Hopefully there will still be a few engineers around then that remember what this 32-bit thing was all about.

Everything starts with the time package that is part of Go’s standard library. The time package provides types for Time, Duration, Ticker, Timer and various utility functions for manipulating these types.

The most commonly used function in this package is probably the time.Now function, which returns the current time as a Time struct. The Time struct has 3 fields:

type Time struct {
    sec int64
	nsec uintptr
	loc *Location
}

Location contains the timezone information for the time.

Duration is used to express the difference between two Times and to configure timers and tickers.

Timer is useful for implementing a timeout, typically as part of a select statement. Ticker can be used to wake up periodically, usually when you are using select in a for loop.

Go’s time package is used in many other places in the Go standard library. When dealing with socket connections that may go slow or stop sending data completely, one uses the SetDeadline functions that are part of the net.Conn interface.

We love writing tests at CloudFlare, and having unit tests that include some kind of random component can turn up interesting issues. You can use the current time to seed random number generators in tests, using:

rand.New(rand.NewSource(time.Now().UnixNano()))

If you’re generating random numbers for a secure application, you really want to be using the crypto/rand package. Interestingly, even the initialization of crypto/rand.Reader incorporates the current time.

The current time also features when one logs something using the log package.

A very useful service called Sourcegraph turns up more than 6000 examples where time.Now is used. For example, the Camlistore code base calls time.Now in about 130 different places.

With time.Now being as pervasive as it is, have you ever wondered how it works? Time to dive deeper.

The most important changes to the way in which Go programs keep time on Linux was committed on 8 November 2012 in changeset 42c8d3aadc40 Let’s analyse the commit message and the code for some clues:

runtime: use vDSO clock_gettime for time.now & runtime.nanotime
on Linux/amd64. Performance improvement aside, time.Now() now 
gets real nanosecond resolution on supported systems.

To understand this commit message better, we first need to review the system calls available on Linux for obtaining the value of the clock.

In the beginning, there was time and gettimeofday, which existed in SVr4, 4.3BSD and was described in POSIX.1-2001. time returns the number of seconds since the Unix epoch, 1970-01-01 00:00:00 UTC and is defined in C as:

time_t time(time_t *t)

time_t is 4 bytes on 32-bit platforms and 8 bytes on 64-bit platforms, hence the Y2038 problem mentioned above.

gettimeofday returns the number of seconds and microseconds since the epoch and is defined in C as:

int gettimeofday(struct timeval *tv, struct timezone *tz)

gettimeofday populates a struct timeval, which has the following fields:

struct timeval {
	time_t tv_sec; /* seconds */
	suseconds_t tv_usec; /* microseconds */
}

gettimeofday yields timestamps that only have microsecond precision. POSIX.1-2008 marks gettimeofday as obsolete, recommending the use of clock_gettime instead, which is defined in C as:

int clock_gettime(clockid_t clk_id, struct timespec *tp)

clock_gettime populates a struct timespec, which has the following fields:

struct timespec {
	time_t tv_sec; /* seconds */
	long tv_nsec; /* nanoseconds */
}

clock_gettime can yield timestamps that have nanosecond precision. The clock ID parameter determines the type of clock to use. Of interest to us are:

• CLOCK_REALTIME: a system-wide clock that measures real time. This clock is affected by discontinuous jumps in the system time and by incremental adjustments by made using the adjtime function or NTP.

• CLOCK_MONOTONIC: a clock that represents monotonic time since some unspecified starting point. This clock is not affected by discontinous jumps in the system time, but is affected by adjtime and NTP.

• CLOCK_MONOTONIC_RAW: similar to CLOCK_MONOTONIC, but not subject to adjustment by adjtime or NTP.

With this background we can look at the code for time.Now. (Hint: click on the function name in godoc to look at the code yourself.)

The time.Now function is implemented using a function called now that is internal to package time, but is actually provided by the Go runtime. In other words, there is no code for the function in package time itself.

Let’s take a closer look at the Linux implementations for the 386 and amd64 platforms. We see that these functions are implemented in assembler and call a function to retrieve the current time. You might have been expecting to see a system call, i.e. an INT 0x80 instruction on 386 or the SYSCALL instruction on amd64, to the kernel at this point, but Go does something much more interesting on Linux.

The Linux kernel provides Virtual Dynamically linked Shared Objects (vDSO) as a way for user space applications to make low-overhead calls to functions that would normally involve a system call to the kernel.

If you’re writing your application in a language that uses glibc, you are probably already getting your time via vDSO. Go doesn’t use glibc, so it has to implement this functionality in its runtime. The relevant code is in vdso_linux_amd64.c in the runtime package.

Finally, if you’re the kind of person that likes to stare into the bowels of your operating system, here’s the kernel side of the vDSO.

vDSO support for time functions is currently 64-bit only, but a kernel patch is in the works to add them on 32-bit platforms. When this happens, the Go runtime code will have to be updated to take advantage of this.

When frequently calling a function to determine the time, you may be interested to know how long it takes to return. In other words, how now is “now” really? For benchmarking purposes, you can make these system calls directly from Go code. We have prepared a patch to Dave Cheney’s excellent autobench project so that you may benchmark these system calls and other time-related functions yourself.

Benchmarks can also help us measure the time saved by calling gettimeofday and clock_gettime via the vDSO mechanism instead of the traditional system call path.

We’ll also use autobench to compare the performance of different versions of Go for the same set of time functions.

All benchmarks numbers below were obtained on an Intel Core i7-3540M CPU running at its maximum clock speed of 3 GHz. The CPU frequency scaling governor was set to performance mode to ensure reliable benchmark results.

We’ll use the Go 1.2 stable release as a baseline.

BenchmarkSyscallTime and BenchmarkVDSOTime measure the time it takes to make a time system call and vDSO call, respectively:

BenchmarkSyscallTime 38.2 ns/op
BenchmarkVDSOTime    3.85 ns/op

BenchmarkSyscallGettimeofday and BenchmarkVDSOGettimeofday measure the time it takes make to call a gettimeofday system call and vDSO call, respectively:

BenchmarkSyscallGettimeofday 59.3 ns/op
BenchmarkVDSOGettimeofday    23.4 ns/op

BenchmarkTimeNow measures the time it takes to call time.Now, which makes an underlying vDSO call to clock_gettime(CLOCK_REALTIME) and converts the returned value to a time.Time struct:

BenchmarkTimeNow 23.6 ns/op

Using autobench, we can also compare different Go versions against each other. To see how far we've come in the last few years, we also compared Go 1.2 to Go 1.0.3, which was released on Sep 27, 2012. The major difference was in the benchmark for time.Now:

benchmark        old ns/op  new ns/op  delta
BenchmarkTimeNow 406        23         -94.19%

To repeat this test with Go 1.0.3, you'll need the fix in changeset 419dcca62a3d to compile if you are using a recent version of GCC.

The speed at which you can tell time also depending on the clock source being used by your kernel. To see which clock sources you have available, run the following command:

$ cat /sys/devices/system/clocksource/clocksource0/available_clocksource
tsc hpet acpi_pm

To see which clock source is currently in use, run this command:

$ cat /sys/devices/system/clocksource/clocksource0/current_clocksource
tsc

Time Stamp Counter (TSC) is a 64-bit register that can be read by the RDTSC instruction. It is faster to read than the High Precision Event Timer (HPET). The ACPI Power Management Timer (APCI PMT) is another timer that is found on many motherboards.

We can also use the same benchmarks above to compare the TSC clock source and a HPET clock source. Doing so requires booting Linux with the clocksource=hpet kernel command line parameter. Here are the results:

benchmark                              tsc ns/op   hpet ns/op  delta
BenchmarkSyscallGettimeofday              59          645      +987.69%
BenchmarkVDSOGettimeofday                 23          598      +2455.56%
BenchmarkSyscallClockGettimeRealtime      58          642      +995.56%
BenchmarkSyscallClockGettimeMonotonic     57          641      +1012.85%
BenchmarkTimeNow                          23          598      +2433.90%

As you can see, querying the HPET clock source takes significantly longer.

Not all CPUs are created equal. To see which TSC features your CPU supports, run the following command:

$ cat /proc/cpuinfo | grep tsc

You will see some or all of the following CPU flags related to the TSC:

• tsc

• rdtscp

• constant_tsc

• nonstop_tsc

The tsc flag indicates that your CPU has the 64-bit TSC register, which has been present since the Pentium.

The rdtscp flag indicates that your CPU supports the newer RDTSCP instruction, in addition to the RDTSC instruction. Intel has an interesting whitepaper on How to Benchmark Code Execution Times on Intel IA-32 and IA-64 Instruction Set Architectures with more details about the differences.

The constant_tsc flag indicates that the TSC runs at constant frequency irrespective of the current frequency, voltage or throttling state of the CPU, commonly referred to as its P- and T-state.

The nonstop_tsc flag indicates that TSC does not stop, irrespective of the CPU’s power saving mode, referred to as its C-state.

These features work in conjunction to provide an invariant TSC. There is more discussion over at the Intel Software forums if you’re interested.

There has also been some interesting work on the time front in the upcoming Go 1.3 release. The time.Sleep function, Ticker, and Timer now use clock_gettime(CLOCK_MONOTONIC) on Linux and other platforms. This work has been a good example of the broader community contributing to improve the core of Go, as can be seen in issue 6007, CL 53010043 and discussion on golang-dev.

That’s it for today. If kernels and clocks excite you, CloudFlare is always looking to hire great Go and Linux kernel engineers. See our Careers page.

It would not have been possible to write this article without the help from some other sources. If you want to know more, here’s some recommended reading:

The book Understanding the Linux Kernel by Daniel Bovet and Marco Cesati has an entire chapter on the timekeeping architecture in the kernel.

More about timers on StackOverflow

Read more about vDSOs.

Read more about clock_gettime.

Sometimes bug reports can be a great source of information on details of the kernel and your CPU. Red Hat Bugzilla #474091 was a gold mine of information on the CPU flags for the TSC.

More at Quora on the the benefits of TSC vs HPET.

StackOverflow also had some information on the ACPI PM clock source.

Finally, some discussion of the differences between CLOCK_MONOTONIC and CLOCK_MONOTONIC_RAW as used in the Twisted framework for Python.

Back in June 2012 CloudFlare started a beta rollout of Google's then new SPDY protocol and we took a detailed look at how SPDY makes web sites faster.

Since then we've watched carefully as SPDY has evolved through different versions and have been keeping an eye on a new Google-driven protocol called QUIC. In August 2012 we rolled out SPDY for all our customers by making it a simple (one click) configuration option.

As SPDY has progressed we've become more and more confident in the protocol and made it automatic for all our Pro, Business and Enterprise customers. No click needed. (Since SPDY runs over SSL/TLS to use it a web site must have an SSL certificate).

Last week we rolled out the most recent version of SPDY, 3.1, for all customers. SPDY/3.1 is supported by Google Chrome and Mozilla Firefox. As older versions of SPDY (particularly SPDY/2) are being deprecated it's vital for us to keep up to date.

To see whether a site is served over SPDY it's possible to use the CloudFlare Claire extension for Google Chrome. Here it's showing that the popular Hacker News site is served over SPDY.

To discover the exact version used you can dive into Chrome's chrome://net-internals where the version is shown for each connection. Here connections to Google, CloudFlare and Hacker News are all using the latest version.

A key advantage of SPDY is its ability to multiplex many HTTP request streams onto a single TCP connection. In the past, various hacks (such as domain sharding) have been used to get around the fact that only sequential, synchronous requests were possible with HTTP over TCP. SPDY changed all that.

SPDY/3 introduced flow control so that SPDY clients (and servers) could control the amount of data they receive on a SPDY connection. SPDY/3.1 extended flow control to individual SPDY streams (each SPDY connection handles multiple simultaneous streams of data). Flow control is important because different clients (think of the differences in available memory in laptops, desktops and mobile phones) will have varying limitations on how much data they can receive at any one time.

Part of the service CloudFlare provides is being on top of the latest advances in Internet and web technologies. We've stayed on top of SPDY and will continue to roll out updates as the protocol evolves (and we'll support HTTP/2 just as soon as it is practical).

The most recent web protocol we've started experimenting with is QUIC. QUIC is radically different from HTTP or SPDY because it uses UDP as its underlying transport. Currently, we have an internal QUIC server running serving a static version of the CloudFlare home page. As our experiments with QUIC progress we'll make an alpha site available for people who, like us, are interested in experimenting with bleeding edge technologies.

If you're interested in playing with QUIC today you'll need to build the test QUIC server and client that are part of the Chromium project, get Google Chrome Canary and enable QUIC in chrome://flags. You'll probably also want the latest Wireshark (the development release) which is capable of decoding QUIC frames.

And once QUIC makes the move from experimental to beta we'll be sure to make it available for our customers.

It’s common knowledge that domain sharding, where the resources in a web page are shared across different domains (or subdomains), is a good thing. It’s a good thing because browsers limit the number of connections per domain: splitting a web page across domains means more connections and hence faster page downloads. Overall domain sharding results in a better end-user experience, and can be a useful way of sharing load across web servers.

But with the adoption of Google’s SPDY protocol the domain sharding situation is totally different. In fact, domain sharding can hurt performance when SPDY is in use and isn’t recommended. To understand why, here’s the popular 4chan.org web site downloaded without SPDY but using SSL (it’s possible to do this comparison without SSL, but less interesting because the timings are very different).

You can see that there are three domains involved: www.4chan.org (from which the initial HTML is downloaded), s.4cdn.org and t.4cdn.org. 4chan is using two domains to shard resources like JavaScript, CSS and images. After the initial HTML is downloaded on line 1, the browser (I used IE 10 here) looks up the DNS entry for s.4cdn.org and t.4cdn.org and opens three connections to each (lines 2 to 7).

In the diagram above the orange represents the TCP connection, and the purple the SSL negotiation. After using those 6 connections to download a resource, the same connections are reused (classic HTTP/1.1 Keep-Alive behaviour) to get further resources. Finally, line 16, there’s a separate connection to send Google Analytics information.

Now take a look at the same site downloaded using SPDY/2 via Google Chrome.

Line 1 shows the same sort of connection, SSL negotiation and download of the page (here it took 591ms to complete vs. 549 ms above). But then the behaviour is totally different. Line 2 shows a single TCP connection and single SSL negotiation to s.4cdn.org. That connection is then used to download all the resources for s.4cdn.org and t.4cdn.org in parallel. Finally, there’s the same, separate Google Analytics connection. What you’re seeing there is SPDY in action.

The SPDY version was slightly faster: the page was visually complete at 1.1s; with SSL without SPDY it was 1.3s (although you’d have to account for differences in paint time between IE 10 and Chrome to really understand those values). There are two important things happening in the SPDY version:

Firstly, Chrome has noticed that s.4cdn.org and t.4cdn.org are the same site (they have the same IP addresses and the certificate for s.4cdn.org is valid for t.4cdn.org as well: it’s a wildcard certificate for 4cdn.org) and so it doesn’t bother with separate SSL connection: one will do. It then requests resources from each of those domains across the same SPDY connection. To do that it simply specifies the correct Host in the SPDY request. These can be see in the chrome://net-internals view. Here are two requests on the same SPDY connection for different Hosts.

t=1386958552557 [st= 1] SPDY_SESSION_SYN_STREAM --> fin = true --> accept: */* accept-encoding: gzip,deflate,sdch accept-language: en-US,en;q=0.8,fr;q=0.6 cache-control: no-cache host: s.4cdn.org method: GET pragma: no-cache referer: https://www.4chan.org/ scheme: https url: /js/fp-combined-compiled.7.js user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36 version: HTTP/1.1 --> stream_id = 5 --> unidirectional = false t=1386958552577 [st= 21] SPDY_SESSION_SYN_STREAM --> fin = true --> accept: image/webp,*/*;q=0.8 accept-encoding: gzip,deflate,sdch accept-language: en-US,en;q=0.8,fr;q=0.6 cache-control: no-cache host: t.4cdn.org method: GET pragma: no-cache referer: https://www.4chan.org/ scheme: https url: /cgl/thumb/1386957763148s.jpg user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36 version: HTTP/1.1 --> stream_id = 7 --> unidirectional = false

So, Chrome has detected that these domains are actually the same machine and made a single connection. That’s great, and doesn’t have a performance impact, but the actual decision to shard at all has had an impact.

Secondly, notice how in the SPDY case there are two TLS negotiations (one for www.4chan.org and one for s.4cdn.org). The site could have loaded much faster if all the resources had been on www.4chan.org (or on domains that shared a certificate; for this reason wildcard certificates work well with SPDY connections because the browser can use a single shared connection) because the entire download could have been done in a SPDY connection. Because 4chan uses a special domain (on a different IP with a different certificate) for resources it’s necessary to set up a new connection. In the example above, all the resources have to wait for a DNS lookup (27 ms), TCP connection (29 ms) and SSL negotiation (71 ms) before the SPDY connection can start requesting them. That’s a total of 127 ms. The page was visually complete in 1100 ms; if a single domain had been used then SPDY would have saved another 127 ms (almost 12% of the time).

So, for SPDY it’s actually better to not shard; for non-SPDY domain sharding remains a useful technique. (If you are interested in the actual test data the IE 10/SSL test is http://www.webpagetest.org/result/131213_FK_QC1/ and the Chrome/SPDY test is http://www.webpagetest.org/result/131213_NE_QCQ/).

The question then becomes, can you have the best of both worlds? With a little DNS trickery it’s possible to set up a site that works well whether SPDY is available or not. Since I don’t have access to 4chan to do live experiments, I copied the www.4chan.org home page and all the included resources to my own web server and set up three domains: r.jgc.org (the root domain), s.jgc.org (equivalent to s.4cdn.org) and t.jgc.org (equivalent to t.4cdn.org). I then manually edited the HTML and CSS so that all the linked resources pointed to either s.jgc.org and t.jgc.org in the same manner as the original 4chan site. But, critically, I used a single certificate for all three domains. Here’s the site being loaded using IE 10 over SSL.

And here’s the site loaded using Chrome with SPDY.

As you can see, the domain sharding worked in IE 10. There are multiple connections to the s.jgc.org and t.jgc.org domains downloading resources in parallel. And the same configuration worked for Chrome with SPDY because it detected that these shared a certificate and used a single SPDY connection for everything (including the initial page download).

In Chrome there was only a single TCP connection and a single DNS lookup needed despite the presence of three domains. The IE 10/SSL version was visually complete in 1100 ms and used 9 TCP/SSL connections (plus one extra for Google Analytics). The Chrome/SPDY version was visually complete 200ms (at 900ms) and used… a single SPDY connection (plus an extra connection for Google Analytics). If you’re interested the IE 10/SSL test is here and the Chrome/SPDY test is here.

For the best performance for the older browsers and the latest, shiny SPDY browsers domain sharding should still be used but using a certificate that covers the domains used means that only a single SPDY connection will be needed.

CloudFlare Makes This EasyCloudFlare has both simple SSL options and push button SPDY available. By setting up SSL on CloudFlare with subdomains you'll automatically get SPDY as well. In the test above it took me about 10 minutes to set up the test subdomains on jgc.org and enable both SSL and SPDY.

Thanks to Andrew Galloni for his assistance reviewing and investigating the interaction between SPDY and domain sharding.

If you're not familiar with the cryptographic protocols involved you might be wondering why one part uses a 128-bit key and another a 2,048-bit key. And you'd be forgiven for wondering why a large key wasn't used throughout and whether a 128-bit key is weaker than a 2,048-bit key. This blog post will explain why a 128-bit symmetric key is, in fact, a bit more secure than a 2,048-bit asymmetric key; you have to look at both the type of encryption being used (symmetric or asymmetric) and the key length to understand the strength of the encryption.

My connection above used a symmetric cipher (RC4_128) with a 128-bitkey and an asymmetric cipher (ECDHE_RSA) with a 2,048-bit key.

You might also have seen other key lengths in use. For example, when I connect to the British Government portal gov.uk I get a TLS connection that uses AES_256_CBC (with a 256-bit key) set up using RSA with a 2,048-bit key. It's not uncommon to see RSA with a 1,024-bit key as well.

To understand these key lengths it's necessary to understand a little about the actual encryption schemes they are used with.

The RC4_128 and AES_256_CBC schemes mentioned above are symmetric cryptographic schemes. Symmetric simply means that the same key is used to encipher and decipher the encrypted web traffic. In one case, a 128-bit key is used, in another a 256-bit key.

Symmetric cryptography is the oldest form there is. When children use a Caesar Cipher (shifting each letter in the alphabet some fixed number of places) they are performing symmetric cryptography. In that case, the key is the number of places to shift letters and there are 26 possible keys (which is roughly like saying the Caesar Cipher has a roughly 5-bit key).

Here's a Caesar Shift with a key of 7 (each letter is moved up in the alphabet 7 places):

ISTHE BESTT HATCO RPORA LBOBB YSHAF TOECA NDOON SHORT NOTIC E PZAOL ILZAA OHAJV YWVYH SIVII FZOHM AVLJH UKVVU ZOVYA UVAPJ L

There are lots of ways to break the Caesar Cipher, but one way is to try out all 26 possible keys. That's really not that hard since there are only 26 possible solutions:

PZAOL ILZAA OHAJV YWVYH SIVII FZOHM AVLJH UKVVU ZOVYA UVAPJ L
----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -

0: PZAOL ILZAA OHAJV YWVYH SIVII FZOHM AVLJH UKVVU ZOVYA UVAPJ L 1: OYZNK HKYZZ NGZIU XVUXG RHUHH EYNGL ZUKIG TJUUT YNUXZ TUZOI K 2: NXYMJ GJXYY MFYHT WUTWF QGTGG DXMFK YTJHF SITTS XMTWY STYNH J 3: MWXLI FIWXX LEXGS VTSVE PFSFF CWLEJ XSIGE RHSSR WLSVX RSXMG I 4: LVWKH EHVWW KDWFR USRUD OEREE BVKDI WRHFD QGRRQ VKRUW QRWLF H 5: KUVJG DGUVV JCVEQ TRQTC NDQDD AUJCH VQGEC PFQQP UJQTV PQVKE G 6: JTUIF CFTUU IBUDP SQPSB MCPCC ZTIBG UPFDB OEPPO TIPSU OPUJD F 7: ISTHE BESTT HATCO RPORA LBOBB YSHAF TOECA NDOON SHORT NOTIC E 8: HRSGD ADRSS GZSBN QONQZ KANAA XRGZE SNDBZ MCNNM RGNQS MNSHB D 9: GQRFC ZCQRR FYRAM PNMPY JZMZZ WQFYD RMCAY LBMML QFMPR LMRGA C 10: FPQEB YBPQQ EXQZL OMLOX IYLYY VPEXC QLBZX KALLK PELOQ KLQFZ B 11: EOPDA XAOPP DWPYK NLKNW HXKXX UODWB PKAYW JZKKJ ODKNP JKPEY A 12: DNOCZ WZNOO CVOXJ MKJMV GWJWW TNCVA OJZXV IYJJI NCJMO IJODX Z 13: CMNBY VYMNN BUNWI LJILU FVIVV SMBUZ NIYWU HXIIH MBILN HINCW Y 14: BLMAX UXLMM ATMVH KIHKT EUHUU RLATY MHXVT GWHHG LAHKM GHMBV X 15: AKLZW TWKLL ZSLUG JHGJS DTGTT QKZSX LGWUS FVGGF KZGJL FGLAU W 16: ZJKYV SVJKK YRKTF IGFIR CSFSS PJYRW KFVTR EUFFE JYFIK EFKZT V 17: YIJXU RUIJJ XQJSE HFEHQ BRERR OIXQV JEUSQ DTEED IXEHJ DEJYS U 18: XHIWT QTHII WPIRD GEDGP AQDQQ NHWPU IDTRP CSDDC HWDGI CDIXR T 19: WGHVS PSGHH VOHQC FDCFO ZPCPP MGVOT HCSQO BRCCB GVCFH BCHWQ S 20: VFGUR ORFGG UNGPB ECBEN YOBOO LFUNS GBRPN AQBBA FUBEG ABGVP R 21: UEFTQ NQEFF TMFOA DBADM XNANN KETMR FAQOM ZPAAZ ETADF ZAFUO Q 22: TDESP MPDEE SLENZ CAZCL WMZMM JDSLQ EZPNL YOZZY DSZCE YZETN P 23: SCDRO LOCDD RKDMY BZYBK VLYLL ICRKP DYOMK XNYYX CRYBD XYDSM O 24: RBCQN KNBCC QJCLX AYXAJ UKXKK HBQJO CXNLJ WMXXW BQXAC WXCRL N 25: QABPM JMABB PIBKW ZXWZI TJWJJ GAPIN BWMKI VLWWV APWZB VWBQK M

The goal of modern symmetric cryptography is to make this sort of'trying out all the possible keys' the only approach to breaking a symmetric cipher. Algorithms like RC4 and AES scramble data based on a key. The key itself is, ideally, randomly chosen from the set of all possible keys.

On a side note, there are now serious problems with RC4 and as better replacements come along (such as ciphersuites based on AES) CloudFlare will update the ciphersuites it uses to provide the best level of protection.

That said the basic idea is that the only way to break into a connection secured with a symmetric cipher is to try out all the keys. Which brings us to 128-bit and 256-bit keys.

A 128-bit key means there are 340,282,366,920,938,463,463,374,607,431,768,211,456 possible keys to try. A 256-bit key has the square of that many keys to try: a hugenumber.

To put that in context imagine trying to test all the keys for a 128-bit AES encryption using the special AES instructions added to the latest Intel microprocessors. These instructions are designed to be very fast and according to Intel's own data decrypting a block of AES encrypted data would take 5.6 cycles on Intel i7 Processor with 4 cores.

Put another way, that processor could try out one key on one block of data in about 1.7 nanoseconds. At that speed it would take it about 1.3 * 10^12 * the age of the universe to check all the keys (you'd probably only have to check half before finding the right one so divide that incredibly long time by two).

Since personal computers became available roughly 1 billion of them have been sold. Imagine that they all had the same top-of-the-line processor and were used to attack a 128-bit key. You'd manage to get the time down to 660 * the age of the universe.

Image (c) BBC TV

So, breaking 128-bit keys by brute force just isn't practical. And breaking 256-bit is even less possible. So, for symmetric ciphers, keys of these lengths make sense.

But that's not the case for asymmetric cryptography.

Asymmetric cryptography works by having two different keys, one for encryption and one for decryption. It's also often called 'public key cryptography' because it's possible to make one key public (allowing someone to encrypt a message) while keeping the other private (only the holder of the private key can decrypt the message encrypted with its related public key).

In order to have these special properties the public and private keys are related by some mathematical process. Aside: in the symmetric examples there's only one key and it's just any value of the right number of bits. This randomness of a symmetric key means it can be relatively short as we saw.

For example, in the popular RSA scheme used with SSL/TLS the public and private keys consist in part of the product of two large prime numbers. Making an RSA key starts with picking two random prime numbers. The security of RSA relies (in part) on the fact that it's easy to choose two random prime numbers, but it's very hard to discover what they are when just given the product of them.

Suppose there are two prime numbers picked at random called p0 and p1. Part of the RSA public (and private) key is called the modulus and it is just p0*p1. If an attacker can decompose (or factor) the modulus into p0 and p1 they can break RSA because they can work out the private key. Mathematicians believe that it is very hard to factor a product of two primes and the security of web transactions relies, in part, on that belief.

Typical RSA key sizes are 1,024 or 2,048 or 4,096 bits. That number is the number of bits in the modulus. For each there will be a pair of primes of roughly 512 bits or 1,024 bits or 2,048 bits depending on the key size picked. Those primes are chosen by some random process (highlighting once again the importance of random number generators).

But we still haven't answered the question of why these key sizes are so large. Just as in the symmetric key case, attacks on say 2,048-bit RSA are based on trying out all keys of a certain size, but unlike the symmetric key scheme not every 2,048-bit number is an RSA key (because it has to be the product of two primes).

So, although the key size is larger there are actually fewer possible RSA keys for any given number of bits that there are for the same symmetric key size. That's because there are only so many prime numbers of that size and below. The RSA scheme can only use pairs of prime numbers, whereas the symmetric schemes can use any number at all of the same size.

This diagram (called an Ulam Spiral) shows the numbers from 1 to 40,000 as black or white dots. The black dots are the primes

Image from Wikipedia

If you used a 256-bit RSA key (roughly consisting of two 128-bit prime numbers multiplied together) you'd quickly find that your encryption had been broken by someone using a fast home PC. There are only so many 128-bit prime numbers and there are fast ways of attacking the factorization problem (such as the General Number Field Sieve that actually make breaking RSA keys a little easier than trying out every single key).

Any time there's a pattern in a cryptographic key it represents a chink in the crytography's armor. For example, in a perfect world, people would pick completely random passwords. Because they don't there are patterns in the passwords and they can be guessed or broken without trying out every possible password.

RSA keys have a distinctive pattern: they are the product of two prime numbers. That provides the chink; today that chink is best exploited by the General Number Field Sieve. In the symmetric key case there are no such patterns: the keys are just large randomly-chosen numbers. (Of course, if you don't pick your symmetric key randomly you might actually be helping an attacker find a way to break your encrypted messages.)

A few years ago the 512-bit RSA key used to sign software for TIcalculators was broken by an individual using a PC that ran for 73 days using the open source msieve and ggnfs prorgrams.

So, asymmetric keys have to be much larger than symmetric keys because there are less of them for a given number of bits, and because there are patterns within the keys themselves.

The ECRYPT II recommendations on key length say that a 128-bit symmetric key provides the same strength of protection as a 3,248-bit asymmetric key. And that those key lengths provide long term protection of data encrypted with them.

The length of time a key is good for is also important. Over time computers get faster and techniques for breaking encryption schemes (particuarly techniques for breaking asymmetric encryption) get better. That 512-bit key used for TI calculators probably looked pretty safe when it was first chosen. And in 1999 a key of that length took a supercomputer to break.

We keep an eye on these reports when choosing ciphers and key lengths to secure our and our customers' communications.

Because of the importance of protecting our customers' communications CloudFlare has also opted to roll out forward secrecy for our SSL/TLS connections. That means that the public/private keys used for connections are generated freshly each time. That prevents bulk attacks where a single public/private key (such as the one used in simple RSA-based certificates) is broken revealing all of the symmetric keys used to secure HTTPS for a web site.

Returning to the HTTPS connection I made to CloudFlare at the start, the key negotiation was done using ECDHE_RSA. That's the ephemeral version of the Diffie-Hellman key exchange mechanism that uses elliptic curves and RSA for authentication. That's quite amouthful. It breaks down like this:

1. The public/private key pair used to this connection was ephemeral: it was only created for this connection. That's what gives 'forward secrecy'.

2. The actual public-key encryption scheme used was Elliptic Curve Diffie-Hellman. Elliptic Curve Cryptography uses a different branch of mathematics than RSA. Looking at the ECRYPT II report shows that a 128-bit symmetric key is as strong as a 3,248-bit asymmetric key; to get the equivalent strength from an Elliptic Curve Cryptographic scheme requires a key with 256-bits.

3. So, Google Chrome set up an ephemeral 256-bit Elliptic Curve Diffie Hellman public/private key pair and used it to agree on a 128-bit symmetric key for the rest of the communication.

4. To prove that the web site really was www.cloudflare.com the 2,048-bit RSA key was used along with the web site's certificate.

So, three different key lengths were used: 128-bit (with RC4), 256-bit (with ECDHE) and 2,048-bit (with RSA). All three key lengths provide similar levels of security.

At around 6:24pm PST / 02:24 UTC (5 Nov. 2012 PST / 6 Nov. 2012 UTC), CloudFlare employees noticed that Google's services were offline. We use Google Apps for things like email so when we can't reach their servers the office notices quickly. I'm on the Network Engineering team so I jumped online to figure out if the problem was local to us or global.

I quickly realised that we were unable to resolve all of Googles services — or even reach 8.8.8.8, Googles public DNS server — so I started troubleshooting DNS.

$ dig +trace google.com

Here's the response I got when I tried to reach any of Google.com's name servers:

google.com. 172800 IN NS ns2.google.com.google.com. 172800 IN NS ns1.google.com.google.com. 172800 IN NS ns3.google.com.google.com. 172800 IN NS ns4.google.com.;; Received 164 bytes from 192.12.94.30#53(e.gtld-servers.net) in 152 ms;; connection timed out; no servers could be reached

The fact that no servers could be reached means something was wrong. Specifically, it meant that from our office network we were unable to reach any of Googles DNS servers.

I started to look at the network layer, see if that's where the problems lay.

PING 216.239.32.10 (216.239.32.10): 56 data bytesRequest timeout for icmp_seq 092 bytes from 1-1-15.edge2-eqx-sin.moratelindo.co.id (202.43.176.217): Time to live exceeded

That was curious. Normally, we shouldn't be seeing an Indonesian ISP (Moratel) in the path to Google. I jumped on one of CloudFlare's routers to check what was going on. Meanwhile, others reports from around the globe on Twitter suggested we weren't the only ones seeing the problem.

To understand what went wrong you need to understand a bit about how networking on the Internet works. The Internet is a collection of networks, known as "Autonomous Systems" (AS). Each network has a unique number to identify it known as AS number. CloudFlare's AS number is 13335, Google's is 15169. The networks are connected together by what is known as Border Gateway Protocol (BGP). BGP is the glue of the Internet — announcing what IP addresses belong to each network and establishing the routes from one AS to another. An Internet "route" is exactly what it sounds like: a path from the IP address on one AS to an IP address onanother AS.

BGP is largely a trust-based system. Networks trust each other to say which IP addresses and other networks are behind them. When you send a packet or make a request across the network, your ISP connects to its upstream providers or peers and finds the shortest path from your ISP to the destination network.

Unfortunately, if a network starts to send out an announcement of a particular IP address or network behind it, when in fact it is not, if that network is trusted by its upstreams and peers then packets can end up misrouted. That is what was happening here.

I looked at the BGP Routes for a Google IP Address. The route traversed Moratel (23947), an Indonesian ISP. Given that I'm looking at the routing from California and Google is operating Data Centre's not far from our office, packets should never be routed via Indonesia. The most likely cause was that Moratel was announcing a network that wasn't actually behind them.

The BGP Route I saw at the time was:

tom@edge01.sfo01> show route 216.239.34.10 inet.0: 422168 destinations, 422168 routes (422154 active, 0 holddown, 14 hidden)+ = Active Route, - = Last Active, * = Both216.239.34.0/24 *[BGP/170] 00:15:47, MED 18, localpref 100 AS path: 4436 3491 23947 15169 I > to 69.22.153.1 via ge-1/0/9.0

Looking at other routes, for example to Google's Public DNS, it was also stuck routing down the same (incorrect) path:

tom@edge01.sfo01> show route 8.8.8.8 inet.0: 422196 destinations, 422196 routes (422182 active, 0 holddown, 14 hidden)+ = Active Route, - = Last Active, * = Both8.8.8.0/24 *[BGP/170] 00:27:02, MED 18, localpref 100 AS path: 4436 3491 23947 15169 I > to 69.22.153.1 via ge-1/0/9.0

(Image Credit: The Simpsons)

Situations like this are referred to in the industry as "route leakage", as the route has "leaked" past normal paths. This isn't an unprecedented event. Google previously suffered a similar outage when Pakistan was allegedly trying to censor a video on YouTube and the National ISP of Pakistan null routed the service's IP addresses. Unfortunately, they leaked the null route externally. Pakistan Telecom's upstream provider, PCCW, trusted what Pakistan Telecom's was sending them and the routes spread across the Internet. The effect was YouTube was knocked offline for around 2 hours.

The case today was similar. Someone at Moratel likely "fat fingered" an Internet route. PCCW, who was Moratel's upstream provider, trusted the routes Moratel was sending to them. And, quickly, the bad routes spread. It is unlikely this was malicious, but rather a misconfiguaration or an error evidencing some of the failings in the BGP Trust model.

The solution was to get Moratel to stop announcing the routes they shouldn't be. A large part of being a network engineer, especially working at a large network like CloudFlare's, is having relationships with other network engineers around the world. When I figured out the problem, I contacted a colleague at Moratel to let him know what was going on. He was able to fix the problem at around 2:50 UTC / 6:50pm PST. Around 3 minutes later, routing returned to normal and Google's services came back online.

Looking at peering maps, I'd estimate the outage impacted around 3–5% of the Internet's population. The heaviest impact will have been felt in Hong Kong, where PCCW is the incumbent provider. If you were in the area and unable to reach Google's services around that time, now you know why.

This all is a reminder about how the Internet is a system built on trust. Today's incident shows that, even if you're as big as Google, factors outside of your direct control can impact the ability of your customers to get to your site so it's important to have a network engineering team that is watching routes and managing your connectivity around the clock. CloudFlare works every day to ensure our customers get the optimal possible routes. We look out for all the websites on our network to ensure that their traffic is always delivered as fast as possible. Just another day in our ongoing efforts to #savetheweb.

Moratel says the issue was caused by an unexpected hardware failure, causing this abnormal condition. This was not a malicious attempt. Moratel immediately shutdown the BGP peering with Google after contact was made while the hardware failure was being looked into.

Thanks for reading all the way to the end. If you enjoyed this post, take a second to learn more about CloudFlare or nominate us for the 2012 Crunchie Award for Best TechnicalInnovation.