Today Cloudflare opened the door on our beta deployment of QUIC with the announcement of our test site: cloudflare-quic.com. It supports the latest draft of the IETF Working Group’s draft standard for QUIC, which at this time is at: draft 14.
The Cloudflare Systems Engineering Team has a long history of investing time and effort to trial new technologies, often before these technologies are standardised or adopted elsewhere. We deployed early experiments in standards such as: HTTP/2,TLS1.3, DNSSEC, DNS over HTTP, Encrypted SNI, when they were still in incubation. We committed to these technologies in their very early stages because we believed that they made for a safer, faster, better internet. And now we’re excited to do the same with QUIC.
In this blog post, we will show you how you can unlock the cloudflare-quic.com achievement and be some of the first people in the world to perform a HTTP transaction over the global internet using QUIC. This will be a moment that you can tell your grandkids about - if they can stop laughing at your stories of cars with wheels and use of antiquated words like: “meme” and “phone”.
But before we begin, let’s take a little bit of time to review what QUIC is. Our previous blog post The Road to QUIC by my colleague Alessandro Ghedini, gives an excellent introduction to QUIC; its goals, its challenges, and many of the technical advantages that will come with it. It is good background reading for this article and a great introduction to the topic of QUIC in general.
If you visit cloudflare-quic.com with your regular web browser, you will be presented with an informative landing page. However, what you see will not be delivered using QUIC, because at the time this blog is posted, your browser doesn’t support IETF QUIC. No graphical browser does.
Some may point out that Google Chrome has had support for QUIC for many years, but we must re-iterate that the protocol supported by Chrome is Google’s own UDP based transport layer protocol. That protocol was once called QUIC but has forfeited that label and now goes by the name gQUIC, and what’s more, the mechanics of gQUIC are now significantly different to IETF QUIC.
Getting QUIC
The only way to access cloudflare-quic.com using the QUIC protocol is to use a command line client from one of the various implementations of QUIC that are actively evolving alongside the IETF standard. Most of these implementations can be found here. If you are familiar with any of these, you are welcome to try them against cloudflare-quic.com however please note that your client of choice must support draft 14 of the IETF QUIC standard.
Our preferred QUIC client, and the one whose output we will be analysing in this blog, comes as part of the ngtcp2 project. The original project is located here: github.com/ngtcp2/ngtcp2, but we are hosting our own copy here: github.com/cloudflare/ngtcp2/tree/quic-draft-14 so that we may be sure you get the exact resources you need for this demonstration.
Before proceeding please be aware that the following instructions will require you to build software from source code. ngtcp2 and its dependencies are buildable on multiple Operating System platforms, however, the processes described below are more likely to succeed on Linux. To start with, you will need:
A POSIX-flavoured operating system, for example: Ubuntu Linux
To install core software development tools: gcc or clang, libc development packages, make, autoconf, automake, autotools-dev, libtool, pkg-config, git
To install some additional software dependencies: C Unit tester: (cunit >=2.1), libev development packages. Check the homepage of Cloudflare ngtcp2 copy if you are unsure.
Once you are confident with your setup, run the following commands to retrieve and build the ngtcp2 client and its major dependency OpenSSL:
$ git clone --depth 1 -b quic-draft-14 https://github.com/tatsuhiro-t/openssl
$ cd openssl
$ ./config enable-tls1_3 --prefix=$PWD/build
$ make
$ make install_sw
$ cd ..
$ git clone -b quic-draft-14 https://github.com/cloudflare/ngtcp2
$ cd ngtcp2
$ autoreconf -i
$ ./configure PKG_CONFIG_PATH=$PWD/../openssl/build/lib/pkgconfig LDFLAGS="-Wl,-rpath,$PWD/../openssl/build/lib"
$ make checkTesting QUIC
If you are still with me, congratulations! The next step is to pre-fabricate a HTTP/1.1 request that we can pass to our QUIC client, in order to avoid typing it out repeatedly. From your ngtcp2 directory, invoke the command:
$ echo -ne "GET / HTTP/1.1\r\nHost: cloudflare-quic.com\r\n\r\n" > cloudflare-quic.reqOne of the promises of QUIC is the new QUIC HTTP protocol, which is another IETF standard being developed in conjunction with the QUIC transport layer. It is a re-engineering of the HTTP/2 protocol to allow it to benefit from the many advantages of QUIC.
The design of QUIC HTTP is in a high state of flux at this time and is an elusive target for software implementors, but it is clearly on the Cloudflare product roadmap. For now, cloudflare-quic.com will use HTTP/1.1 for utility and simplicity.
Now it’s time to invoke the ngtcp2 command line client and establish your QUIC connection to cloudflare-quic.com:
$ examples/client cloudflare-quic.com 443 -d cloudflare-quic.reqTo be perfectly honest, the debugging output of the ngtcp2 client is not particularly pretty, but who cares! You are now a QUIC pioneer, riding the crest of a new technological wave! Your reward will be the eye-rolls of the teenagers of 2050.
The HANDSHAKE
Let’s go over some of the ngtcp2 debugging output that you hopefully can see after invoking your HTTP request over QUIC, and at the same time, let’s relate this output back to some important features of the QUIC protocol.
Client HELLO
01 I00000000 0x07ff706cb107568ef7116f5f58a9ed9010 pkt tx pkt 0 dcid=0xba006470cf7c05009e219ff201e4adbef8a3 scid=0x07ff706cb107568ef7116f5f58a9ed9010 type=Initial(0x7f) len=0
02 I00000000 0x07ff706cb107568ef7116f5f58a9ed9010 frm tx 0 Initial(0x7f) CRYPTO(0x18) offset=0 len=309
03 I00000000 0x07ff706cb107568ef7116f5f58a9ed9010 frm tx 0 Initial(0x7f) PADDING(0x00) len=878
04 I00000000 0x07ff706cb107568ef7116f5f58a9ed9010 rcv loss_detection_timer=1537267827966896128 last_hs_tx_pkt_ts=1537267827766896128 timeout=200Above is what the QUIC protocol calls the client initial packet. It is the packet that is sent to establish a completely new connection between the client and the QUIC server.
The element: scid on line 01 is an example of a source connection ID. This is the unique number that the client chooses for itself when sending an initial packet. In the example output above, the value of the client scid is: 0x07ff706cb107568ef7116f5f58a9ed9010 but you will see a different value. In the ngtcp2 client utility, this number is purely random, as the QUIC connection will only last as long as the command runs, and therefore doesn’t need to carry much meaning. In future, more complex QUIC clients (such as web browsers) will likely choose their source connection IDs more carefully. Future QUIC servers will certainly do this, as connection IDs are a good place to encode information.
Encoding information in source connection ids is of particular interest to an organisation like Cloudflare, where a single IP address can represent thousands of physical servers. To support QUIC in an infrastructure like ours, routing of UDP QUIC packets will need to be done to a greater level of precision than can be represented in an IP address, and large, data packed connection IDs will be very useful for this. But enough about us, this blog is about you!
The element: dcid, also on line 01, is the destination connection ID. In the client initial phase, this is always random as per the QUIC specification, because the client wants to be sure that it is treated as ‘new’ by the QUIC server. A random dcid, particularly one that is the maximum allowed length of 144bits, combined with a large source connection id, has a sufficiently high statistical chance of being unique so as to not clash with a connection id that the QUIC server has already registered. Later we will see what the QUIC server does with the random destination connection ID that the client has chosen.
On line 02, we see that the client initial packet includes a CRYPTO frame that contains the TLS client hello message. This clearly demonstrates one of the significant advantages in the design of QUIC: the overlapping of transport layer connection establishment and TLS layer negotiation. Both of these processes necessitate some back and forth between a client and server for both TLS over TCP and for QUIC.
In TLS over TCP the two processes happen one after the other:
You can count a total of FOUR round trips between the client & the server before a HTTP request can be made! Now compare that with QUIC, where they happen at the same time:
That’s a 50% reduction! With just TWO round trips before you can start making HTTP requests.
Returning to the analysis of the ngtcp2 debug output, we can see the client initial packet adds a PADDING frame in order to bring the packet to a minimum size mandated by the QUIC specification. The reason for this is twofold:
Firstly, to ensure that the network between the QUIC client and server can support satisfactorily large UDP packets. Sadly UDP is a second class citizen on the wide internet, generally only being used to transmit single, small, unrelated packets. QUIC counters all three of these patterns, so the rationale here is: if it’s going to get stuck, better to find out early. The quality of network support for streams of UDP will hopefully evolve alongside QUIC.
Secondly, to reduce the effectiveness of amplification attacks. This type of attack is where bad actors take advantage of network services that produce server responses vastly greater in size than the barely-validated request that solicited them. By spoofing the address of a victim, a bad actor can bombard the victim with large volumes of server responses given a relatively small volume of requests to the server. By requiring that an initial request be large, QUIC helps to make the amplification value much lower. UDP based amplification attacks are a very real issue, and you can read Cloudflare's account of a such an attack here.
QUIC defines a number of other mechanisms to protect against amplification attacks as well as DDoS attacks and you will see some of these a bit later.
Server HELLO
Further down you will see the first packet returned from the server, which is the server initial packet:
01 I00000160 0x07ff706cb107568ef7116f5f58a9ed9010 pkt rx pkt 0 dcid=0x07ff706cb107568ef7116f5f58a9ed9010 scid=0x3afafde2c24248817832ffe545d874a2a01f type=Initial(0x7f) len=111
02 I00000160 0x07ff706cb107568ef7116f5f58a9ed9010 frm rx 0 Initial(0x7f) CRYPTO(0x18) offset=0 len=90
…
03 I00000314 0x07ff706cb107568ef7116f5f58a9ed9010 cry remote transport_parameters negotiated_version=0xff00000e
04 I00000314 0x07ff706cb107568ef7116f5f58a9ed9010 cry remote transport_parameters supported_version[0]=0xff00000e
…
05 I00000314 0x07ff706cb107568ef7116f5f58a9ed9010 frm tx 3 Initial(0x7f) ACK(0x0d) largest_ack=1 ack_delay=0(0) ack_block_count=0
06 I00000314 0x07ff706cb107568ef7116f5f58a9ed9010 frm tx 3 Initial(0x7f) ACK(0x0d) block=[1..0] block_count=1The response destination connection id (dcid on line 01) is the client’s original source connection ID (scid) which in this example is: 0x07ff706cb107568ef7116f5f58a9ed9010.
The server has now discarded the client’s randomly-chosen dcid after finding that the client is ‘new’, and replaced it with its own connection ID which you can see as the packet source connection ID scid on line 01, which in this example is: 0x3afafde2c24248817832ffe545d874a2a01.
Starting from this point, both the QUIC client and server recognise each other’s connection IDs, opening the door to a powerful feature of QUIC: connection migration. Connection migration will allow QUIC clients and servers to change their IP addresses and ports, but still maintain the QUIC connection. QUIC packets arriving from or to the new IP/port can continue to be handled because the connection ID, which has not changed, will act as the primary identifier of the connection context. For our first cloudflare-quic.com demonstration, connection migration is not supported, but we’ll be working on this as we develop our QUIC offerings.
The server initial packet contains the next part of the TLS handshake, found in the CRYPTO frame on line 01, which is the first part of the TLS server hello and may contain elements such as handshake key material and the beginning of the server’s certificate chain.
Lines 03 and 04 show the exchange of transport parameters, which are QUIC specific configuration values declared by one side to the other and used to control various aspects of the connection. These parameters are encoded and transmitted within the TLS handshake. This not only reiterates the close relationship between the TLS and transport layers in QUIC, but also demonstrates QUIC’s focus on security, as the exchange of these parameters will be protected against outside tampering as part of the TLS handshake.
Lines 05 and 06 show an example of some acknowledgement frames being sent from the client to the server. Acknowledgements are part of the QUIC loss detection mechanism that deals with data losses that inevitably happen on large networks, however during the handshake phase, acknowledgements also have another use: to hint at the validity of the client by proving to the server that a client is truly interested in communicating with the server and is not at a spoofed address.
Without any form of source validation, QUIC servers will severely limit the amount of data that they send to clients. This protects helpless, spoofed victims of amplification attacks (in conjunction with the client initial packet minimum size requirement described above), and also helps protect the QUIC server from the equivalent of a TCP SYN attack, by constraining the commitment that the QUIC server will make to an unvalidated client.
For Cloudflare, there are vast unknowns in regard to DDoS and SYN style attacks against QUIC and it is a subject we are supremely interested in. While these threat models remain unknown, our protections around cloudflare-quic.com will be effective but… remorseless.
The REQUEST
Once the TLS handshake is complete, we can see the transmission of the first layer 7 data:
01 I00000315 0xac791937b009b7a61927361d9d453b48e0 pkt tx pkt 0 dcid=0x3afafde2c24248817832ffe545d874a2a01f scid=0x07ff706cb107568ef7116f5f58a9ed9010 type=Short(0x00) len=0
02 I00000315 0xac791937b009b7a61927361d9d453b48e0 frm tx 0 Short(0x00) STREAM(0x13) id=0x0 fin=1 offset=0 len=45 uni=0
03 Ordered STREAM data stream_id=0x0
04 00000000 47 45 54 20 2f 20 48 54 54 50 2f 31 2e 31 0d 0a |GET / HTTP/1.1..|
05 00000010 48 6f 73 74 3a 20 63 6c 6f 75 64 66 6c 61 72 65 |Host: cloudflare|
06 00000020 2d 71 75 69 63 2e 63 6f 6d 0d 0a 0d 0a |-quic.com....|This fragment of the HTTP transaction is transmitted inside what is called a QUIC STREAM, seen on line 02. QUIC streams are one or more communication channels multiplexed within a QUIC connection. QUIC streams are analogous to discrete TCP connections in that they provide data ordering and reliability guarantees, as well as data exchange that is independent from one another. But QUIC streams have some other advantages:
Firstly, QUIC streams are extremely fast to create as they rely on the authenticated client server relationship previously established by the QUIC connection. Evidence of this can be seen in the example above where a stream’s data is transmitted in the same packet that the stream was established.
Secondly, because ordering and reliability are independent for each QUIC stream, the loss of data belonging to one stream will not affect any other streams, providing a solution to the head of line blocking problem that affects protocols that multiplex over TCP, like HTTP/2.
The RESPONSE
Now you should be able to see the fruit of your QUIC toil: the HTTP response!
01 I00001755 0x07ff706cb107568ef7116f5f58a9ed9010 con recv packet len=719
02 I00001755 0x07ff706cb107568ef7116f5f58a9ed9010 pkt rx pkt 3 dcid=0x07ff706cb107568ef7116f5f58a9ed9010 scid=0x type=Short(0x00) len=0
03 I00001755 0x07ff706cb107568ef7116f5f58a9ed9010 frm rx 3 Short(0x00) MAX_DATA(0x04) max_data=1048621
04 I00001755 0x07ff706cb107568ef7116f5f58a9ed9010 frm rx 3 Short(0x00) STREAM(0x12) id=0x0 fin=0 offset=0 len=675 uni=0
Ordered STREAM data stream_id=0x0
05 00000000 48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0d |HTTP/1.1 200 OK.|
…
06 I00001755 0x07ff706cb107568ef7116f5f58a9ed9010 con recv packet len=45
07 I00001755 0x07ff706cb107568ef7116f5f58a9ed9010 pkt rx pkt 4 dcid=0x07ff706cb107568ef7116f5f58a9ed9010 scid=0x type=Short(0x00) len=0
08 I00001755 0x07ff706cb107568ef7116f5f58a9ed9010 frm rx 4 Short(0x00) STREAM(0x16) id=0x0 fin=0 offset=675 len=5 uni=0
Ordered STREAM data stream_id=0x0
09 00000000 31 63 65 0d 0a |1ce..|
…
10 I00001755 0x07ff706cb107568ef7116f5f58a9ed9010 con recv packet len=503
11 I00001755 0x07ff706cb107568ef7116f5f58a9ed9010 pkt rx pkt 5 dcid=0x07ff706cb107568ef7116f5f58a9ed9010 scid=0x type=Short(0x00) len=0
12 I00001755 0x07ff706cb107568ef7116f5f58a9ed9010 frm rx 5 Short(0x00) STREAM(0x16) id=0x0 fin=0 offset=680 len=462 uni=0
Ordered STREAM data stream_id=0x0
13 00000000 3c 21 44 4f 43 54 59 50 45 20 68 74 6d 6c 3e 0a |<!DOCTYPE html>.|
…As can be seen on line 04, the response arrives on the same QUIC STREAM on which it was sent: (0x0).
Many other familiar faces can be seen: line 05: the start of the response headers, line 09: the chunked encoding header and line 13: the start of the response body. It looks almost… normal!
Summary
Thank you for following us on this QUIC odyssey! We understand that the process of building the ngtcp2 example client may be new for some people, but we urge you to keep trying and make use of online resources to help you if you come up against anything unexpected.
But if all went well, and you managed to see the HTTP response from cloudflare-quic.com, then: Congratulations! You and your screen full of debugging gibberish are on the crest of a new wave of internet communication.
Please take a screenshot or a selfie!
Please tell us about it in the comments below!
Please take some time to compare the output you see with the points of interest that I have highlighted above.
And...please visit our blog again to keep up with our developments with QUIC, as support for this exciting new protocol develops.
Subscribe to the blog for daily updates on all our Birthday Week announcements.