My team: the Cloudflare PROTOCOLS team is responsible for termination of HTTP traffic at the edge of the Cloudflare network. We deal with features related to: TCP, QUIC, TLS and Secure Certificate management, HTTP/1 and HTTP/2. Over Q1, we were responsible for implementing the Enhanced HTTP/2 Prioritization product that Cloudflare announced during Speed Week.

This is a very exciting project to be part of, and doubly exciting to see the results of, but during the course of the project, we had a number of interesting realisations about NGINX: the HTTP oriented server onto which Cloudflare currently deploys its software infrastructure. We quickly became certain that our Enhanced HTTP/2 Prioritization project could not achieve even moderate success if the internal workings of NGINX were not changed.

Due to these realisations we embarked upon a number of significant changes to the internal structure of NGINX in parallel to the work on the core prioritization product. This blog post describes the motivation behind the structural changes, how we approached them, and what impact they had. We also identify additional changes that we plan to add to our roadmap, which we hope will improve performance further.

Enhanced HTTP/2 Prioritization aims to do one thing to web traffic flowing between a client and a server: it provides a means to shape the many HTTP/2 streams as they flow from upstream (server or origin side) into a single HTTP/2 connection that flows downstream (client side).

Enhanced HTTP/2 Prioritization allows site owners and the Cloudflare edge systems to dictate the rules about how various objects should combine into the single HTTP/2 connection: whether a particular object should have priority and dominate that connection and reach the client as soon as possible, or whether a group of objects should evenly share the capacity of the connection and put more emphasis on parallelism.

As a result, Enhanced HTTP/2 Prioritization allows site owners to tackle two problems that exist between a client and a server: how to control precedence and ordering of objects, and: how to make the best use of a limited connection resource, which may be constrained by a number of factors such as bandwidth, volume of traffic and CPU workload at the various stages on the path of the connection.

The key to prioritisation is being able to compare two or more HTTP/2 streams in order to determine which one’s frame is to go down the pipe next. The Enhanced HTTP/2 Prioritization project necessarily drew us into the core NGINX codebase, as our intention was to fundamentally alter the way that NGINX compared and queued HTTP/2 data frames as they were written back to the client.

Very early in the analysis phase, as we rummaged through the NGINX internals to survey the site of our proposed features, we noticed a number of shortcomings in the structure of NGINX itself, in particular: how it moved data from upstream (server side) to downstream (client side) and how it temporarily stored (buffered) that data in its various internal stages. The main conclusion of our early analysis of NGINX was that it largely failed to give the stream data frames any 'proximity'. Either streams were processed in the NGINX HTTP/2 layer in isolated succession or frames of different streams spent very little time in the same place: a shared queue for example. The net effect was a reduction in the opportunities for useful comparison.

We coined a new, barely scientific but useful measurement: Potential, to describe how effectively the Enhanced HTTP/2 Prioritization strategies (or even the default NGINX prioritization) can be applied to queued data streams. Potential is not so much a measurement of the effectiveness of prioritization per se, that metric would be left for later on in the project, it is more a measurement of the levels of participation during the application of the algorithm. In simple terms, it considers the number of streams and frames thereof that are included in an iteration of prioritization, with more streams and more frames leading to higher Potential.

What we could see from early on was that by default, NGINX displayed low Potential: rendering prioritization instructions from either the browser, as is the case in the traditional HTTP/2 prioritization model, or from our Enhanced HTTP/2 Prioritization product, fairly useless.

With the goal of improving the specific problems related to Potential, and also improving general throughput of the system, we identified some key pain points in NGINX. These points, which will be described below, have either been worked on and improved as part of our initial release of Enhanced HTTP/2 Prioritization, or have now branched out into meaningful projects of their own that we will put engineering effort into over the course of the next few months.

Write queue reclamation was successfully shipped with our release of Enhanced HTTP/2 Prioritization and ironically, it wasn’t a change made to the original NGINX, it was in fact a change made against our Enhanced HTTP/2 Prioritization implementation when we were part way through the project, and it serves as a good example of something one may call: conservation of data, which is a good way to increase Potential.

Similar to the original NGINX, our Enhanced HTTP/2 Prioritization algorithm will place a cohort of HTTP/2 data frames into a write queue as a result of an iteration of the prioritization strategies being applied to them. The contents of the write queue would be destined to be written the downstream TLS layer.  Also similar to the original NGINX, the write queue may only be partially written to the TLS layer due to back-pressure from the network connection that has temporarily reached write capacity.

Early on in our project, if the write queue was only partially written to the TLS layer, we would simply leave the frames in the write queue until the backlog was cleared, then we would re-attempt to write that data to the network in a future write iteration, just like the original NGINX.

The original NGINX takes this approach because the write queue is the only place that waiting data frames are stored. However, in our NGINX modified for Enhanced HTTP/2 Prioritization, we have a unique structure that the original NGINX lacks: per-stream data frame queues where we temporarily store data frames before our prioritization algorithms are applied to them.

We came to the realisation that in the event of a partial write, we were able to restore the unwritten frames back into their per-stream queues. If it was the case that a subsequent data cohort arrived behind the partially unwritten one, then the previously unwritten frames could participate in an additional round of prioritization comparisons, thus raising the Potential of our algorithms.

The following diagram illustrates this process:

We were very pleased to ship Enhanced HTTP/2 Prioritization with the reclamation feature included as this single enhancement greatly increased Potential and made up for the fact that we had to withhold the next enhancement for speed week due to its delicacy.

In Cloudflare infrastructure, we map the many streams of a single HTTP/2 connection from the eyeball to multiple HTTP/1.1 connections to the upstream Cloudflare control plane.

As a note: it may seem counterintuitive that we downgrade protocols like this, and it may seem doubly counterintuitive when I reveal that we also disable HTTP keepalive on these upstream connections, resulting in only one transaction per connection, however this arrangement offers a number of advantages, particularly in the form of improved CPU workload distribution.

When NGINX monitors its upstream HTTP/1.1 connections for read activity, it may detect readability on many of those connections and process them all in a batch. However, within that batch, each of the upstream connections is processed sequentially, one at a time, from start to finish: from HTTP/1.1 connection read, to framing in the HTTP/2 stream, to HTTP/2 connection write to the TLS layer.

The existing NGINX workflow is illustrated in this diagram:

By committing each streams’ frames to the TLS layer one stream at a time, many frames may pass entirely through the NGINX system before backpressure on the downstream connection allows the queue of frames to build up, providing an opportunity for these frames to be in proximity and allowing prioritization logic to be applied.  This negatively impacts Potential and reduces the effectiveness of prioritization.

The Cloudflare Enhanced HTTP/2 Prioritization modified NGINX aims to re-arrange the internal workflow described above into the following model:

Although we continue to frame upstream data into HTTP/2 data frames in the separate iterations for each upstream connection, we no longer commit these frames to a single write queue within each iteration, instead we arrange the frames into the per-stream queues described earlier. We then post a single event to the end of the per-connection iterations, and perform the prioritization, queuing and writing of the HTTP/2 data frames of all streams in that single event.

This single event finds the cohort of data conveniently stored in their respective per-stream queues, all in close proximity, which greatly increases the Potential of the Edge Prioritization algorithms.

In a form closer to actual code, the core of this modification looks a bit like this:

ngx_http_v2_process_data(ngx_http_v2_connection *h2_conn,
                         ngx_http_v2_stream *h2_stream,
                         ngx_buffer *buffer)
{
    while ( ! ngx_buffer_empty(buffer) {
        ngx_http_v2_frame_data(h2_conn,
                               h2_stream->frames,
                               buffer);
    }

    ngx_http_v2_prioritise(h2_conn->queue,
                           h2_stream->frames);

    ngx_http_v2_write_queue(h2_conn->queue);
}

To this:

ngx_http_v2_process_data(ngx_http_v2_connection *h2_conn,
                         ngx_http_v2_stream *h2_stream,
                         ngx_buffer *buffer)
{
    while ( ! ngx_buffer_empty(buffer) {
        ngx_http_v2_frame_data(h2_conn,
                               h2_stream->frames,
                               buffer);
    }

    ngx_list_add(h2_conn->active_streams, h2_stream);

    ngx_call_once_async(ngx_http_v2_write_streams, h2_conn);
}

ngx_http_v2_write_streams(ngx_http_v2_connection *h2_conn)
{
    ngx_http_v2_stream *h2_stream;

    while ( ! ngx_list_empty(h2_conn->active_streams)) {
        h2_stream = ngx_list_pop(h2_conn->active_streams);

        ngx_http_v2_prioritise(h2_conn->queue,
                               h2_stream->frames);
    }

    ngx_http_v2_write_queue(h2_conn->queue);
}

There is a high level of risk in this modification, for even though it is remarkably small, we are taking the well established and debugged event flow in NGINX and switching it around to a significant degree. Like taking a number of Jenga pieces out of the tower and placing them in another location, we risk: race conditions, event misfires and event blackholes leading to lockups during transaction processing.

Because of this level of risk, we did not release this change in its entirety during Speed Week, but we will continue to test and refine it for future release.

Nginx has an internal buffer region to store connection data it reads from upstream. To begin with, the entirety of this buffer is Ready for use. When data is read from upstream into the Ready buffer, the part of the buffer that holds the data is passed to the downstream HTTP/2 layer. Since HTTP/2 takes responsibility for that data, that portion of the buffer is marked as: Busy and it will remain Busy for as long as it takes for the HTTP/2 layer to write the data into the TLS layer, which is a process that may take some time (in computer terms!).

During this gulf of time, the upstream layer may continue to read more data into the remaining Ready sections of the buffer and continue to pass that incremental data to the HTTP/2 layer until there are no Ready sections available.

When Busy data is finally finished in the HTTP/2 layer, the buffer space that contained that data is then marked as: Free

The process is illustrated in this diagram:

You may ask: When the leading part of the upstream buffer is marked as Free (in blue in the diagram), even though the trailing part of the upstream buffer is still Busy, can the Free part be re-used for reading more data from upstream?

The answer to that question is: NO

Because just a small part of the buffer is still Busy, NGINX will refuse to allow any of the entire buffer space to be re-used for reads. Only when the entirety of the buffer is Free, can the buffer be returned to the Ready state and used for another iteration of upstream reads. So in summary, data can be read from upstream into Ready space at the tail of the buffer, but not into Free space at the head of the buffer.

This is a shortcoming in NGINX and is clearly undesirable as it interrupts the flow of data into the system. We asked: what if we could cycle through this buffer region and re-use parts at the head as they became Free? We seek to answer that question in the near future by testing the following buffering model in NGINX:

On a number of occasions in the above text, I have mentioned the TLS layer, and how the HTTP/2 layer writes data into it. In the OSI network model, TLS sits just below the protocol (HTTP/2) layer, and in many consciously designed networking software systems such as NGINX, the software interfaces are separated in a way that mimics this layering.

The NGINX HTTP/2 layer will collect the current cohort of data frames and place them in priority order into an output queue, then submit this queue to the TLS layer. The TLS layer makes use of a per-connection buffer to collect HTTP/2 layer data before performing the actual cryptographic transformations on that data.

The purpose of the buffer is to give the TLS layer a more meaningful quantity of data to encrypt, for if the buffer was too small, or the TLS layer simply relied on the units of data from the HTTP/2 layer, then the overhead of encrypting and transmitting the multitude of small blocks may negatively impact system throughput.

The following diagram illustrates this undersize buffer situation:

If the TLS buffer is too big, then an excessive amount of HTTP/2 data will be committed to encryption and if it failed to write to the network due to backpressure, it would be locked into the TLS layer and not be available to return to the HTTP/2 layer for the reclamation process, thus reducing the effectiveness of reclamation. The following diagram illustrates this oversize buffer situation:

In the coming months, we will embark on a process to attempt to find the ‘goldilocks’ spot for TLS buffering: To size the TLS buffer so it is big enough to maintain efficiency of encryption and network writes, but not so big as to reduce the responsiveness to incomplete network writes and the efficiency of reclamation.

The Enhanced HTTP/2 Prioritization project has the lofty goal of fundamentally re-shaping how we send traffic from the Cloudflare edge to clients, and as results of our testing and feedback from some of our customers shows, we have certainly achieved that! However, one of the most important aspects that we took away from the project was the critical role the internal data flow within our NGINX software infrastructure plays in the outlook of the traffic observed by our end users. We found that changing a few lines of (albeit critical) code, could have significant impacts on the effectiveness and performance of our prioritization algorithms. Another positive outcome is that in addition to improving HTTP/2, we are looking forward to carrying our newfound skills and lessons learned and apply them to HTTP/3 over QUIC.

We are eager to share our modifications to NGINX with the community, so we have opened this ticket, through which we will discuss upstreaming the event re-ordering change and the buffer partial re-use change with the NGINX team.

As Cloudflare continues to grow, our requirements on our software infrastructure also shift. Cloudflare has already moved beyond proxying of HTTP/1 over TCP to support termination and Layer 3 and 4 protection for any UDP and TCP traffic. Now we are moving on to other technologies and protocols such as QUIC and HTTP/3, and full proxying of a wide range of other protocols such as messaging and streaming media.

For these endeavours we are looking at new ways to answer questions on topics such as: scalability, localised performance, wide scale performance, introspection and debuggability, release agility, maintainability.

If you would like to help us answer these questions and know a bit about: hardware and software scalability, network programming, asynchronous event and futures based software design, TCP, TLS, QUIC, HTTP, RPC protocols, Rust or maybe something else?, then have a look here.

Today Cloudflare opened the door on our beta deployment of QUIC with the announcement of our test site: cloudflare-quic.com. It supports the latest draft of the IETF Working Group’s draft standard for QUIC, which at this time is at: draft 14.

The Cloudflare Systems Engineering Team has a long history of investing time and effort to trial new technologies, often before these technologies are standardised or adopted elsewhere. We deployed early experiments in standards such as: HTTP/2,TLS1.3, DNSSEC, DNS over HTTP, Encrypted SNI, when they were still in incubation. We committed to these technologies in their very early stages because we believed that they made for a safer, faster, better internet. And now we’re excited to do the same with QUIC.

In this blog post, we will show you how you can unlock the cloudflare-quic.com achievement and be some of the first people in the world to perform a HTTP transaction over the global internet using QUIC. This will be a moment that you can tell your grandkids about - if they can stop laughing at your stories of cars with wheels and use of antiquated words like: “meme” and “phone”.

But before we begin, let’s take a little bit of time to review what QUIC is. Our previous blog post The Road to QUIC by my colleague Alessandro Ghedini, gives an excellent introduction to QUIC; its goals, its challenges, and many of the technical advantages that will come with it. It is good background reading for this article and a great introduction to the topic of QUIC in general.

If you visit cloudflare-quic.com with your regular web browser, you will be presented with an informative landing page. However, what you see will not be delivered using QUIC, because at the time this blog is posted, your browser doesn’t support IETF QUIC. No graphical browser does.

Some may point out that Google Chrome has had support for QUIC for many years, but we must re-iterate that the protocol supported by Chrome is Google’s own UDP based transport layer protocol. That protocol was once called QUIC but has forfeited that label and now goes by the name gQUIC, and what’s more, the mechanics of gQUIC are now significantly different to IETF QUIC.

The only way to access cloudflare-quic.com using the QUIC protocol is to use a command line client from one of the various implementations of QUIC that are actively evolving alongside the IETF standard. Most of these implementations can be found here. If you are familiar with any of these, you are welcome to try them against cloudflare-quic.com however please note that your client of choice must support draft 14 of the IETF QUIC standard.

Our preferred QUIC client, and the one whose output we will be analysing in this blog, comes as part of the ngtcp2 project. The original project is located here: github.com/ngtcp2/ngtcp2, but we are hosting our own copy here: github.com/cloudflare/ngtcp2/tree/quic-draft-14 so that we may be sure you get the exact resources you need for this demonstration.

Before proceeding please be aware that the following instructions will require you to build software from source code. ngtcp2 and its dependencies are buildable on multiple Operating System platforms, however, the processes described below are more likely to succeed on Linux. To start with, you will need:

• A POSIX-flavoured operating system, for example: Ubuntu Linux

• To install core software development tools: gcc or clang, libc development packages, make, autoconf, automake, autotools-dev, libtool, pkg-config, git

• To install some additional software dependencies: C Unit tester: (cunit >=2.1), libev development packages. Check the homepage of Cloudflare ngtcp2 copy if you are unsure.

Once you are confident with your setup, run the following commands to retrieve and build the ngtcp2 client and its major dependency OpenSSL:

$ git clone --depth 1 -b quic-draft-14 https://github.com/tatsuhiro-t/openssl
$ cd openssl
$ ./config enable-tls1_3 --prefix=$PWD/build
$ make
$ make install_sw
$ cd ..
$ git clone -b quic-draft-14 https://github.com/cloudflare/ngtcp2
$ cd ngtcp2
$ autoreconf -i
$ ./configure PKG_CONFIG_PATH=$PWD/../openssl/build/lib/pkgconfig LDFLAGS="-Wl,-rpath,$PWD/../openssl/build/lib"
$ make check

If you are still with me, congratulations! The next step is to pre-fabricate a HTTP/1.1 request that we can pass to our QUIC client, in order to avoid typing it out repeatedly. From your ngtcp2 directory, invoke the command:

$ echo -ne "GET / HTTP/1.1\r\nHost: cloudflare-quic.com\r\n\r\n" > cloudflare-quic.req

One of the promises of QUIC is the new QUIC HTTP protocol, which is another IETF standard being developed in conjunction with the QUIC transport layer. It is a re-engineering of the HTTP/2 protocol to allow it to benefit from the many advantages of QUIC.

The design of QUIC HTTP is in a high state of flux at this time and is an elusive target for software implementors, but it is clearly on the Cloudflare product roadmap. For now, cloudflare-quic.com will use HTTP/1.1 for utility and simplicity.

Now it’s time to invoke the ngtcp2 command line client and establish your QUIC connection to cloudflare-quic.com:

$ examples/client cloudflare-quic.com 443 -d cloudflare-quic.req

To be perfectly honest, the debugging output of the ngtcp2 client is not particularly pretty, but who cares! You are now a QUIC pioneer, riding the crest of a new technological wave! Your reward will be the eye-rolls of the teenagers of 2050.

Let’s go over some of the ngtcp2 debugging output that you hopefully can see after invoking your HTTP request over QUIC, and at the same time, let’s relate this output back to some important features of the QUIC protocol.

01 I00000000 0x07ff706cb107568ef7116f5f58a9ed9010 pkt tx pkt 0 dcid=0xba006470cf7c05009e219ff201e4adbef8a3 scid=0x07ff706cb107568ef7116f5f58a9ed9010 type=Initial(0x7f) len=0
02 I00000000 0x07ff706cb107568ef7116f5f58a9ed9010 frm tx 0 Initial(0x7f) CRYPTO(0x18) offset=0 len=309
03 I00000000 0x07ff706cb107568ef7116f5f58a9ed9010 frm tx 0 Initial(0x7f) PADDING(0x00) len=878
04 I00000000 0x07ff706cb107568ef7116f5f58a9ed9010 rcv loss_detection_timer=1537267827966896128 last_hs_tx_pkt_ts=1537267827766896128 timeout=200

Above is what the QUIC protocol calls the client initial packet. It is the packet that is sent to establish a completely new connection between the client and the QUIC server.

The element: scid on line 01 is an example of a source connection ID. This is the unique number that the client chooses for itself when sending an initial packet. In the example output above, the value of the client scid is: 0x07ff706cb107568ef7116f5f58a9ed9010 but you will see a different value. In the ngtcp2 client utility, this number is purely random, as the QUIC connection will only last as long as the command runs, and therefore doesn’t need to carry much meaning. In future, more complex QUIC clients (such as web browsers) will likely choose their source connection IDs more carefully. Future QUIC servers will certainly do this, as connection IDs are a good place to encode information.

Encoding information in source connection ids is of particular interest to an organisation like Cloudflare, where a single IP address can represent thousands of physical servers. To support QUIC in an infrastructure like ours, routing of UDP QUIC packets will need to be done to a greater level of precision than can be represented in an IP address, and large, data packed connection IDs will be very useful for this. But enough about us, this blog is about you!

The element: dcid, also on line 01, is the destination connection ID. In the client initial phase, this is always random as per the QUIC specification, because the client wants to be sure that it is treated as ‘new’ by the QUIC server. A random dcid, particularly one that is the maximum allowed length of 144bits, combined with a large source connection id, has a sufficiently high statistical chance of being unique so as to not clash with a connection id that the QUIC server has already registered. Later we will see what the QUIC server does with the random destination connection ID that the client has chosen.

On line 02, we see that the client initial packet includes a CRYPTO frame that contains the TLS client hello message. This clearly demonstrates one of the significant advantages in the design of QUIC: the overlapping of transport layer connection establishment and TLS layer negotiation. Both of these processes necessitate some back and forth between a client and server for both TLS over TCP and for QUIC.

In TLS over TCP the two processes happen one after the other:

You can count a total of FOUR round trips between the client & the server before a HTTP request can be made! Now compare that with QUIC, where they happen at the same time:

That’s a 50% reduction! With just TWO round trips before you can start making HTTP requests.

Returning to the analysis of the ngtcp2 debug output, we can see the client initial packet adds a PADDING frame in order to bring the packet to a minimum size mandated by the QUIC specification. The reason for this is twofold:

Firstly, to ensure that the network between the QUIC client and server can support satisfactorily large UDP packets. Sadly UDP is a second class citizen on the wide internet, generally only being used to transmit single, small, unrelated packets. QUIC counters all three of these patterns, so the rationale here is: if it’s going to get stuck, better to find out early. The quality of network support for streams of UDP will hopefully evolve alongside QUIC.

Secondly, to reduce the effectiveness of amplification attacks. This type of attack is where bad actors take advantage of network services that produce server responses vastly greater in size than the barely-validated request that solicited them. By spoofing the address of a victim, a bad actor can bombard the victim with large volumes of server responses given a relatively small volume of requests to the server. By requiring that an initial request be large, QUIC helps to make the amplification value much lower. UDP based amplification attacks are a very real issue, and you can read Cloudflare's account of a such an attack here.

QUIC defines a number of other mechanisms to protect against amplification attacks as well as DDoS attacks and you will see some of these a bit later.

Further down you will see the first packet returned from the server, which is the server initial packet:

01 I00000160 0x07ff706cb107568ef7116f5f58a9ed9010 pkt rx pkt 0 dcid=0x07ff706cb107568ef7116f5f58a9ed9010 scid=0x3afafde2c24248817832ffe545d874a2a01f type=Initial(0x7f) len=111
02 I00000160 0x07ff706cb107568ef7116f5f58a9ed9010 frm rx 0 Initial(0x7f) CRYPTO(0x18) offset=0 len=90
…
03 I00000314 0x07ff706cb107568ef7116f5f58a9ed9010 cry remote transport_parameters negotiated_version=0xff00000e
04 I00000314 0x07ff706cb107568ef7116f5f58a9ed9010 cry remote transport_parameters supported_version[0]=0xff00000e
…
05 I00000314 0x07ff706cb107568ef7116f5f58a9ed9010 frm tx 3 Initial(0x7f) ACK(0x0d) largest_ack=1 ack_delay=0(0) ack_block_count=0
06 I00000314 0x07ff706cb107568ef7116f5f58a9ed9010 frm tx 3 Initial(0x7f) ACK(0x0d) block=[1..0] block_count=1

The response destination connection id (dcid on line 01) is the client’s original source connection ID (scid) which in this example is: 0x07ff706cb107568ef7116f5f58a9ed9010.

The server has now discarded the client’s randomly-chosen dcid after finding that the client is ‘new’, and replaced it with its own connection ID which you can see as the packet source connection ID scid on line 01, which in this example is: 0x3afafde2c24248817832ffe545d874a2a01.

Starting from this point, both the QUIC client and server recognise each other’s connection IDs, opening the door to a powerful feature of QUIC: connection migration. Connection migration will allow QUIC clients and servers to change their IP addresses and ports, but still maintain the QUIC connection. QUIC packets arriving from or to the new IP/port can continue to be handled because the connection ID, which has not changed, will act as the primary identifier of the connection context. For our first cloudflare-quic.com demonstration, connection migration is not supported, but we’ll be working on this as we develop our QUIC offerings.

The server initial packet contains the next part of the TLS handshake, found in the CRYPTO frame on line 01, which is the first part of the TLS server hello and may contain elements such as handshake key material and the beginning of the server’s certificate chain.

Lines 03 and 04 show the exchange of transport parameters, which are QUIC specific configuration values declared by one side to the other and used to control various aspects of the connection. These parameters are encoded and transmitted within the TLS handshake. This not only reiterates the close relationship between the TLS and transport layers in QUIC, but also demonstrates QUIC’s focus on security, as the exchange of these parameters will be protected against outside tampering as part of the TLS handshake.

Lines 05 and 06 show an example of some acknowledgement frames being sent from the client to the server. Acknowledgements are part of the QUIC loss detection mechanism that deals with data losses that inevitably happen on large networks, however during the handshake phase, acknowledgements also have another use: to hint at the validity of the client by proving to the server that a client is truly interested in communicating with the server and is not at a spoofed address.

Without any form of source validation, QUIC servers will severely limit the amount of data that they send to clients. This protects helpless, spoofed victims of amplification attacks (in conjunction with the client initial packet minimum size requirement described above), and also helps protect the QUIC server from the equivalent of a TCP SYN attack, by constraining the commitment that the QUIC server will make to an unvalidated client.

For Cloudflare, there are vast unknowns in regard to DDoS and SYN style attacks against QUIC and it is a subject we are supremely interested in. While these threat models remain unknown, our protections around cloudflare-quic.com will be effective but… remorseless.

Once the TLS handshake is complete, we can see the transmission of the first layer 7 data:

01 I00000315 0xac791937b009b7a61927361d9d453b48e0 pkt tx pkt 0 dcid=0x3afafde2c24248817832ffe545d874a2a01f scid=0x07ff706cb107568ef7116f5f58a9ed9010 type=Short(0x00) len=0
02 I00000315 0xac791937b009b7a61927361d9d453b48e0 frm tx 0 Short(0x00) STREAM(0x13) id=0x0 fin=1 offset=0 len=45 uni=0
03 Ordered STREAM data stream_id=0x0
04 00000000  47 45 54 20 2f 20 48 54  54 50 2f 31 2e 31 0d 0a  |GET / HTTP/1.1..|
05 00000010  48 6f 73 74 3a 20 63 6c  6f 75 64 66 6c 61 72 65  |Host: cloudflare|
06 00000020  2d 71 75 69 63 2e 63 6f  6d 0d 0a 0d 0a           |-quic.com....|

This fragment of the HTTP transaction is transmitted inside what is called a QUIC STREAM, seen on line 02. QUIC streams are one or more communication channels multiplexed within a QUIC connection. QUIC streams are analogous to discrete TCP connections in that they provide data ordering and reliability guarantees, as well as data exchange that is independent from one another. But QUIC streams have some other advantages:

Firstly, QUIC streams are extremely fast to create as they rely on the authenticated client server relationship previously established by the QUIC connection. Evidence of this can be seen in the example above where a stream’s data is transmitted in the same packet that the stream was established.

Secondly, because ordering and reliability are independent for each QUIC stream, the loss of data belonging to one stream will not affect any other streams, providing a solution to the head of line blocking problem that affects protocols that multiplex over TCP, like HTTP/2.

Now you should be able to see the fruit of your QUIC toil: the HTTP response!

01 I00001755 0x07ff706cb107568ef7116f5f58a9ed9010 con recv packet len=719
02 I00001755 0x07ff706cb107568ef7116f5f58a9ed9010 pkt rx pkt 3 dcid=0x07ff706cb107568ef7116f5f58a9ed9010 scid=0x type=Short(0x00) len=0
03 I00001755 0x07ff706cb107568ef7116f5f58a9ed9010 frm rx 3 Short(0x00) MAX_DATA(0x04) max_data=1048621
04 I00001755 0x07ff706cb107568ef7116f5f58a9ed9010 frm rx 3 Short(0x00) STREAM(0x12) id=0x0 fin=0 offset=0 len=675 uni=0
Ordered STREAM data stream_id=0x0
05 00000000  48 54 54 50 2f 31 2e 31  20 32 30 30 20 4f 4b 0d  |HTTP/1.1 200 OK.|
…
06 I00001755 0x07ff706cb107568ef7116f5f58a9ed9010 con recv packet len=45
07 I00001755 0x07ff706cb107568ef7116f5f58a9ed9010 pkt rx pkt 4 dcid=0x07ff706cb107568ef7116f5f58a9ed9010 scid=0x type=Short(0x00) len=0
08 I00001755 0x07ff706cb107568ef7116f5f58a9ed9010 frm rx 4 Short(0x00) STREAM(0x16) id=0x0 fin=0 offset=675 len=5 uni=0
Ordered STREAM data stream_id=0x0
09 00000000  31 63 65 0d 0a                                    |1ce..|
…
10 I00001755 0x07ff706cb107568ef7116f5f58a9ed9010 con recv packet len=503
11 I00001755 0x07ff706cb107568ef7116f5f58a9ed9010 pkt rx pkt 5 dcid=0x07ff706cb107568ef7116f5f58a9ed9010 scid=0x type=Short(0x00) len=0
12 I00001755 0x07ff706cb107568ef7116f5f58a9ed9010 frm rx 5 Short(0x00) STREAM(0x16) id=0x0 fin=0 offset=680 len=462 uni=0
Ordered STREAM data stream_id=0x0
13 00000000  3c 21 44 4f 43 54 59 50  45 20 68 74 6d 6c 3e 0a  |<!DOCTYPE html>.|
…

As can be seen on line 04, the response arrives on the same QUIC STREAM on which it was sent: (0x0).

Many other familiar faces can be seen: line 05: the start of the response headers, line 09: the chunked encoding header and line 13: the start of the response body. It looks almost… normal!

Thank you for following us on this QUIC odyssey! We understand that the process of building the ngtcp2 example client may be new for some people, but we urge you to keep trying and make use of online resources to help you if you come up against anything unexpected.

But if all went well, and you managed to see the HTTP response from cloudflare-quic.com, then: Congratulations! You and your screen full of debugging gibberish are on the crest of a new wave of internet communication.

• Please take a screenshot or a selfie!

• Please tell us about it in the comments below!

• Please take some time to compare the output you see with the points of interest that I have highlighted above.

• And...please visit our blog again to keep up with our developments with QUIC, as support for this exciting new protocol develops.

Subscribe to the blog for daily updates on all our Birthday Week announcements.