Enjoy a slice of QUIC, and Rust!

During last year’s Birthday Week we announced early support for QUIC, the next generation encrypted-by-default network transport protocol designed to secure and accelerate web traffic on the Internet.

We are not quite ready to make this feature available to every Cloudflare customer yet, but while you wait we thought you might enjoy a slice of quiche, our own open-source implementation of the QUIC protocol written in Rust.

Quiche will allow us to keep on top of changes to the QUIC protocol as the standardization process progresses and experiment with new features more easily. Let’s have a quick look at it together.

The main design principle that guided quiche’s initial development was exposing most of the QUIC complexity to applications through a minimal and intuitive API, but without making too many assumptions about the application itself, in order to allow us to reuse the same library in different contexts.

For example, while we think Rust is great, most of the stack that deals with HTTP requests on Cloudflare’s edge network is still written in good ol’ C, which means that our QUIC implementation would need to be integrated into that.

The quiche API can process QUIC packets received from network sockets and generate packets to send back, but will not touch the sockets themselves. It also exposes a thin layer on top on the Rust API itself to make integration into C/C++ (and other languages) easier.

The application is responsible for fetching data from the network (e.g. via sockets), passing it to quiche, and sending the data that quiche generates back into the network. The application also needs to handle timers, with quiche telling it when to wake-up (this is required for retransmitting lost packets once the corresponding retransmission timeouts expire for example). This leaves the application free to decide how to best implement the I/O and event loop support, depending on the support offered by the operating system or the networking framework used.

Thanks to this we were able to integrate quiche into our NGINX fork (though this is not ready to be open-sourced just yet) without major changes to NGINX internals. Quiche can also be built together with cURL to power cURL’s very early (and very experimental) QUIC support. And of course you can use quiche to implement QUIC clients and servers written in Rust as well.

A couple years ago we migrated our entire HTTPS stack to BoringSSL, the crypto and TLS library developed by Google. This allowed us to streamline our stack (we’d previously had to maintain our own internal patches to OpenSSL for many of the features that BoringSSL offers out-of-the-box) as well as ship exciting new features more quickly.

For those not following the QUIC standardization process, the QUIC protocol itself makes use of TLS 1.3 as part of its connection handshake, so it made sense that our QUIC implementation would also use BoringSSL to implement that part of the protocol.

As far as QUIC is concerned, the TLS library needs to provide negotiation of cryptographic parameters (including encryption secrets), which are then used by the QUIC layer itself to encrypt/decrypt the packets on the wire. The TLS record layer is replaced by QUIC framing to avoid overhead and duplication so that TLS handshake messages are carried directly on top of encrypted QUIC packets.

This makes integrating with existing TLS implementations more challenging since they would need to expose the raw TLS handshake messages as-is, without record layer or protection which would be handled by quiche itself.BoringSSL offers a dedicated API that can be used by QUIC implementations, which required a few tweaks along the way as you would expect with something so new and experimental, but was overall a breeze to integrate into quiche.

While the TLS handshake is implemented using BoringSSL’s API directly (via Rust’s FFI support), to implement QUIC’s packet protection we decided to use ring, a very popular Rust library that provides safe and fast cryptographic primitives.

Ring offers most of the same cryptographic primitives you’d get with BoringSSL’s libcrypto, but exposed through an intuitive and safe Rust API. In fact, ring uses some of the same fast implementations of cryptographic algorithms that BoringSSL also uses, but exposed through a nicer API.

However QUIC’s use of cryptography is somewhat unique and, uhm, exotic: the packet’s payload protection uses standard AEAD (“authenticated encryption with associated data”) algorithms like AES-GCM and ChaCha20-Poly1305, but the protection for the packet’s header is different and was designed specifically for QUIC to prevent middle-boxes on the network from intercepting some of the packet’s metadata (like the packet number).

Ring didn’t originally expose the primitives required to implement QUIC’s header protection, but adding support for them was easy enough, and our changes, now also open-source, were officially released in ring v0.14.0 and available for everyone to use.

While quiche is one of the more recent additions to the list of QUIC implementations (its first commit only dates back 3 months or so), it’s already able to interoperate with the other more mature implementations and exercise many of QUIC’s features.

Quiche, just like QUIC itself, is not “done” (or perfect for that matter) yet. Bugs will be found and fixed, new exciting features implemented (and then more bugs will be found), and API compatibility broken, as we gain experience and learn from wider deployment of QUIC on the Internet.