What we've been doing with Go

Almost two years ago CloudFlare started working with Go. What started as an experiment on one network and concurrency heavy project has turned into full, production use of Go for multiple services. Today Go is at the heart of CloudFlare's services including handling compression for high-latency HTTP connections, our entire DNS infrastructure, SSL, load testing and more.

I first wrote about CloudFlare's use of Go back in July 2012. At that time there was only one CloudFlare project, Railgun, written in Go, but others were starting to germinate around the company. Today we have many major projects in Go. So, we celebrate Go's 4th birthday with a short list of interesting things we've written in Go.

RRDNS is a DNS proxy that was written to help scale CloudFlare's DNS infrastructure. Our DNS was already very fast and we wanted to make it faster, more reliable and resilient to attack.

RRDNS provides response rate limiting to stop DNS attacks, caching to lower the load on the database, load balancing to detect downed upstreams, seamless binary upgrade (with no service interruption), CNAME flattening and more. It is built on a modular framework that allows the implementation of each behaviour to exist in separately maintained modules.

This modularity was trivial to enforce with interfaces, allowing each module to remain strictly self-contained but retain the flexibility of implementation (some use cgo, others have background workers). A goroutine is dedicated to each request to force isolation where panics are recovered and logged instead of crashing the server.

The guarantees needed to avoid leaving the server in a bad state when handling panics would be impossible without the defer mechanism Go provides. Other language features simplified the complexity of writing new modules; garbage collection allows us to avoid complex reference counting schemes and concentrate on the business logic. Managing the concurrent requests' access to shared data was also easy, either by wrapping this in a goroutine or using read-write locks.

Railgun is CloudFlare's compression technology that's used to speed up connections between CloudFlare's data centers and origin web servers (especially when there is high latency between them). It's 100% written in Go and performs on the fly compression of web pages (and other textual assets) against recent versions of those pages to send the absolute minimum data possible.

Railgun also helps cache previously uncacheable assets (such as dynamically generated web pages) by spotting the often small changes between web pages over time, or the small changes between different users. Railgun is now widely used by CloudFlare customers to create responsive web sites wherever the end user is in the world.

It is now widely used by CloudFlare customers and hosting partners and achieves impressive real world speedups, including faster TTFB and better page load time, and bandwidth savings which translate into additional savings for people using services like AWS.

Red October is a cryptographically secure implementation of the two-man rule control mechanism. It is a software-based encryption and decryption server. The server allows authorized individuals to encrypt a payload in such a way that no one individual can decrypt it. To decrypt the payload, at least two authorized individuals must be logged into the server. The decryption of the payload is cryptographically tied to the login credentials of the authorized individuals and is only kept in memory for the duration of the decryption.

We'll be using Red October to secure very sensitive material (such as private encryption keys) so that no single CloudFlare employee (or single attacker) can get access to them. In coming months we'll write more about the cryptographic underpinnings of keeping CloudFlare secure.

The SSL Bundler allows us to take a customer's own SSL certificate and compute the fastest, shortest chain of intermediate certificates that can be used to verify the connection. When someone uploads a custom SSL certificate, we use our directory of intermediate certificates to build all the possible chains from the uploaded cert to a trusted browser root. We then rank these chains based on a number of factors including:

1. The length of the certificate chain

2. The ubiquity of the root certificate in browsers and other clients

3. The security of each step in the chain (e.g., does their Extended Key Usage include Server Authentication)

4. The length of the validity period of all the steps in the chain

The result is a server bundle that is small, fast and strong while having ubiquitous browser and client support. We then present that chain of certificates when an SSL connection is made so that the browser can securely verify the SSL connection as quickly as possible.

We're also experimenting with Go-based software like CoreOS (we're working with their go-raft library) and Docker. We have internal tools for load testing written in Go for Kyoto Tycoon, HTTP servers, SPDY and memcached. And we've open-sourced our Go stream processing library, a Go wrapper for high-performance syslogging, our changes to Go bindings for Kyoto Cabinet, and a tool for 'curling' SPDY sites. And, of course, we've been contributing directly to Go itself.

All in all Go is one of the main languages in use at CloudFlare and from here it looks like it has a bright future.

Happy 4th Birthday Go!

(And, yes, we're hiring Go programmers in London and San Francisco)