The other day I blogged here about our new Railgun software that speeds up the back haul between CloudFlare data centers and our clients' servers. At CloudFlare we're using a number of different languages depending on the task: C or C++ for all core services, PHP for the main web site, Lua for customization of nginx and an extensive amount of JavaScript. Railgun is slightly different as it's about 4,000 lines of Go of which about 3,000 are code (not comments).
Image source: stanleylieber.com; created by Renée French
We chose to use Go for Railgun because Railgun is inherently highly concurrent. A single instance of the Railgun client should be able to handle large numbers of requests from the CloudFlare data center for content and then multiplex them across an Internet connection to be handled. Go's concurrency makes writing software that must scale up and down very easy.
Railgun makes extensive use of goroutines and channels. Goroutines handle both the multiplexed Internet connections (of which there could be many between a single CloudFlare data center and our clients) and the connections needed to get content from origin web servers and provide the content back to the nginx server that sends it on to the web browser.
Probably the nicest thing about goroutines and channels is that they make it easy to create 'fire and forget until needed' systems. You create a channel, create a goroutine that communicates on that channel and then read from the channel when needed (perhaps using a select statement).
(Aside: for those who studied computer science, Go owes a lot to Hoare's CSP and Dijkstra's Guarded Commands.
A small example of a goroutine inside Railgun is this unique ID generator. It generates a sequence of IDs that are used to identify streams (a stream contains a single HTTP request) being sent between the CloudFlare data center and a client site.
It works by adding data to a SHA1 hash and each time a read is made on the channel id a new string ID is created by hashing the data. The whole thing is running as an independent goroutine that only does work when needed. (You can play with this code live here).
Another powerful aspect of Go is the way in which it handles object orientation. Go has a notion of an interface which is used to identify a capability of an object and where it can be used. One common interface is io.Writer. To be an io.Writer an object has to implement the Write function which has the signature Write(p []byte) (n int, err error). Any object that implements that can be used wherever an io.Writer is needed.
In Railgun there's a simple object called a Counter that is an io.Writer. It turns an ordinary io.Writer into one that writes, but also counts how much it's written. When its Write is called it keeps track of the number of bytes and calls the underlying io.Writer. It looks like this:
As an example of its use, here's how the unique ID generator above can be altered to count the number of bytes of data that have been written to the SHA1 hash. Since h implements io.Writer it can be passed to counter.New and it can be used to write data to the hash and keep a count. Reading from the count channel would retrieve how many bytes had been written. (See the live version for an example).
Part of the reason Railgun is so small is that Go's library is extensive and easy to work with. Go has libraries for HTTP, raw network connections, URL manipulation, TLS, many different types of serialization systems, cryptographic hashing, compression, and the more mundane string manipulation, date/time, and logging.
Another reason Go has been helpful is that is generates a single executable that can be distributed to our clients. There's no complex dependency chain or layout of shared libraries to worry about.