Pushing Nginx to its limit with Lua

At CloudFlare, Nginx is at the core of what we do. It is part of the underlying foundation of our reverse proxy service. In addition to the built-in Nginx functionalities, we use an array of custom C modules that are specific to our infrastructure including load balancing, monitoring, and caching. Recently, we've been adding more simple services. And they are almost exclusively written in Lua.

I wanted to share more about how we are augmenting Nginx with new capabilities using Lua and provide some examples so you can do the same.

Lua is a scripting language. Specifically, it is a full-featured multi-paradigm language with a simple syntax and semantics that resemble JavaScript or Scheme. Lua also has an interesting story to it, as it is one of the only languages from an emerging country that has had worldwide impact.

Lua has always meant to be embedded with larger systems written in other languages (like C and C++), and has thrived at staying very minimal and easy to integrate. As a result, Lua is popular within videogames, security oriented software, and, morerecently, Wikipedia.

Nginx+Lua is a self-contained web server embedding the scripting language Lua. Powerful applications can be written directly inside Nginx without using cgi, fastcgi, or uwsgi. By adding a little Lua code to an existing Nginx configuration file, it is easy to add small features. To see it yourself, at the end of this post I've included some logging code that can be added to any existing configuration.

One of the core benefits of Nginx+Lua is that it is fully asynchronous. Nginx+Lua inherits the same event loop model that has made Nginx a popular choice of webserver. "Asynchronous" simply means that Nginx can interrupt your code when it is waiting on a blocking operation, such as an outgoing connection or reading a file, and run the code of another incoming HTTP Request.

All the Lua code is written in a sequential fashion. The asynchronous logic is hidden to the Nginx+Lua programmer. If you are familiar with other event-driven webservers, that means no callbacks. In addition, Nginx+Lua is blazingly fast, leveraging the LuaJIT interpreter.

You can install it from source by compiling the lua nginx-module with your existing Nginx. If you chose that path you will also need a Lua interpreter. LuaJIT-2.0.0 is recommended.

Or, you can use the tested ngx_openresty bundle. ngx_openresty comes loaded with Nginx, 3rd party modules, Lua libraries and other goodies. If youalready use Nginx without 3rd party modules, from your Linux distribution for instance, you can safely swap it out with ngx_openresty. (Quick shout-out to my CloudFlare colleague Yichun Zhang who wrote ngx_openresty. Thanks, Yichun!)

What makes Nginx, and therefore Nginx+Lua, really fast is the asychronous model and the event loop that Nginx relies on. To stay within that model, outgoing communication that is outside of Nginx has to be treated carefully. It is not recommended that you use classic LuaSocket, and instead it is recommended that you rely on the built-in ngx_luasockets.

However, with a multitude of openresty libraries to "speak" SQL, memcached, and Redis, as well as the DNS built on top of ngx_lua sockets, this isn't really a problem in practice.

Here is an example of how to build and run a simple log aggregator for Nginx. You can add it to any of your own existing configuration. This is the output once the aggregated logs are funneled to a time series system:

This particular example graph shows the average number of requests per second on certain nodes of the CloudFlare infrastructure.

Let's assume you already have a working webapp in Nginx, or that you use the proxy_pass directives to upstream to an Apache server.

First, add some lines in the Nginx conf to look at .lua files, and use a 1MB space of shared memory between your Nginx workers. ($prefix is relative to your Nginx install).

Next, add a little Lua snippet to calculate request_time for each request, and aggregate it into shared memory using a logging library available. Here is a simple logging library that I built.

This snippet can be used directly inline in your Nginx conf, using the log_by_lua directive.

The last step to complete the example is a system to collect and/or display logs. In the full example, we set the aggregation as a separate server listening on a different port.

You should now have a functioning Nginx+Lua modification running in your environment.

This example showed how Lua found its way into our system at CloudFlare, but we soon realized that it wasn't limited to aggregating and printing logs. Using the same phases that Nginx has laid out for processing HTTP requests, it is becoming possible to add interesting new capabilities to Nginx, with almost as much control as a custom C module, while being pleasant and easy to write.

For instance, the access phase can be seen as a programmatic .htaccess, and even more. Whereas the content phase is where your web application would go.

Nginx+Lua has become a foundation for the work that I do at CloudFlare. As a long-time C developer, I am constantly struck by how powerful and extremely expressive Lua can be, while being simple and approachable as well.

Sometimes, simple is beautiful.

PS - We're hiring Lua programmers who are interested in working at extreme scale. Check out the Systems Engineer listing on our careers page if you're interested.