Making Edge Side Includes (ESI) Automatic and Easy

On HTTP, caching is done at the file level. A browser will cache the JPEG, CSS, and Javascript files on a page. However, the HTML of most pages is dynamically generated. As a result, the pages cannot be cached. This is unfortunate because the HTML of even highly dynamic pages rarely changes more than 10%. The 90% of the HTML that is the same from one request to the next is transmitted needlessly.

On the web, compression equals performance. If you can compress a response by 50% you will, roughly, double network performance. Given that 90%+ of HTML doesn't need to be transmitted over the network, if you could only transmit the actually dynamic parts of the content then you'd get a massive performance increase.

Recognizing this opportunity, traditional content delivery network (CDN) vendors created the Edge Side Include (ESI) protocol. The protocol was submitted as an official standard to the World Wide Web Consortium (W3C) but it was never accepted. A handful of other old school CDNs today support ESI, although it's adoption has been slow.

Here's how ESI works: when you create a web page you determine what parts are static and dynamic. You implement the static portions as a file that you upload to your CDN. Within that file you include tags that reference the dynamic portions of the content, with URL of where to fetch the dynamic portions from. The CDN fetches each of these dynamic resources and combines them with the static portion in order to render the HTML of the page before it sent across the network back to the browser.

If that sounds easy to implement then you likely haven't done much web development. To get a sense of the complexity, check out this 106 page ESI developer's guide. While ESI can theoretically deliver significant performance benefits, the pain of actually developing for it is significant. And, once you've developed for it, there's significant process lock-in: good luck ever leaving. We think you shouldn't have to learn a new programming language or change a single line of your code just to make your site fast.

And if you're spending your time bending your HTML so that a CDN can serve it up better then you're not spending it on developing your actual web site.

Yesterday, we posted about Railgun and how it lets you cache what was previously uncacheable content. One way of thinking about Railgun is that it is like automatic ESI support without the work. Rather than you having to tag your own content to mark what is static and what is dynamic, Railgun automatically determines the static portions of HTML and caches that at the edge. Dynamic portions of HTML are always fetched from the origin without you needing to change a single line of code.

Moreover, the caching logic is responsive to what is actually happening on the page. If different elements on the page change at different rates then Railgun's cache will deliver them optimally, never wasting a byte that doesn't otherwise need to be transmitted. And, since you don't need to change how you write code in order to support Railgun, there's no process lock-in if you ever decide to turn the service off. While youwon't get the benefits of Railgun without CloudFlare, you won't need to completely rewrite your code. In fact, you won't need to change a thing.

We talk to a lot of web publishers and the constant refrain we hear is that the performance and security tools that are available to them are too expensive and too complicated. We've had nearly half a million websites sign up for CloudFlare largely because we focused on these two issues. Railgun takes ESI, another technology that was previously reserved for only those sites with huge budgets and dedicated CDN management teams, and makes it available in a way that is affordable and easy to implement.

Because Railgun requires software to be installed on the origin server, we have limited its availability to our Business and Enterprise customers. However, our plan is to roll it out to CloudFlare's Free- and Pro-level customers if they are hosted on a CloudFlare Optimized Hosting Partner. If you're interested in Railgun, you can upgrade to CloudFlare Business or Enterprise. Alternatively, ping your hosting provider to know they should become a CloudFlare Optimized Host. It's free for hosts and, if they tell us you're the person who convinced them to sign up, we'll send you a T-shirt and make sure you're one of the first of the host's customers to get access to Railgun.