Railgun in the real world: faster web page load times

In past blog posts I've described CloudFlare's Railgun technology that is designed to greatly speed up the delivery of non-cached pages. Although CloudFlare caches about 65% of the resources needed to make up a page, something like 35% can't be cached because they are dynamically generated or marked as 'do not cache'. And those 35% are often the initial HTML of the page that must be downloaded before anything else.

To solve that problem CloudFlare came up with a delta compression technique that recognizes that even dynamically-generated or personalized pages change only a little over time or between users. Railgun uses that compression technique to greatly reduce the amount of data that is sent over the Internet to CloudFlare's data centers from backend web servers. The result is faster delivery of the critical HTML that the browser must receive before it can download the rest of the page.

Testing with Railgun showed that very large compression ratios were possible and they resulted in a large speedup in page delivery. But two questions remained: "what's the effect in the real world?" and "how much difference does that make to page load time?".

We're now able to give some answers to those questions. The first hosting partner to roll out Railgun is Montreal-based Vexxhost. They gave us a sample of 51 web sites that they've enabled Railgun on and allowed us to run performance tests to see what difference Railgun makes. We decided to measure three things: how much faster the HTML is delivered, what the compression ratio is and how much page load time changes.

To get useful numbers we decided to load pages multiple times (each page was loaded 20 times with and without Railgun for a total of 40 downloads) and median values were used. Testing was done by downloading the pages from a machine in London, UK. The median round trip time between the nearest CloudFlare data center (where Railgun was running) and the origin web servers was 78ms.

On the 51 sites supplied by Vexxhost we saw a median speedup on downloading the HTML of 1.43x. To put that another way that means that the median time to download the HTML of the web pages decreased to 70% of what it was without Railgun.

Of the 51 sites 11 saw a speedup up of greater than 2x (i.e. the time to download the HTML of the web page more than halved) and for 8 of the sites the speedup was greater than 3x (i.e. the time to download the HTML of the web page was cut to a third of the original).

The median compression ratio achieved by Railgun was 0.65% (i.e. the page was reduced to 0.65% of its size). Of the 51 sites, only 9 saw a compression ratio greater than 3% (i.e. most of the pages were reduced to just a tiny percentage of their original size).

It's this huge compression that enables Railgun to speedup HTML delivery dramatically.

Another measurement to look at is Time To First Byte (how long it takes for the first byte of a page to be delivered to the browser). This is measured as the time from starting the TCP connection to the server to the moment the first byte is received from the server. Railgun has an effect on TTFB as well. The median improvement in TTFB was to drop it to 90% of the non-Railgun-accelerated value.

But HTML delivery is one thing, what's the real end-user visible effect? i.e. how does this translate to a difference in page load time.

Railgun makes a difference to page load time because it accelerates the download of the initial HTML which has to occur before the rest of the page downloads. Downloading the HTML faster helps the entire page download more quickly. Here's an example of the effect of Railgun on CloudFlare's Plans page. This small test was done from the same machine in London as all the other tests. First here's the waterfall for that page without Railgun enabled.

The page load time was 1.83s. Now with Railgun enabled the page load time dropped to 1.15s because the time to download the initial HTML dropped.

Of course, that's just one test. Repeating the test 10 times with and without Railgun saw a median page load time of 1.59s with Railgun and 2.59s without (making the Railgun accelerated time 61% of the non-accelerated page load time). A similar test with CloudFlare's home page showed a median Railgun-accelerated page load time of 2.56s and without Railgun of 3.2s (i.e. Railgun makes the page load time drop to 83% of what it was).

To measure page load time on the 51 sites supplied by Vexxhost we set up PhantomJS (a headless browser that uses the WebKit for engine) on the same machine as used for the measurements above. A small script enabled us to generate HAR files of the download of entire web pages (including the JavaScript, CSS, HTML and images) and to extract the page load time (we use the 'onload' time).

These page load times include assets that are not accelerated by CloudFlare or by Railgun so they show realistic figures of how Railgun helps. Nevertheless, Railgun helps across the sites picked by Vexxhost with a median decrease in page load time to 89% of the original time. The best increase in median page load time was 56%. A small number of sites didn't see an improvement in page load time (they correspond to sites that didn't get a significant Railgun speedup because they typically only had a small amount of HTML).

A comparison of the same site downloaded via Railgun and not can be seen in these two images. The decrease in page load time is due to the decrease in time to get the initial HTML. Here's the page loading without Railgun:

And with Railgun the intial HTML load is accelerated resulting in a faster overall load time.

The difficulty with measuring page load time to see the Railgun-related improvement is that page load time is highly variable as different assets (especially from sites that are not accelerated by a CDN like CloudFlare) cause the page load time to vary enormously. To get a picture of the expected page load time improvement we can move on from measurement to estimation to check that measurements are similar the expected improvement.

One obvious question is to ask how much improvement Railgun can bring to a web site. To work that out you need to know two numbers: the page load time (call it p) and the time to download the initial HTML (call that h). Both values can be obtained from the Developer view in Safari or Chrome or from Firebug.

Railgun will be able to decrease time h. Using the figures above the median improvement would be 70% so you'd expect a page that takes p seconds to load to take roughly p - 0.3 * h with Railgun. In the CloudFlare example above p was 1.83s and h was 0.949s. The formula would give a Railgun page load time of 1.83 - 0.3 * 0.949 = 1.55s (the actual value 1.15s because Railgun did better than the median for that particular page).

In general, the larger the initial HTML the more Railgun can help. Very small pages won't require many round trips between the origin server and Railgun edge server, but larger pages will benefit from the delta compression. And Railgun helps when the web browser and origin server are far apart (for example, when a web site is accessed from around the world, Railgun will help eliminate the round trip time between a web surfer in one country and a web server in another).

To double check the measured performance above we ran a prediction for the sites that Vexxhost gave us. To predict the speedup generated by Railgun we first loaded each page 20 times using PhantomJS and extracted the median page load time (p) and the median time to download the initial HTML (h).

Then using the measured median speedup in the initial HTML load (see the first section above) we predicted that change in page load time by accelerating the initial HTML load and leaving all the other asset load times fixed.

The prediction showed that the median page would load in 93% of the non-Railgun-accelerated time. The measured times were 89%. As with the prediction for the speedup of a CloudFlare page, the measured times are better than the crude predictor, but both show the importance of accelerating the initial HTML load.

In real world tests Railgun gives a median decrease in the page load time to 89% of the non-accelerated time. That translates directly into an improved experience for the end user, and because Railgun runs everywhere in the CloudFlare network it means that page load times are improved for users wherever they are in the world.

Of course, none of this means that web page authors can be complacent about page load time. CloudFlare provides many tools to accelerate web page delivery and web page authors need to be mindful of slow assets and use tools like YSlow to make web page as fast as possible. They need to be particularly mindful of slow third-party assets (such as JavaScript libraries or Like and Share buttons loaded from other domains) as these directly affect page load time.

In fact, the greatest benefit from Railgun comes for sites that have already optimized page load time. Railgun will help drastically reduce the time taken for the already optimized page to reach our edge servers and be sent on to end users. In contrast, a page that has not been optimized may rely on tens of slow or third-party assets that must be downloaded for the page to be ready masking the effects of Railgun.

In a future post I'll look at Railgun performance when accelerating RESTful APIs. And I'll look at the effect of Railgun on subsequent page loads where static assets will be in local cache: in that case Railgun acceleration will be even more noticeable as the HTML download time will be a greater proportion of the total page load time.