Mapping Factorio with Leaflet

The following is a guest post by Jacob Hands, Creator of FactorioMaps.com. He is building a community site for the game Factorio centered around sharing user creations.

Factorio is a game about building and maintaining factories. Players mine resources, research new technology and automate production. Resources move along the production line through multiple means of transportation such as belts and trains. Once production starts getting up to speed, alien bugs start to attack the factory requiring strong defenses.

A Factorio factory producing many different items.

A Factorio military outpost fighting the alien bugs.

A Factorio map view of a small factory, that’s still too big to easily share fully with screenshots.

At FactorioMaps.com, I am building a place for the community of Factorio players to share their factories as interactive Leaflet maps. Due to the size and detail of the game, it can be difficult to share an entire factory through a few screenshots. A Leaflet map provides a Google Maps-like experience allowing viewers to pan and zoom throughout the map almost as if they are playing the game.

Leaflet maps contain thousands of small images for X/Y/Z coordinates. Amazon S3 and Google Cloud Storage are the obvious choices for low-latency object storage. However, after 3.5  months in operation, FactorioMaps.com contains 17 million map images (>1TB). For this use-case, $0.05 per 10,000 upload API calls and $0.08 to 0.12/GB for egress would add up quickly. Backblaze B2 is a better fit because upload API calls are free, egress bandwidth is $0.00/GB to Cloudflare, and storage is 1/4th the price of the competition.

Backblaze B2 requires a prefix of /file/bucketName on all public files, which I don’t want. To remove it, I added a VPS proxy to rewrite paths and add a few 301 redirects. Unfortunately, the latency from the user -> VPS -> B2 was sub-par averaging 800-1200ms in the US.

Leaflet maps work by loading images at the user's X/Y/Z coordinates to render the current view. As a map is zoomed in, it requires 4x as many images to show the same area. That means 75% of a map's images are in the max rendered zoom level.

A diagram of how each zoom level is 4x larger than the previous

With hosting working, it's time to start making the site faster. The majority of image requests come from the first few zoom levels, representing less than 25% of a given map's images. Adding a local SSD cache on the VPS containing all except the last 1-3 zoom levels for each map reduces latency for 66% of requests. The problem with SSD storage is it's difficult to scale with ever-increasing data and is still limited to the network and CPU performance of the server it occupies.

Cloudflare Workers can run JavaScript using the Service Workers API which means the path rewrites and redirects the VPS was accomplishing could run on Cloudflare's edge.

While Google Cloud Storage is more expensive than B2, it has much lower latency to the US and worldwide destinations because of their network and multi-regional object storage. However, it's not time to move the whole site over to GCS just yet; the upload API calls alone would cost $85 for 17 million files.

The first few zoom levels are stored in GCS, while the rest are in B2. Cloudflare Workers figure out where files are located by checking both sources simultaneously. By doing this, 66% of requested files come from GCS with a mean latency of <350ms, while only storing 24% of files on GCS. Another benefit to using B2 as the primary storage is if GCS becomes too expensive in the future, I can move all requests to B2.

// Race GCS and B2
let gcsReq = new Request('https://storage.googleapis.com/bucketName' + url.pathname, event.request)
let b2Req = new Request(getB2Url(request) + '/bucketName' + url.pathname, event.request);

// Fetch from GCS and B2 with Cloudflare caching enabled
let gcsPromise = fetch(gcsReq, cfSettings);
let b2Promise = fetch(b2Req, cfSettings);

let response = await Promise.race([gcsPromise, b2Promise]);
if (response.ok) {
    return response;
}

// If the winner was bad, find the one that is good (if any)
response = await gcsPromise;
if (response.ok) {
    return response;
}

response = await b2Promise;
if (response.ok) {
    return response;
}

// The request failed/doesn't exist
return response;

The Cloudflare Workers dashboard contains a few analytics for subrequests, but there is no way to see what responses came from B2 vs. GCS. Fortunately, it’s easy to send request stats to a 3rd party service like StatHat with a few lines of JavaScript.

 // Fetch from GCS and B2 with caching
let reqStartTime = Date.now();
let gcsPromise = fetch(gcsReq, cfSettings);
let b2Promise = fetch(b2Req, cfSettings);

let response = await Promise.race([gcsPromise, b2Promise]);
if (response.ok) {
    event.waitUntil(logResponse(event, response, (Date.now() - reqStartTime)));
    return response;
}

The resulting stats prove that GCS is serving the majority of requests, and Cloudflare caches over 50% of those requests. The code for the logResponse function can be found here.  

Tracking request time surfaced another issue. Requests to B2 from countries outside of North America are still quite slow. Cloudflare's Argo can reduce latency by over 50%, but is too expensive to enable for the whole site. Additionally, it would be redundant to smart-route content from GCS that Google already does an excellent job of keeping latency down. Cloudflare request headers include the country of origin, making it trivial to route this subset of requests through an Argo-enabled domain.

// Use CF Argo for non-US/CA users
function getB2Url(request) {
    let b2BackendUrl = 'https://b2.my-argo-enabled-domain.com/file';
    let country = request.headers.get('CF-IPCountry')
    if (country === 'US' || country === 'CA') {
        b2BackendUrl = 'https://f001.backblazeb2.com/file';
    }
    return b2BackendUrl;
}

Cloudflare Workers are an excellent fit for my project; they enabled me to make a cost-effective solution to hosting Leaflet maps at scale. Check out https://factoriomaps.com for performant Leaflet maps, and if you play Factorio, submit your Factorio world to share with others!