Ulaanbaatar, Mongolia

Whenever you get into a conversation about exotic travel or ponder visiting the four corners of the globe, inevitably you end up discussing Ulaanbaatar in Mongolia. Travelers want to experience the rich culture and vivid blue skies of Mongolia; a feature which gives the country its nickname of “Land of the Eternal Blue Sky”.

Ulaanbaatar (or Ulan Bator; but shortened to UB by many) is the capital of Mongolia and located nearly a mile above sea level just outside the Gobi Desert - a desert that spans a good percentage of Central Asia’s Mongolia. (The rest of the Gobi Desert extends into China). The country is nestled squarely between Russia to the north and China to the south. It’s also home to some of the richest and ancient customs and festivals around. It’s those festivals that successfully draw in the tourists who want to experience something quite unique. Luckily, even with all the tourists, Mongolia has managed to keep its local customs; both in the cities and within its nomadic tribes.

via Wikipedia

History also has drawn explorers and conquerors to and from the region; but more on that later.

Any avid reader of our blogs will know that we frequently explain that the expansion of our network provides customers and end-users with both more capacity and less latency. That goal (covering 95% of the Internet population with 10 milliseconds or less of latency) means that Mongolia was seriously on our radar.

Now we have a data center in Ulaanbaatar, Mongolia, latency into that blue sky country is significantly reduced. Prior to this data center going live we were shipping bits into the country via Hong Kong, a whopping 1,800 miles away (or 50 milliseconds if we talk latency). That's far! We know this new data center is a win-win for both mobile and broadband customers within the country and for Cloudflare customers as a whole.

Ulaanbaatar is city number 154 on Cloudflare’s network. Our expansion into cities like Ulaanbaatar doesn’t just happen instantly; it takes many teams within Cloudflare in order to successfully deploy in a place like this.

However, before deploying, we need to negotiate a place to deploy into. A new city requires a secure data center for us to build into. A bandwidth partner is also required. We need to get access to the local networks and to also acquire cache-fill bandwidth in order to operate our CDN. Once we have those items negotiated, we can focus on the next steps. Any site we build has to match our own stringent security standards (we are PCI/DSS compliant – hence all our data centers need to also be PCI/DSS compliant). That’s a paperwork process, which surprisingly takes longer than most other stages (because we care about security).

Then logistics kicks in. A BOM (Bill of Materials) is created. Serial numbers recorded. Power plugs chosen. Fun fact: Cloudflare data centers are nearly all identical, except the power cables. While we live in a world where fiber optic cables and connectors are universal, independent of location (or speed in some cases), the power connections we receive for our data centers vary widely as we expand around the globe.

The actual shipping is possibly the more interesting part of the logistics process. Getting a few pallets of hardware strapped up and ready to move is only a small part of the process. Paperwork again becomes the all-powerful issue. Each country has its own customs and import process, each shipping company has its own twist on how to process things, and each shipment needs to be coordinated with a receiving party. Our logistics team pulls off this miracle for new sites, upgrades for existing sites, replacement parts for existing sites, all while sometimes calmly listening to mundane on-hold music from around the globe.

Then our hardware arrives! Seriously, this is a biggie and those around the office that follow these new city builds are always celebrating on those days. The logistics team has done their part; now it’s time for the deployment team to kick-in.

The deployment team’s main goal is to get hardware racked-and-stacked in a site where (in most cases) we are contracting out the remote-hands to do this work. Sometimes we send people to a site to build it; however, that’s not a scalable process and hence we use local remote-hands contractors to do this heavy-lifting and cabling. There are diagrams, there are instructions, there are color-coded cables (‘cause the right cable needs to go into the right socket). Depending on the size of the build; it can be just a days work or up-to a weeks worth of work. We vary our data center sizes based on the capacity needs for each city. Once racked-and-stacked there is one more job to get done within the Infrastructure team. They get the initial private network connection enabled and secured. That single connection provides us with the ability to continue to the next step. Actually setting up the network and servers at the new site.

Every new data center site is shipped with zero configuration loaded into network hardware and compute servers. They all ship raw with no Cloudflare knowledge embedded into them. This is by design. The network teams first goal is to configure the routers and switches, which is mainly a bootstrap process in order for the hardware to phone-home and securely request its full configuration setup. We have previously written about our extensive network automation methods. In the case of a new site, it’s not that different. Once the site can communicate back home, it’s clear to the software base that its configuration is out of date. Updates are pushed to the site and network monitoring is automatically enabled.

But let's not paint a perfect rosy picture. There can still be networking issues. Just one is worth pointing-out as it’s a recurring issue and one that plagues the industry globally. Fiber optic cables can sometimes be plugged in with their receive and transmit sides reversed. It’s a 50:50 chance of being right. Sometimes it just amazes us that we can’t get this fixed; but … a quick swap of the two connectors and we are back in business!

No conversation about Mongolia would be valid unless we discuss Genghis Khan. Back in the 13th century, Genghis Khan founded the Mongol Empire. He unified the tribes of what is now Mongolia (and beyond). He established the largest land empire in history and is well described both online and via various TV documentaries (The History Channel doesn’t skimp when it comes to covering him). Genghis Khan was a name of honor that he didn’t receive till 1206. Before that he was just named “Temujin”.

Photo by SarahTz CC by/2.0

Around 30 miles outside Ulaanbaatar is the equestrian statue of Genghis Khan on the banks of the Tuul River in Gorkhi Terelj National Park. Pictured above, this statue is 131 feet tall and built from 250 tons of steel.

We get to announce our new city (and country) data center during a very special time. The Golden Eagle Festival takes place during the first weekend of October (that’s October 6 and 7 this year). It’s a test of a hunter’s speed and agility. In this case the hunters (nomads of Mongolia) are practicing an ancient technique of using Golden Eagles to hunt. It takes place in the far west of the country in the Altai Mountains.

The most famous festival in Mongolia is the Naadam Festival in mid-July. So many things going on within that festival, all of which is a celebration of Mongolian and nomadic culture. The festival celebrates the local sports (wrestling, archery, and horse racing) along with plenty of arts. The opening ceremony can be quite elaborate.

When discussing Mongolia, travelers nearly always want to make sure their itineraries overlap at least one of these festivals!

Last week was Cloudflare’s Birthday Week and we announced many services and products. Rest-assured that everything we announced is instantly available in Ulaanbaatar, Mongolia (data center city 154) just like it’s available everywhere else on Cloudflare’s network.

One final trivia point regarding our Ulaanbaatar data center. With Ulaanbaatar live, we now have datacenters covering the letters A thru Z (from Amsterdam to Zurich), i.e. with U added, every letter is now fully represented.

If you like what you’ve read and want to come join our infrastructure team, our logistics team, our network team, our SRE team, or any of the teams that help with these global expansions, then we would love to see your resume or CV. Look here for job opening.