Each day at CloudFlare is full of surprises.
As it turns out, it takes a lot of work to stop massive attacks and to help make the web faster. Over the past six months, our entire team has contributed in every way imaginable to more than double the capacity of our global network. Below is a behind-the-scenes look into how we keep our global network running.
Along the way we’ve encountered many surprises—some fun and some cringe worthy—that have taught us about our team, our data centers and overcoming challenges that occasionally seem beyond our control.
CloudFlare team: always online
San Jose, US (SJC): Our counsel, Ken, is great at pumpkin carving, and even better at standing up to protect the privacy rights of our users (including against Kanye West’s army of lawyers). What you may not have known is that he is happiest in the data center (not to mention our cabling was much prettier when he finished!).
Ken, our counsel
Los Angeles, US (LAX): Our engineers monitor our network around the clock. Occasionally this means juggling multiple tasks. During our most recent upgrade, Joshua (Systems Reliability Engineer & super dad) managed to snatch a moment to put his kids to bed while managing simultaneous upgrades in Los Angeles and Stockholm.
Chicago, US (ORD): Just before our Chicago upgrade we learned that our carrier had misplaced a shipment of memory. Fortunately, Nitin (Special Projects) averted disaster and got the courier to radio the driver (it took some convincing!), find and grab our DIMMs, and get the install done in time. When most people think of Chicago they think of the ‘95-96 Chicago Bulls (arguably the greatest basketball team to step foot on this earth). When we think of Chicago, our minds turn to the bullwhip effect. We precisely plan every data center launch and upgrade throughout the entire supply chain—from cables to servers to shipping schedules—to control against situations where precautions amplify errors.
Dallas, US (DFW): At CloudFlare, stamina is key. Trey (Solution Engineer) experienced this first hand. Despite working through the night on our Dallas upgrade, he still managed to catch a 6:30 AM flight the following morning to San Antonio, where he ran a workshop for our friends at Rackspace.
Ashburn, US (IAD): An important customer meeting the following morning didn’t stop Matthew (CEO) and Trey (Solution Engineer) from working through the night to upgrade our Ashburn facility. Trey even realized he could use his toenail clippers to save time cutting zipties and keep the the install moving.
Home is where the datacenter is
Hong Kong, HK (HKG): After speaking at an Internet security conference in China, Joshua (Special Projects Lead) spent three consecutive nights upgrading our Seoul, Tokyo and Hong Kong data centers. In addition to the data center, he found airport lounges and taxis to be equally habitable.
London, GB (LHR): CloudFlare’s first international office opened in London in 2013. Since then we’ve enlisted an amazing team of engineers to keep our network humming 24x7x365. On more than a few occasions, James (Systems Reliability Engineer), Marty (Support Engineer) and Simon (Support Engineer) have found a warm room full of servers in our London facility a comfort on a bitter winter night.
Stockholm, SE (ARN): Simon (Support Engineer) managed to navigate to the local Kjell to pick up a few needed power adapters, and then braved the 1° C cold outside of our Stockholm data center for a bit longer than he would have liked while waiting for an access card.
Miami, US (MIA): We take security seriously, and so do our data center partners. Justin (Systems Reliability Engineer) was at least a little intimidated to find guards armed with machine guns protecting the entrance of our Miami facility.
Tokyo, JP (NRT): We love our data centers so much that we even name our conference rooms after them. As our San Francisco office expands (we’ve now knocked down two walls!), we’ve named the latest NRT.
New challenges
Atlanta, US (ATL): Each of our racks around the world are fitted with high tech PDUs (power distribution units) that allow for remote monitoring and power cycling. This allows us to monitor our infrastructure in real-time, and react at a moment’s notice. When our colocation provider in Atlanta told us that our PDU wouldn’t fit, giving up wasn’t an option. Joshua (Special Projects Lead) proposed a rack extender to do the trick!
Paris, FR (CDG): Imagine being told that equipment you had just shipped 5,000 miles across the globe was about to be sent right back. Jérôme (Network Engineer), one of our resident French speakers, saved the day and made sure our equipment stayed right where it belonged: working hard in support of one of our busiest datacenters. Merci Jérôme!
Seattle, US (SEA): We install console servers with out-of-band, cellular Internet access in each of our data centers to remotely manage our infrastructure in the case our primary Internet connectivity is lost. While this makes it easier to address connectivity issues, installing the equipment itself can occasionally be more difficult. With a SIM card stubbornly lodged into our console server in Seattle, Jerome (Partner Engineer) used what he had available—namely, dental floss and a pair of forceps—to get the job done in a way that even MacGyver would approve.
Seoul, KR (ICN): Korea ranks near the top of most challenging locations to import equipment into (right up there with Warsaw). Fortunately, having facilitated hundreds of shipments in dozens of countries, Nitin (Special Projects) was able to break through a two month logjam in which Incheon airport became a temporary home for some of our equipment. Among his other talents, Nitin can now hum the FedEx and DHL songs in nearly any language of your choice.
Valparaíso, CL (SCL): Sometimes events are just out of one’s control. To launch our newest data center we had to wait through multiple customs strikes before equipment could arrive. Still, the show went on. Tom (Network Engineer) worked through Christmas to shave 170ms off of latency for our users in Latin America.
What’s in store for 2014, you ask? Over the next 12 months we will significantly expand our data center footprint, adding facilities in regions we currently lack coverage: Latin America, the Middle East, Africa, and parts of Asia. China is our second largest market, Brazil is our third: in 2014 we’ll be significantly expanding our network to better serve these customers.
If this sounds like fun, and if you enjoy a few surprises every once in a while, please consider joining us. We’re actively recruiting for someone passionate and talented to assist with our expansion.