This is a guest post from Rodney Gibbs. Rodney is the CIO of The Texas Tribune, a nonprofit media organization that covers public policy, politics, and government. He and his team recently supported major livestreamed events at South by Southwest (SXSW), a conference that attracts more than 70,000 music, arts and digital media aficionados.
A few days before the start of the SXSW, I got a call from Hugh Forrest, the head of interactive programming at the festival. The Texas Tribune has worked with SXSW organizers on previous events such as SXSW Edu. Our organization was also recently in the news for livestreaming the Texas state filibuster that lasted 11 hours, gaining more than 200,000 concurrent viewers from 187 countries around the world.
Hugh was looking for help with a couple of events that were sure to be hugely popular - livestreamed interviews of two major speakers - Julian Assange and Edward Snowden. Julian Assange is the founder of WikiLeaks and has been living in the Ecuadorian embassy in London since 2012. Edward Snowden is the NSA contractor who released classified files detailing the US spy agency’s internet-tapping programs around the world. Hugh wanted to work with a news organization to livestream the SXSW interviews so he turned to The Texas Tribune.
Don’t mess with Texas
On Thursday before the festival, we put together a plan to help us prepare for a super high-level of traffic and being on an international stage. We assembled a team, including our director of technology, Amanda Krauss, who listed all the things we’d need to do to make sure our website was ready to handle a crush of web visitors. More importantly, we wanted to tighten up security and make it difficult for any malicious actors to mess with us.
Harper Reed is an advisor to our organization. In a past life, he was the CTO for Obama for America, managing the website for Obama 2012. We asked him to look over our list and see if he would add anything. He made two suggestions - one of which was adding CloudFlare. It was Friday night at 4pm - about 16 hours before the Assange interview, when I reached out to CloudFlare’s CEO Matthew Prince via LinkedIn. We got a call back soon thereafter and our team spent the rest of the night setting up and testing CloudFlare on our website.
Our hard work paid off - the Assange livestream went off smoothly.
Be your own forcing function - don’t assume you’re not going to get hacked
We had a lot of loose ends on our to-do list in terms of security, scaling, and elasticity - for us it was good to have these livestreamed interviews as a forcing function. But we were lucky - we had Harper who had learned the hard way and generously shared what he learned so that it went smoothly for us.
It seems like the going rule of thumb nowadays is ‘Don’t assume you’re not going to get hacked, just assume you need to figure out what you’re going to do about it.’ The hardest thing is to learn something under pressure. We got our security measures set up pretty quickly but if I could offer advice I would suggest doing the following:
- Implement 2-factor authentication for your third party apps
- Enforce strong passwords for your CMS and apps users, making sure people don’t use the same password elsewhere
- Close down ports you don’t need - starting, for example, by closing down everything but ports 80 and 443
- If you're worried you'll come under attack, put up a static site as your main site, disconnecting it from any back-end databases. A hacked database is irrelevant if your static site is still up.
Finally, we host our website on Amazon’s S3 but thought it would be good to put CloudFlare in front of it to defend against DDoS or other possible web threats and also speed up any content that wasn’t on Amazon. If you implement CloudFlare, the other piece of advice I’d offer is make sure you don’t mistakenly reveal your origin IPs externally (or ‘leak’ them).
A 10X spike in traffic with viewers from 200+ countries
On the day of the Snowden interview, The Texas Tribune saw a 10X spike in traffic. Typically 90% of our traffic is Texas, with the rest coming from New York, Chicago, and Washington DC. This time more than 80% of our visitors were new to the site, hailing from over 200 countries. Visitors from the Netherlands totalled more than half of what we saw from the US and the top country list also included Belgium and Mexico in addition to your usual Western European suspects. It felt good to be able to serve international traffic.
We also got a few emails thanking us. One from Brazil said, ‘I caught the end of the telecast and am just so impressed with what you've done and the technology that permitted me to see it! Thank you is not enough to say for your organization!’ That’s the kind of thanks that makes working late nights worthwhile.
We had an incredibly smooth experience, no downtime, no lagging, no security problems - it all went super smoothly. A lot of folks pitched in to make these events happen. I tweeted out my thanks and it bears repeating here.
Big thx to @harper @maycotte @clockwerks @CloudFlare @Chartbeat & @livestream for help w/ @texastribune's #Snowden #SXSW stream. — Rodney Gibbs (@rgibbs) March 10, 2014
I hope you can learn from our experiences and your next major event goes as smoothly as ours did.