CloudFlare Blog

Mirage 2.0: Solving the Mobile Browsing Speed Challenge

Matthew Prince — Thu, 13 Jun 2013 04:00:00 -0700

Almost exactly a year ago, CloudFlare announced a feature called Mirage. Mirage was designed to make the loading of images faster in two primary ways: 1) deliver smaller images for devices with smaller screens; and 2) "lazy load" images only when they appeared in the viewport. Both of these optimizations were designed primarily to accelerate web performance on mobile devices.

Mobile devices present a number of challenges to delivering fast web performance. Because they rely on radio networks, the bandwidth to a mobile phone or tablet is often slow. However, the problem isn't limited to just slow bandwidth. Mobile connections are much more likely to experience "loss." To optimize for mobile performance you need to prioritize the most important data and download it first and you need to minimize the number of individual connections in order to limit the impact of packet loss.

The first version of Mirage was designed to accomplish these goals, but it was relatively naive in the way that it did it. We would store multiple versions of images, which make up the bulk of the data transferred for most websites, and then attempt to deliver the one that best matched the screen size. The problem was that the new versions of the images often weren't perfectly matched for the layout of the page or the size of the screen, especially if the page relied on the image's actual dimensions rather than including dimensions in the tag.

For the last year, we've studied sites using Mirage and taken what we've learned to refine and improve every aspect of the feature. Today we're excited to announce Mirage 2.0 which is designed from the ground up to solve the mobile browsing speed challenge.

Virtualized Images

Mirage 2.0 starts with the idea of image virtualization. When CloudFlare caches an image on our network for a site with Mirage 2.0 enabled, we store two versions. The first version is the full-resolution image, the second is a virtualized image that includes meta data about all the full-resolution image's dimensions but with the image itself is massively reduced in size. The reduced sized version typically as little as 1% the size of the full-resolution image.

If you enable Mirage 2.0, CloudFlare's network modifies the image tags on your page on the fly so they can be loaded by the Virtualized Image Packager ("VIP"). In parallel with the HTML of your page loading, the Mirage 2.0 VIP begins downloading the virtualized images that appear on the page. The VIP will virtualize images served from your own domain as well as images served from third party domains (e.g., Flickr or Imgur). Because the virtualized images have the full-resolution image's dimensions embedded as meta data, the VIP is able to place the images into the browser's DOM correctly sized so the browser can almost immediately begin the process of rendering the page.

Minimizing Requests

Rather than initiating a new request for each image, the VIP is able to stream all the images from CloudFlare's network with a single request. This uses the same mechanism we created for Rocket Loader, our Javascript performance accelerator. This means that even a page with hundreds of images can begin rendering in the browser with as few as two requests. Even users on slow mobile connections can begin interacting with the page immediately, rather than having to wait for all the full-resolution images to load.

After the page is rendered with the virtualized images, the VIP begins to replace them with the full-resolution versions. Since the images are already correctly sized for their tags on the page, the browser does not need to reflow the page as the full-resolution versions are loaded. The VIP prioritizes what full-resolution images to load first based on what images are in the browser's viewport. Visually, images appear to "rez" in, starting as low quality and then coming into sharp focus, similar to how a progressive JPEGs load in a browser.

While you can enable CloudFlare features such as Polish in order to optimize your images, by default Mirage 2.0 does not transcode or otherwise alter the original full-resolution images. The VIP will pull third party content directly from the original servers without passing through CloudFlare's network -- unless, of course, the third part is also using CloudFlare.

Learning Loader

With Mirage 2.0, we've also completely rethought how we detect different browsers and respond to their capabilities. Mirage 2.0 is optimized to be more or less aggressive depending on the capabilities of the browser as well as its connection to the Internet. An iPhone connecting to the web over a wifi network is optimized for different loading priorities than the same device connecting over a cellular network. We even detect the different download speeds of cellular networks from LTE to 3G to Edge and optimize for each connection speed appropriately.

Mirage 2.0 gathers real browsing intelligence from all its connections which we then use to further optimize the VIP's performance. As more sites enable Mirage 2.0 the CloudFlare's systems automatically begins to optimize for the fastest possible browsing experience from any device on any network. In other words, the same way we use data about security threats in order to protect the sites on our network, we are now using data about real user's browsers around the world in order to ensure everyone on the CloudFlare network has the fastest possible site.

Reviews Are In

We've been testing Mirage 2.0 on some of our most image heavy sites that get significant traffic from mobile browsers. The reaction has been terrific: "As one of the largest image sharing sites in the world, speed has always been really important to us," explained Alan Schaaf, founder and CEO of Imgur. "We've invested a lot of time into getting images to load as fast as possible over mobile networks, especially since we've been developing our mobile app, and we've seen great improvements with Mirage 2.0. We're really happy that CloudFlare continues to launch innovative products to ensure pages on Imgur.com load as fast as possible."

You can see Mirage 2.0 in action for yourself in the following video:

Available Now

Mirage 2.0 is currently in beta and will be made available over the next few weeks to all paid CloudFlare accounts, including our lowest level PRO accounts which are priced at only $20/month. Mirage 2.0 will fully replace the original version of Mirage in the following months and users with the old Mirage enabled will be upgraded to the newer, better version. Given the importance of mobile browsing, and the massive performance benefit Mirage 2.0 delivers with a single click, we think it is one of the most compelling features we've ever offered. Give it a try and let us know what you think.

CloudFlare will be at HostingCon 2013 in Full Force

Kristin Tarr — Wed, 12 Jun 2013 10:00:00 -0700

The CloudFlare team will be at HostingCon 2013 in Austin next week. This is our third year at the show and we have a lot of things in store for partners.

Here's a sneak peek:

Complimentary limousine transfers from Austin-Bergstrom International Airport to the Hilton Austin hotel on Sunday, June 16th. Reserve your spot today!
New CloudFlare tshirts
Live music to supercharge your day during breakfast each morning
Charging stations at our booth (#523) to keep your devices supercharged
Bigger and better Nerf Railguns. There is limited quantity, so be sure to visit us at booth #523 to get your Railgun

CloudFlare Railguns ready for HostingCon 2013

We are looking forward to connecting with our current partners and meeting new partners at the show. If you are already a CloudFlare Certified Partner, be sure to stop by and introduce yourself. If you are not a partner yet, stop by to learn more about how CloudFlare can reduce your server load, improve the performance of your network, block spammers, botnets and other web threats, and provide DDOS protection. More details about the CloudFlare Certified Partner program here.

Here's where the CloudFlare team will be all week:

Sunday, June 16th

Limo transfers from Austin-Bergstrom International Airport to the Hilton Austin Hotel. Registration is still open, reserve your spot now!

Monday, June 17th

7:45am-8:45am: CloudFlare sponsored breakfast located in the Level 4, Ballroom D Foyer - Live music by Alternator Jones
5:00pm onwards: Come find the CloudFlare team at the welcome reception!

Tuesday, June 18th

7:45am-8:45am: CloudFlare sponsored breakfast located in the Level 4, Ballroom D Foyer - Live music by Jackie Venson
12:00pm-4:00pm: CloudFlare is in Exhibit Hall 4 at booth #523
4:00pm-6:30pm: Visit our booth during the exhibit hall happy hour for a beverage and to supercharge your mobile phone!

Wednesday, July 18th

8:00am-10:00am: CloudFlare sponsored breakfast located in the Level 4, Ballroom D Foyer - Live music by Sean Evan
9:00am-9:45am: Our co-founder and CEO Matthew Prince will be speaking on the IPv6 panel discussion "Now is the Time for IPv6" in room #18D
9:00-9:45am: Maria Karaivanova and John Roberts from CloudFlare will be co-hosting a talk on partnerships, "Strategies for Successful Partnerships" in room #16
12:00-4:00pm: CloudFlare is in Exhibit Hall 4 at Booth #523

Connect with us on Twitter during the event to find out where we are and what's coming up next:

#hostingcon, @hostingcon, @CloudFlare

See you in Austin!

CloudFlare, PRISM, and Securing SSL Ciphers

Matthew Prince — Wed, 12 Jun 2013 02:00:00 -0700

Over the last week we've closely watched the disclosures about the alleged NSA PRISM program. At CloudFlare, we have never been approached to participate in PRISM or any other similar program. We do, from time to time, receive subpoenas and court orders. A human being on our team reviews each of these requests manually. When we determine that a request is too broad, we push back to limit the scope of the request. Whenever possible, we disclose to all affected customers the fact that we have received a subpoena or court order and allow them an opportunity to challenge it before we respond.

One of the ways we limit the scope of orders we receive is by limiting the data we store. I have written before about how CloudFlare limits what we log and purge most log data within a few hours. For example, we cannot disclose the visitors to a particular website on CloudFlare because we do not currently store that data.

To date, CloudFlare has never received an order from the Foreign Intelligence Surveillance Act (FISA) court. Moreover, we believe that due process requires court review of executive orders. As a policy, we challenge any orders that have not been reviewed and approved by a court. As part of these challenges, we always request the right to disclose at least the fact that we received such an order but we are not always granted that request.

While we understand the need for secrecy in some investigations, we are troubled when laws limit our ability to acknowledge that we have even received certain kinds of requests. CloudFlare fully supports the calls for transparency today by other web companies like Google, Microsoft, and Facebook. At a minimum, we request the law be updated to allow companies to disclose the number of FISA orders and National Security Letters (NSLs) they have received. We believe this is a modest request which does not harm the integrity of legitimate investigations while allowing for an informed public debate over the use of these measures.

As we set policy, one of our guiding principles is that we should neither make the job of law enforcement easier, nor should we make it harder, than it would have been if CloudFlare did not exist. If the NSA is listening in on any transactions traversing our network, they are not doing so with our blessing, consent, or knowledge.

Making Sense of PRISM

As we've followed the PRISM story, we've tried to reconcile how the PRISM slides could be accurate while so many tech executives have denied participation in the program. One theory that surfaced was that the NSA had broken the private SSL keys of a select number of web giants. Our theory was that this could explain how companies were added over time -- as their private SSL keys were cracked -- while their executives wouldn't have any knowledge of what was happening.

Even the name of the program -- "PRISM" -- led credence to this theory. Prisms are often used with fiber optic cables in order to split the light the cables carry into multiple copies. This is not new technology. In 2006 in Room 641a of a data center in San Francisco, AT&T installed a beam splitter to siphon traffic from their optical network, reportedly at the request of the NSA.

SSL should protect these communications. However, with most SSL ciphers, the private key remains the same for all sessions. As a result, if the NSA were to record encrypted traffic, they could later break the SSL key used to secure the traffic and then use the broken key to decrypt what they previously recorded.

Breaking SSL

Breaking a SSL key is hard, but not impossible. Doing so is just a matter of computational power and time. For example, it is known that using a 2009-era PC cranking away for about 73 days you can reverse engineer a 512-bit key. Each bit in a key's length doubles the effective computational power needed to break the key. So, if the key were 513 bits long, you'd expect the same modest PC 132 days to break the key. These tasks are highly parallelizable, so if you have two modest PCs then you'd expect the time to break the 513-bit key to drop down to 66 days again. (Note: this assumes a naive factorization algorithm. The state of the art is to use a generalized number field sieve. This reduces the rate of complexity growth to something that is sub-exponential. This means if you know what you're doing the problem doesn't double in difficulty with each additional bit.)

It is not inconceivable that the NSA has data centers full of specialized hardware optimized for SSL key breaking. According to data shared with us from a survey of SSL keys used by various websites, the majority of web companies were using 1024-bit SSL ciphers and RSA-based encryption through 2012. Given enough specialized hardware, it is within the realm of possibility that the NSA could within a reasonable period of time reverse engineer 1024-bit SSL keys for certain web companies. If they'd been recording the traffic to these web companies, they could then use the broken key to go back and decrypt all the transactions.

While this seems like a compelling theory, ultimately, we remain skeptical this is how the PRISM program described in the slides actually works. Cracking 1024-bit keys would be a big deal and likely involve some cutting-edge cryptography and computational power, even for the NSA. The largest SSL key that is known to have been broken to date is 768 bits long. While that was 4 years ago, and the NSA undoubtedly has some of the best cryptographers in the world, it's still a considerable distance from 768 bits to 1024 bits -- especially given the slide suggests Microsoft's key would have to had been broken back in 2007.

Moreover, the slide showing the dates on which "collection began" for various companies also puts the cost of the program at $20M/year. That may sound like a lot of money, but it is not for an undertaking like this. Just the power necessary to run the server farm needed to break a 1024-bit key would likely cost in excess of $20M/year. While the NSA may have broken 1024-bit SSL keys as part of some other program, if the slide is accurate and complete, we think it's highly unlikely they did so as part of the PRISM program. A not particularly glamorous alternative theory is that the NSA didn't break the SSL key but instead just cajoled rogue employees at firms with access to the private keys -- whether the companies themselves, partners they'd shared the keys with, or the certificate authorities who issued the keys in the first place -- to turn them over. That very well may be possible on a budget of $20M/year.

Making SSL More Secure

Today many web companies have largely transitioned from 1024-bit SSL to significantly stronger 2048-bit keys. (Remember, for a naive algorithm, each bit doubles the time it takes to break the key, so a 2048-bit key isn't twice as strong, it is 2^1024 times as strong.) Based on the SSL survey data, Twitter has led the way, shifting 100 percent of its HTTPS traffic to a 2048-bit key in mid-2010. By the end of 2012, the following websites had approximately the amount of requests in the parenthesis shifted to 2048-bit SSL:

outlook.com (100%)
microsoft.com (98%)
live.com (90%)
skype.com (88%)
apple.com (85%)
yahoo.com (82%)
bing.com (79%)
hotmail.com (33%)

Facebook is the laggard of the bunch and today is still using a 1024-bit key for all HTTPS requests.

Google is a notable anomaly. The company uses a 1024-bit key, but, unlike all the other companies listed above, rather than using a default cipher suite based on the RSA encryption algorithm, they instead prefer the Elliptic Curve Diffie-Hellman Ephemeral (ECDHE) cipher suites. Without going into the technical details, a key difference of ECDHE is that they use a different private key for each user's session. This means that if the NSA, or anyone else, is recording encrypted traffic, they cannot break one private key and read all historical transactions with Google. The NSA would have to break the private key generated for each session, which, in Google's case, is unique to each user and regenerated for each user at least every 28-hours.

While ECDHE arguably already puts Google at the head of the pack for web transaction security, to further augment security Google has publicly announced that they will be increasing their key length to 2048-bit by the end of 2013. Assuming the company continues to prefer the ECDHE cipher suites, this will put Google at the cutting edge of web transaction security.

SSL on CloudFlare

There is good news in all of this. If you're using SSL on CloudFlare, your site is already at this cutting edge. We issue 2048-bit keys by default and prefer the ECDHE cipher suites. Today, most modern browsers running on up-to-date operating systems will support ECDHE. In our tests, approximately two thirds of HTTPS requests to our network support ECDHE. The remaining traffic quietly falls back on a more standard cipher suite without the visitor noticing.

Looking Ahead

Ultimately, CloudFlare's value proposition is built on trust. Core to that trust is ensuring transactions passing through our network are fundamentally secure. We will continue to work on both policy and technology to ensure the security and integrity of our network.

PRISM has sparked a conversation on privacy and transparency broadly -- among citizens, between companies, and with our governments. At CloudFlare, we are actively engaged in this conversation at many levels. Our mission is to build a better web and we believe privacy and transparency are critical to its foundation.

Happy IPv6 Day: Usage On the Rise, Attacks Too

Matthew Prince — Thu, 06 Jun 2013 08:00:00 -0700

June 6th is known as World IPv6 Day so we thought it was a good time to look at the trends in IPv6 usage across CloudFlare's network. Two big themes we've seen: 1) IPv6 usage is growing steadily, but at the current pace we're still going to be living with IPv4 for many years to come; and 2) while the majority of IPv6 traffic comes from legitimate users on mobile networks, attackers too are beginning to launch attacks over the protocol.

IPv6 Growth

CloudFlare has supported IPv6 on our network for the last year and a half. We have become one of the largest providers of the IPv6 web because we offer a free IPv6 gateway that allows any website to be available over IPv6 even if a site's origin network doesn't yet support the protocol. For the last year, we've enabled IPv6 for customers on CloudFlare by default. Today, IPv6 is enabled for more than 1 million of our customers' websites.

Since the beginning of 2013, IPv6 connections as a percentage of CloudFlare's total traffic fluctuate daily with the minimum 0.849% on January 5 to a maximum of 1.645% on June 3, 2013. If look at the overall trend, IPv6 connections to our network have grown 26.5% since the start of the year.

Digging into where IPv6 connections are coming from it appears the majority of the growth has been from mobile network providers. Increasingly, traffic from mobile devices to the web has passed over IPv6. We saw a significant drop in IPv6 connection from mid-March through early-April when it appears a large mobile operator appears to have disabled and then reenabled IPv6 connectivity from their network.

While the overall increase in IPv6 usage is encouraging, the trend unfortunately indicates we are going to be living with IPv4 for some time to come. At current growth rates, assuming adoption of IPv6 is linear, it will take almost 67 years for IPv6 connections to surpass IPv4 connections and the last IPv4 connection won't be retired until May 10, 2148.

Things are a bit more optimistic if IPv6 adoption turns out to be exponential rather than linear. In that case, IPv6 connections will surpass IPv4 in about 5 years and 9 months. Not long thereafter, we'll extinguish IPv4 entirely on January 10, 2020. Our guess is the reality will be somewhere between the linear and exponential case. Regardless of what IPv6's adoption curve looks like, as a CloudFlare user you're covered. We anticipate we will be operating a dual-stack network with both IPv4 and IPv6 support for all our customers until IPv4 is fully retired, whether that takes 7 years or 140.

IPv6 Attacks

While the majority of IPv6 connections today are coming from legitimate users on mobile networks, over the last two months we've seen a marked increase in the number of IPv6-based web attacks. Largely these have been DDoS attacks. The attacks have typically been both Layer 4 (e.g., SYN floods) as well as Layer 7 (e.g., application layer attacks).

To date, the IPv6-based DDoS attacks have been relatively modest. The largest we've seen to date generated approximately 3 gigabits per second of traffic and accompanied a much larger traditional IPv4-based DDoS.

While a novelty, these attacks don't cause significant harm to CloudFlare's systems. We designed CloudFlare anticipating the transition to IPv6, so our defenses assume an IPv6-enabled world. We speculate, however, that attackers may be targeting IPv6 as a way of bypassing older protections that base their protection largely on IPv4 blacklists.

IPv6 makes a strict blacklist on a per-IP basis much more challenging since the number of addresses available to an attacker can be significantly larger. This is a challenge that large blacklist operators like Spamhaus are currently thinking through. While IPv6 can present a challenge to some attack filtering strategies, it also presents opportunities. For example, since IPv6 reduces the need for NATs and provides users addresses that are routable all the way to the end device, we believe over time IPv6 will provide the ability to build significantly more accurate whitelists.

We will continue to monitor overall IPv6 growth rates as well as interesting trends in IPv6-based attacks. In the meantime, there's no better way to celebrate World IPv6 Day than signing up for CloudFlare and ensuring your site is automatically available for the increasing percentage of users that are accessing it over IPv6. It's free and will only take you 5 minutes to join the modern web.

Today's Network Issue

Matthew Prince — Fri, 31 May 2013 13:10:00 -0700

Today at 16:13 UTC a large amount of traffic began hitting our Los Angeles data center. We have an in-house team that monitors our network 24x7x365 and immediately all their alarms went off. We initially thought it was a very large attack. In fact, it was something much trickier to resolve.

Background

CloudFlare makes wide use of Anycast routing. This gives us a very large capacity to stop huge DDoS attacks. The challenge is managing the routing to ensure that traffic goes to the correct place.

CloudFlare buys bandwidth to connect to the Internet via what are known as transit providers. The first transit provider we used starting back in 2010 was a company called nLayer. They have been a terrific partner over the years.

In the last year, nLayer merged with GTT. Then, about a month ago, GTT/nLayer purchased Inteliquent (aka., TINET). Over the last few weeks, GTT/nLayer has been consolidating their network with Inteliquent's. When this is complete, GTT/nLayer will move from a Tier 2 network provider to one of the small handful of Tier 1 network providers.

Bumps

Today's issue was an indirect result of this migration. GTT/nLayer previously connected to Global Crossing, another large transit provider that is now owned by Level3. As part of the GTT/nLayer/Inteliquent consolidation, Level3 switched a route to being between Global Crossing and GTT/nLayer's route to instead be between Level3 and GTT/nLayer.

For most non-Anycasted traffic, this wouldn't cause any disruption. In our case, it shifted a large amount of traffic that would usually hit data centers on the east coast of the United States and Europe to all hit our facility in Los Angeles. In the worst case, this caused some machines in Los Angeles to overload, returning 502 Gateway Errors. Other visitors may have seen packet loss and slow connections as some links were saturated.

It wasn't immediately obvious what the cause of the issue was. We worked directly with GTT/nLayer's network team to rebalance traffic which temporarily put additional load on Seattle, then Dallas, then Chicago. While usually only customers nearby affected data centers would see an issue, in this case traffic as far away as Europe was landing in the wrong place.

Whether a visitor was affected or not was a bit of a crapshoot. We use multiple transit providers, so if your ISP wasn't connected to Level3 and you weren't naturally hitting an overloaded data center then you likely saw no problem. Overall, we estimate that around 10% of connections to our network were impacted for an approximately 20 minute window. A small percentage of users may have seen issues for a longer period of impact depending on their connection to Level3 and if they were pulled to more than one affected location.

Responsibility

Level3 or GTT/nLayer had no way of knowing how the changes they were making to their systems would affect us downstream.

While this was a very tricky situation for us to anticipate or even diagnose when it was happening, the responsibility lies with us to ensure our routing is getting people to the right locations and no facilities are overburdened. We've added this scenario to the conditions that we guard against so a similar change upstream should not affect us in the future.

The GTT/nLayer migration is scheduled to be completed today. One of the benefits of connecting to Tier 1 providers is route stability. While today's network issue was painful, I am encouraged that the underlying reason for the issue stems from an effort to build a more robust, stable, and reliable network.

Syrian Internet Restored

Matthew Prince — Wed, 08 May 2013 13:10:00 -0700

Yesterday, Syria's Internet connectivity was cut off from the rest of the world. At 14:12 UTC, approximately 19 hours and 30 minutes after it had been shut down, connectivity was returned. Here's a BGPlay video of routes being restored within the country.

Two interesting points. First, the government has stated that the outage was the result of a cable cut. Based on what we've seen, we believe this is highly unlikely. Syria's network connects to the rest of the Internet at four distinct points that are geographically separated. For traffic to be terminated entirely, all four connection points would need to be severed simultaneously.

Moreover, the video of the outage, as well as the video of the routes being restored, show the systematic withdrawl of BGP routes across all of Syria's providers. This is not the signature we see when there is an actual cable cut.

Second, while most of the Internet was cut off in Syria, it appears there was a small portion of Syrian IP space that continued to have connectivity. Specifically, the following IP ranges behind AS29256:

46.53.0.0/17
78.110.96.0/20
94.141.192.0/19

Those prefixes continued to be announced to Deutche Telecom which means they would have continued to have access to the Internet.

We don't know who is behind that IP space. We're still investigating whether we saw any Internet traffic coming from that IP space. The fact that they were still available, however, further discredits the assertation that this was a cable cut.

Here is a graph showing the last 24 hours of Syrian traffic to the CloudFlare network.

How Syria Turned Off the Internet (Again)

Matthew Prince — Tue, 07 May 2013 16:13:00 -0700

Today at 18:48 UTC, Syria dropped off the Internet. Based on the data we collect from our network, as well as reports from other organizations monitoring network routes, it appears that someone systematically withdrew the BGP (Border Gateway Protocol) routes from the country's border routers. This is the same technique that was used to withdraw Syrian Internet access last November.

The video below, which we generated using BGPlay, shows the routes in to the Syrian Internet being withdrawn:

The graph below shows the requests to CloudFlare's network from the Syrian Internet space over the last 6 hours (times are UTC):

We will continue to monitor Syrian traffic and post updates here if we see connectivity return.

Cribs: CloudFlare London Edition

John Graham-Cumming — Sun, 28 Apr 2013 22:28:00 -0700

It's official, CloudFlare has arrived in London.

CloudFlare's first international office opened this month near St. Paul's Cathedral in London. We decided to open an office outside Silicon Valley for two major reasons: to get access to high quality software engineering, network operations and technical support folks, and to expand our 24/7 operations and support. Not to mention, London has a vibrant start-up community that we are very happy to now be a part of.

London is 8 hours ahead of San Francisco making it the perfect location for a hand-over from a team arriving at work at 0900 in California (the London team is nearing the end of the day at 1700). By extending working hours a little it's easy to get 24 hour operations and support with just two offices.

Lobby of our new London office

And London's start-up community means there's a pool of talented people in engineering, operations and support for CloudFlare to hire from. Plus it enables CloudFlare to take part in the many meetups and user groups that flourish in and around Tech City some of which we sponsor.

We choose to be in the St. Paul's area because of its good public transport links to all parts of London and because the building we are in has everything from a restaurant, sports club to state of the art bicycle storage. Near by One New Change and the surrounding area are full of shops and eateries. There's also Smithfield Market and the Barbican.

CloudFlare London hard at work

And being in London brings us close to many of customers and partners such as GoSquared, Namecheap, and WebHostingBuzz.

We're actively hiring in both San Francisco and London. Check out our careers page.

W3TC and WP Super Cache Vulnerability Discovered, We've Automatically Patched

Matthew Prince — Thu, 25 Apr 2013 00:36:00 -0700

The team at the research firm Sucuri announced a serious vulnerability to W3TC and WP Super Cache this afternoon. (Update: it appears the vulnerability was first reported on WordPress.org about a month ago.) The vulnerability allows remote PHP code to be executed locally on a server for anyone running either of the two most popular WordPress caching plugins. This is a serious vulnerability as it could allow an attacker to execute code on your server.

Here are the versions of each plugin that are vulnerable:

W3 Total Cache (version 0.9.2.8 and below are vulnerable, version 0.9.2.9 and up are not vulnerable) / upgrade here
WP Super Cache (version 1.2 and below are vulnerable, version 1.3.x and up are not vulnerable) / upgrade here

As a precaution, CloudFlare has applied a rule to our network which protects against this specific vulnerability in both plugins. The protection is applied for all CloudFlare accounts automatically, even free accounts. You do not need to do anything to enable the protection.

Even with this protection in place, if you are running either of these plugins you should upgrade immediately (W3TC Upgrade / WP Super Cache Upgrade). The vulnerability is serious enough that we recommend you disable the plugins until you have completed an upgrade. If you're not already a CloudFlare customer, you can signup for free to get protection immediately.

Technical Details

The attack takes advantage of several functions in these plugins including: mfunc, mclude, and dynamic-cached-content. An attacker can execute a PHP command running on the server by pasting a comment to a WordPress blog running a vulnerable version of W3 Total Cache or WP Super Cache. For example, if you are running a vulnerable version of the plugins, the following will result in your current PHP version being printed in the comment:

<!--mfunc echo PHP_VERSION; --><!--/mfunc-->

While this is harmless, the same mfunc call in either plugin can run other arbitrary commands on your server. This could be used to gain access to the server, execute arbitrary database commands, or remotely install malware. Again, this is a very severe vulnerability and all W3TC and W3 Super Cache users should upgrade immediately (W3TC Upgrade / WP Super Cache Upgrade).

What CloudFlare Logs

Matthew Prince — Tue, 23 Apr 2013 17:24:00 -0700

Over the last few weeks, we've had a number of requests for information about what data CloudFlare logs when someone visits a site on our network. While we have provided a Privacy Policy that outlines how we keep information private, I wanted to take the time to clarify our customer log retention policies.

What CloudFlare Logs

When you visit a site on CloudFlare's network, we record information about that visit. If you run a web server you'll be familiar with these logs as they're similar to an Apache access log. We log data for two reasons: 1) to help us identify security threats and attacks hitting our customers in order to mitigate them; and 2) in order to identify performance bottlenecks and errors on our system.

It's somewhat hard to fathom the scale of the log data that we generate. Every minute of every day we generate more than 20GB (compressed) of log data. That translates, at our current volume, to more than 10 Petabytes of storage needed to store a year's worth of logs, and, due to our continued growth, that volume that has been doubling every 4 months or so. Today, even if we wanted to, we don't have the ability to retain all the logs we generate. This means that, for most customers, we discard access logs within 4 hours of them being recorded.

For our Enterprise customers, we offer an optional feature that allows them to export their raw log files in Apache format. This requires us to store log files for a longer period of time in order to allow them to be downloaded. By default, we store logs for these customers for 3 days.

Crunching Data

Since CloudFlare does not keep the raw logs, it is impossible for us to answer questions like: tell me all the visitors who have been to a particular website on CloudFlare's network.

However, CloudFlare does generate aggregate data, so we can provide analytics back to customers. We use the aggregated data to populate things like the CloudFlare Analytics page which includes numbers of hits, page views, bandwidth consumed and unique visitors. As logs are received, we run a stream processing engine that extracts this summary data. This data is correlated in each of our edge data centers and then sent to one of our core facilities in order to report through our UI.

This same data summary engine also looks for attack patterns, which is then used to provide security protection for our customer's websites. Using this engine, we can identify an attack on one site, usually in less than 1 minute, and then push updated security rules that then protect every site using CloudFlare from that same attack.

Access logs for most customers are stored briefly at the edge of our network and then deleted within 4 hours. If there is an error, those logs are transmitted back to one of our core facilities in order for us to diagnose the error. Error logs sent to core are currently kept for 1 week then discarded.

The Future

Going forward, we want to allow customers who would like to have more insight into the visitors to their sites to be able to choose to do so. As we do, we will provide details on how any feature we add changes our log retention policy, and we will continue to be guided by the principle that our customers should be able to understand and control what data is being stored about visitors to their sites.

Patching the Internet in Realtime: Fixing the Current WordPress Brute Force Attack

Matthew Prince — Thu, 11 Apr 2013 23:23:00 -0700

There is currently a significant attack being launched at a large number of WordPress blogs across the Internet. The attacker is brute force attacking the WordPress administrative portals, using the username "admin" and trying thousands of passwords. It appears a botnet is being used to launch the attack and more than tens of thousands of unique IP addresses have been recorded attempting to hack WordPress installs.

One of the concerns of an attack like this is that the attacker is using a relatively weak botnet of home PCs in order to build a much larger botnet of beefy servers in preparation for a future attack. These larger machines can cause much more damage in DDoS attacks because the servers have large network connections and are capable of generating significant amounts of traffic. This is a similar tactic that was used to build the so-called itsoknoproblembro/Brobot botnet which, in the Fall of 2012, was behind the large attacks on US financial institutions.

Patching the Internet

We just pushed a rule out through CloudFlare's WAF that detects the signature of the attack and stops it. Rather than limiting this to only paying customers, CloudFlare is rolling it out the fix to all our customers automatically, including customers on our free plan. If you are a WordPress user and you are using CloudFlare, you are now protected from this latest brute force attack.

Because CloudFlare sits in front of a significant portion of web requests we have the opportunity to, literally, patch Internet vulnerabilities in realtime. We will be providing information about the attack back to partners who are interested in hardening their internal defenses for customers who are not yet on CloudFlare.

If you are running a WordPress blog and want to ensure you are protected from this attack, you can sign up for CloudFlare's free plan and the protection is automatic. We'll continue to monitor the details of the attack and publish details about what we learn.

Continuing the Conversations from Parallels Summit 2013

Kristin Tarr — Thu, 11 Apr 2013 01:12:00 -0700

As we wrote about before, we attended Parallels Summit in February where we hosted "Conversations with CloudFlare" - live video interviews with industry experts.

Below are highlights and video clips from five of those conversations. Tune in to hear the latest in mobile services, how hosting providers are expanding across the world, and what these companies are looking forward to in 2013.

Elliot Noss, CEO of TuCows
*When it comes to mobile, TuCows is giving AT&T and Verizon a run for their money with Ting. Elliot explains the benefits of Ting, what's new with their mobile service and why wifi on your phone is all the rage... *

Merjin de Brabander, Business Manager at Hostnet
For 2013, Hostnet's application and POA program is really starting to take off. Listen in as Merjin discusses how they've grown and what new services they're bringing to their customers in 2013...

JT Smith, Owner of ViUX
*Providing an arrange of hosting services, ViUX is improving performance and reliability for their customers and considers themselves a complete Parallels shop. One area they pride themselves on? Customer service... *

Kris Anderson, Director of Support at Limestone Networks
*Limestone Networks is in the business of mission critical data. Their infrastructure supports some of the most reliable data centers around. Their support motto? Simple. Solid. Superior. *

Magnus Hult, Executive Vice President at Atomia
Atomia is making it easier than ever to run a business. They focus on the automation and billing platform for hosting providers so businesses can focus on what they do best. Hear what Atomia is working on for 2013...

How the CloudFlare Team Got Into Bondage (It's Not What You Think)

Matthew Prince — Mon, 08 Apr 2013 09:18:00 -0700

(Image courtesy of ferelswirl)

At CloudFlare, we're always looking for ways to eliminate bottlenecks. We're only able to deal with the very large amount of traffic that we handle, especially during large denial of service attacks, because we've built a network that can efficiently handle an extremely high volume of network requests. This post is about the nitty gritty of port bonding, one of the technologies we use, and how it allows us to get the maximum possible network throughput out of our servers.

Generation Three

A rack of equipment in CloudFlare's network has three core components: routers, switches, and servers. We own and install all our own equipment because it's impossible to have the flexibility and efficiency you need to do what we do running on someone else's gear. Over time, we've adjusted the specs of the gear we use based on the needs of our network and what we are able to cost effectively source from vendors.

Most of the equipment in our network today is based on our Generation 3 (G3) spec, which we deployed throughout 2012. Focusing just on the network connectivity for our G3 gear, our routers have multiple 10Gbps ports which connect out to the Internet as well as in to our switches. Our switches have a handful of 10Gbps ports that we use to connect to our routers and then 48 1Gbps ports that connect to the servers. Finally, our servers have 6 1Gbps ports, two on the motherboard (using Intel's chipset) and four on an Intel PCI network card. (There's an additional IPMI management port as well, but it doesn't figure into this discussion.)

To get high levels of utilization and keep our server spec consistent and flexible, each of the servers in our network can perform any of the key CloudFlare functions: DNS, front-line, caching, and logging. Cache, for example, is spread across multiple machines in a facility. This means if we add another drive to one of the servers in a data center, then the total available storage space for the cache increases for all the servers in that data center. What's good about this is that, as we need to, we can add more servers and linearly scale capacity across storage, CPU, and, in some applications, RAM. The challenge is that in order to pull this off there needs to be a significant amount of communication between servers across our local area network (LAN).

When we originally started deploying our G3 servers in early 2012, we treated each 1Gbps port on the switches and routers discretely. While each server could, in theory, handle 6Gbps of traffic, each port could only handle 1Gbps. Usually this was no big deal because we load balanced customers across multiple servers in multiple data centers so on no individual server port was a customer likely to burst over 1Gbps. However, we found that, from time to time, when a customer would come under attack, traffic to individual machines could exceed 1Gbps and overwhelm a port.

When A Problem Comes Along...

The goal of a denial of service attack is to find a bottleneck and then send enough garbage requests to fill it up and prevent legitimate requests from getting through. At the same time, our goal when mitigating such an attack is first to ensure the attack doesn't harm other customers and then to stop the attack from hurting the actual target.

For the most part, the biggest attacks by volume we see are Layer 3 attacks. In these, packets are stopped at the edge of our network and never reach our server infrastructure. As the very large attack against Spamhaus showed, we have a significant amount of network capacity at our edge and are therefore able to stop these Layer 3 attacks very effectively.

While the big Layer 3 attacks get the most attention, an attack doesn't need to be so large if it can affect another, narrower bottleneck. For example, switches and routers are largely blind to Layer 7 attacks, meaning our servers need to process the requests. That means the requests associated with the attack need to pass across the smaller, 1Gbps port on the server. From time to time, we've found that these attacks reached a large enough scale to overwhelm a 1Gbps port on one of our servers, making it a potential bottleneck.

Beyond raw bandwidth, the other bottleneck with some attacks centers on network interrupts. In most operating systems, every time a packet is received by a server, the network card generates an interrupt (known as an IRQ). An IRQ is effectively an instruction to the CPU to stop whatever else it's doing and deal with an event, in this case a packet arriving over the network. Each network adapter has multiple queues per port that receive these IRQs and then hands them to the server's CPU. The clock speed and driver efficiency in the network adapters, and
message passing rate of the bus, effectively sets the maximum number of interrupts per second, and therefore packets per second, a server's network interface can handle.

In certain attacks, like large SYN floods which send a very high volume of very small packets, there can be plenty of bandwidth on a port but a CPU can be bottlenecked on IRQ handling. When this happens it can shut down a particular core on a CPU or, in the worst case if IRQs aren't properly balanced, shut down the whole CPU. To better deal with these attacks, we needed to find a way to more intelligently spread IRQs across more interfaces and, in turn, more CPU cores.

Both these problems are annoying if it affects the customer under attack, but it is unacceptable it spills over and affects customers who are not under attack. To ensure that would never happen, we needed to find a way to both increase network capacity and ensure that customer attacks were isolated from one another. To accomplish this we launched what we affectionately refer to in the office as "Project Bondage."

Getting Into Bondage

To deal with these challenges we started by implementing what is known as port bonding. The idea of port bonding is simple: use the resources of multiple ports in aggregate in order to support more traffic than any one port can on its own. We use a custom operating system based on the Debian line of Linux. Like most Linux varieties, our OS supports seven different port bonding modes:

[0] Round-robin: Packets are transmitted sequentially through list of connections
[1] Active-backup: Only one connection is active, when it fails another is activated
[2] Balance-xor: This will ensure packets to a given destination from a given source will be the same over multiple connections
[3] Broadcast: Transmits everything over every active connection
[4] 802.3ad Dynamic Link Aggregation: Creates aggregation groups that share the same speed and duplex settings. Switches upstream must support 802.3ad.
[5] Balance-tlb: Adaptive transmit load balancing — outgoing traffic is balanced based on total amount being transmitted
[6] Balance-alb: Adaptive load balancing — includes balance-tlb and balances incoming traffic by using ARP negotiation to dynamically change the source MAC addresses of outgoing packets

We use mode 4, 802.3ad Dynamic Link Aggregation. This requires switches that support 802.3ad (our workhorse switch is a Juniper 4200, which does). Our switches are configured to send packets from each stream to the same network interface. If you want to experiment with port bonding yourself, the next section covers the technical details of exactly how we set it up.

The Nitty Gritty

Port bonding is configured on each server. It requires two Linux components that you can apt-get (assuming you're using a Debian-based Linux) if they're not already installed: ifenslave and ethtool. To initialize the bonding driver we use the following command:

modprobe bonding mode=4 miimon=100 xmit_hash_policy=1 lacp_rate=1

Here's how that command breaks down:

mode=4: 802.3ad Dynamic Link Aggregation mode described above
miimon=100: indicates that the devices are polled every 100ms to check for * connection changes, such as a link being down or a link duplex having changed.
xmit_hash_policy=1: instructs the driver to spread the load over interfaces based on the source and destination IP address instead of MAC address
lacp_rate=1: sets the rate for transmitting LACPDU packets, 0 is once every 30 seconds, 1 is every 1 second, which allows our network devices to automatically configure a single logical connection at the switch quickly

After the bonding driver is initialized, we bring down the servers' network interfaces:

ifconfig eth0 downifconfig eth1 down

We then bring up the bonding interface:

ifconfig bond0 192.168.0.2/24 up

We then enslave (seriously, that's the term) the interfaces in the bond:

ifenslave bond0 eth0ifenslave bond0 eth1

Finally, we check the status of the bonded interface:

cat /proc/net/bonding/bond0

From an application perspective, bonded ports appear as a single logical network interface with a higher maximum throughput. Since our switch recognizes and supports 802.3ad Dynamic Link Aggregation, we don't have to make any changes to its configuration in order for port bonding to work. In our case, we aggregate three ports (3Gbps) for handling external traffic and the remaining three ports (3Gbps) for handling intra-server traffic across our LAN.

Working Out the Kinks

Expanding the maximum effective capacity of each logical interface is half the battle. The other half is ensuring that network interrupts (IRQs) don't become a bottleneck. By default most Linux distributions rely on a service called irqbalance to set the CPU affinity of each IRQ queue. Unfortunately, we found that irqbalance does not effectively isolate each queue from overwhelming another on the same CPU. The problem with this is, because of the traffic we need to send from machine to machine, external attack traffic risked disrupting internal LAN traffic and affecting site performance beyond the customer under attack.

To solve this, the first thing we did was disable irqbalance:

/etc/init.d/irqbalance stopupdate-rc.d irqbalance remove

Instead, we explicitly setup IRQ handling to isolate our external and internal (LAN) networks. Each of our servers has two physical CPUs (G3 hardware uses a low-watt version of Intel Westmere line of CPUs) with six physical cores each. We use Intel's hyperthreading technology which effectively doubles the number of logical CPU cores: 12 per CPU or 24 per server.

Each port on our NICs has a number of queues to handle incoming requests. These are known as RSS (Receive Side Scaling) queues. Each port has 8 RSS queues, we have 6 1Gbps NIC ports per server, so a total of 48 RSS queues. These 48 RSS queues are allocated to the 24 cores, with 2 RSS queues per core. We divide the RSS queues between internal (LAN) traffic and external traffic and bind each type of traffic to one of the two server CPUs. This ensures that even large SYN floods that may affect a machine's ability to handle more external requests won't keep it from handling requests from other servers in the data center.

The Results

The net effect of these changes allows us to handle 30% larger SYN floods per server and increases our maximum throughput per site per server by 300%. Equally importantly, by custom tuning our IRQ handling, it has allowed us to ensure that customers under attack are isolated from those who are not while still delivering the maximum performance by fully utilizing all the gear in our network.

Near the end of 2012, our ops and networking teams sat down to spec our next generation of gear, incorporating everything we've learned over the previous year. One of the biggest changes we're making with G4 is the jump from 1Gbps network interfaces up to 10Gbps network interfaces on our switches and servers. Even without bonding, our tests of the new G4 gear show that it significantly increases both maximum throughput and IRQ handling. Or, put more succinctly: this next generation of servers is smokin' fast.

The first installations of the G4 gear is now in testing in a handful of our facilities. After testing, we plan to roll out worldwide over the coming months. We're already planning a detailed tour of the gear we chose, an explanation of the decisions we made, and performance benchmarks to show you how this next generation of gear is going to make CloudFlare's network even faster, safer, and smarter. That's a blog post I'm looking forward to writing. Stay tuned!

The DDoS That Almost Broke the Internet

Matthew Prince — Wed, 27 Mar 2013 16:35:00 -0700

The New York Times this morning published a story about the Spamhaus DDoS attack and how CloudFlare helped mitigate it and keep the site online. The Times calls the attack the largest known DDoS attack ever on the Internet. We wrote about the attack last week. At the time, it was a large attack, sending 85Gbps of traffic. Since then, the attack got much worse. Here are some of the technical details of what we've seen.

Growth Spurt

On Monday, March 18, 2013 Spamhaus contacted CloudFlare regarding an attack they were seeing against their website spamhaus.org. They signed up for CloudFlare and we quickly mitigated the attack. The attack, initially, was approximately 10Gbps generated largely from open DNS recursors. On March 19, the attack increased in size, peaking at approximately 90Gbps. The attack fluctuated between 90Gbps and 30Gbps until 01:15 UTC on on March 21.

The attackers were quiet for a day. Then, on March 22 at 18:00 UTC, the attack resumed, peaking at 120Gbps of traffic hitting our network. As we discussed in the previous blog post, CloudFlare uses Anycast technology which spreads the load of a distributed attack across all our data centers. This allowed us to mitigate the attack without it affecting Spamhaus or any of our other customers. The attackers ceased their attack against the Spamhaus website four hours after it started.

Other than the scale, which was already among the largest DDoS attacks we've seen, there was nothing particularly unusual about the attack to this point. Then the attackers changed their tactics. Rather than attacking our customers directly, they started going after the network providers CloudFlare uses for bandwidth. More on that in a second, first a bit about how the Internet works.

Peering on the Internet

The "inter" in Internet refers to the fact that it is a collection of independent networks connected together. CloudFlare runs a network, Google runs a network, and bandwidth providers like Level3, AT&T, and Cogent run networks. These networks then interconnect through what are known as peering relationships.

When you surf the web, your browser sends and receives packets of information. These packets are sent from one network to another. You can see this by running a traceroute. Here's one from Stanford University's network to the New York Times' website (nytimes.com):

1  rtr-servcore1-serv01-webserv.slac.stanford.edu (134.79.197.130)  0.572 ms
2  rtr-core1-p2p-servcore1.slac.stanford.edu (134.79.252.166)  0.796 ms
3  rtr-border1-p2p-core1.slac.stanford.edu (134.79.252.133)  0.536 ms
4  slac-mr2-p2p-rtr-border1.slac.stanford.edu (192.68.191.245)  25.636 ms
5  sunncr5-ip-a-slacmr2.es.net (134.55.36.21)  3.306 ms
6  eqxsjrt1-te-sunncr5.es.net (134.55.38.146)  1.384 ms
7  xe-0-3-0.cr1.sjc2.us.above.net (64.125.24.1)  2.722 ms
8  xe-0-1-0.mpr1.sea1.us.above.net (64.125.31.17)  20.812 ms
9  209.249.122.125 (209.249.122.125)  21.385 ms

There are three networks in the above traceroute: stanford.edu, es.net, and above.net. The request starts at Stanford. Between lines 4 and 5 it passes from Stanford's network to their peer es.net. Then, between lines 6 and 7, it passes from es.net to above.net, which appears to provide hosting for the New York Times. This means Stanford has a peering relationship with ES.net. ES.net has a peering relationship with Above.net. And Above.net provides connectivity for the New York Times.

CloudFlare connects to a large number of networks. You can get a sense of some, although not all, of the networks we peer with through a tool like Hurricane Electric's BGP looking glass. CloudFlare connects to peers in two ways. First, we connect directly to certain large carriers and other networks to which we send a large amount of traffic. In this case, we connect our router directly to the router at the border of the other network, usually with a piece of fiber optic cable. Second, we connect to what are known as Internet Exchanges, IXs for short, where a number of networks meet in a central point.

Most major cities have an IX. The model for IXs are different in different parts of the world. Europe runs some of the most robust IXs, and CloudFlare connects to several of them including LINX (the London Internet Exchange), AMS-IX (the Amsterdam Internet Exchange), and DE-CIX (the Frankfurt Internet Exchange), among others. The major networks that make up the Internet --Google, Facebook Yahoo, etc. -- connect to these same exchanges to pass traffic between each other efficiently. When the Spamhaus attacker realized he couldn't go after CloudFlare directly, he began targeting our upstream peers and exchanges.

Headwaters

Once the attackers realized they couldn't knock CloudFlare itself offline even with more than 100Gbps of DDoS traffic, they went after our direct peers. In this case, they attacked the providers from whom CloudFlare buys bandwidth. We, primarily, contract with what are known as Tier 2 providers for CloudFlare's paid bandwidth. These companies peer with other providers and also buy bandwidth from so-called Tier 1 providers.

There are approximately a dozen Tier 1 providers on the Internet. The nature of these providers is that they don't buy bandwidth from anyone. Instead, they engage in what is known as settlement-free peering with the other Tier 1 providers. Tier 2 providers interconnect with each other and then buy bandwidth from the Tier 1 providers in order to ensure they can connect to every other point on the Internet. At the core of the Internet, if all else fails, it is these Tier 1 providers that ensure that every network is connected to every other network. If one of them fails, it's a big deal.

Anycast means that if the attacker attacked the last step in the traceroute then their attack would be spread across CloudFlare's worldwide network, so instead they attacked the second to last step which concentrated the attack on one single point. This wouldn't cause a network-wide outage, but it could potentially cause regional problems.

We carefully select our bandwidth providers to ensure they have the ability to deal with attacks like this. Our direct peers quickly filtered attack traffic at their edge. This pushed the attack upstream to their direct peers, largely Tier 1 networks. Tier 1 networks don't buy bandwidth from anyone, so the majority of the weight of the attack ended up being carried by them. While we don't have direct visibility into the traffic loads they saw, we have been told by one major Tier 1 provider that they saw more than 300Gbps of attack traffic related to this attack. That would make this attack one of the largest ever reported.

The challenge with attacks at this scale is they risk overwhelming the systems that link together the Internet itself. The largest routers that you can buy have, at most, 100Gbps ports. It is possible to bond more than one of these ports together to create capacity that is greater than 100Gbps however, at some point, there are limits to how much these routers can handle. If that limit is exceeded then the network becomes congested and slows down.

Over the last few days, as these attacks have increased, we've seen congestion across several major Tier 1s, primarily in Europe where most of the attacks were concentrated, that would have affected hundreds of millions of people even as they surfed sites unrelated to Spamhaus or CloudFlare. If the Internet felt a bit more sluggish for you over the last few days in Europe, this may be part of the reason why.

Attacks on the IXs

In addition to CloudFlare's direct peers, we also connect with other networks over the so-called Internet Exchanges (IXs). These IXs are, at their most basic level, switches into which multiple networks connect and can then pass bandwidth. In Europe, these IXs are run as non-profit entities and are considered critical infrastructure. They interconnect hundreds of the world's largest networks including CloudFlare, Google, Facebook, and just about every other major Internet company.

Beyond attacking CloudFlare's direct peers, the attackers also attacked the core IX infrastructure on the London Internet Exchange (LINX), the Amsterdam Internet Exchange (AMS-IX), the Frankfurt Internet Exchange (DE-CIX), and the Hong Kong Internet Exchange (HKIX). From our perspective, the attacks had the largest effect on LINX which caused impact over the exchange and LINX's systems that monitor the exchange, as visible through the drop in traffic recorded by their monitoring systems. (Corrected: see below for original phrasing.)

The congestion impacted many of the networks on the IXs, including CloudFlare's. As problems were detected on the IX, we would route traffic around them. However, several London-based CloudFlare users reported intermittent issues over the last several days. This is the root cause of those problems.

The attacks also exposed some vulnerabilities in the architecture of some IXs. We, along with many other network security experts, worked with the team at LINX to better secure themselves. In doing so, we developed a list of best practices for any IX in order to make them less vulnerable to attacks.

Two specific suggestions to limit attacks like this involve making it more difficult to attack the IP addresses that members of the IX use to interchange traffic between each other. We are working with IXs to ensure that: 1) these IP addresses should not be announced as routable across the public Internet; and 2) packets destined to these IP addresses should only be permitted from other IX IP addresses. We've been very impressed with the team at LINX and how quickly they've worked to implement these changes and add additional security to their IX and are hopeful other IXs will quickly follow their lead.

The Full Impact of the Open Recursor Problem

At the bottom of this attack we once again find the problem of open DNS recursors. The attackers were able to generate more than 300Gbps of traffic likely with a network of their own that only had access 1/100th of that amount of traffic themselves. We've written about how these mis-configured DNS recursors as abomb waiting to go off that literally threatens the stability of the Internet itself. We've now seen an attack that begins to illustrate the full extent of the problem.

While lists of open recursors have been passed around on network security lists for the last few years, on Monday the full extent of the problem was, for the first time, made public. The Open Resolver Project made available the full list of the 21.7 million open resolvers online in an effort to shut them down.

We'd debated doing the same thing ourselves for some time but worried about the collateral damage of what would happen if such a list fell into the hands of the bad guys. The last five days have made clear that the bad guys have the list of open resolvers and they are getting increasingly brazen in the attacks they are willing to launch. We are in full support of the Open Resolver Project and believe it is incumbent on all network providers to work with their customers to close any open resolvers running on their networks.

Unlike traditional botnets which could only generate limited traffic because of the modest Internet connections and home PCs they typically run on, these open resolvers are typically running on big servers with fat pipes. They are like bazookas and the events of the last week have shown the damage they can cause. What's troubling is that, compared with what is possible, this attack may prove to be relatively modest.

As someone in charge of DDoS mitigation at one of the Internet giants emailed me this weekend: "I've often said we don't have to prepare for the largest-possible attack, we just have to prepare for the largest attack the Internet can send without causing massive collateral damage to others. It looks like you've reached that point, so... congratulations!"

At CloudFlare one of our goals is to make DDoS something you only read about in the history books. We're proud of how our network held up under such a massive attack and are working with our peers and partners to ensure that the Internet overall can stand up to the threats it faces.

Correction: The original sentence about the impact on LINX was "From our perspective, the attacks had the largest effect on LINX which for a little over an hour on March 23 saw the infrastructure serving more than half of the usual 1.5Tbps of peak traffic fail." That was not well phrased, and has been edited, with notation in place.

Page Rules Reordering Now Available

Michelle Zatlyn — Fri, 22 Mar 2013 21:53:00 -0700

Page Rules are powerful tools for controlling how CloudFlare works on your site on a page-by-page basis. Customers customize CloudFlare with Page Rules based on their specific needs, including changing or extending caching, forwarding URLs, or disabling certain features for specific pages or directories.

Today, we're making managing Page Rules even easier with Page Rules reordering.

Page Rules are applied in the order they appear in your CloudFlare dashboard, from top to bottom. The order matters, since the first Page Rule to match the request is applied. So, if you want to apply aggressive caching to a specific set of pages but exclude caching on one login or admin URL, you'd put the exclude caching Page Rule above the aggressive caching Page Rule.

Before today, reordering Page Rules was cumbersome. You had to delete and re-add Page Rules in the order that you wanted them to be applied. Now, you can easily reorder Page Rules by clicking on the icon to the left and dragging them up or down.

If you're not familiar with Page Rules or not sure how to use them, review this tutorial:

https://support.cloudflare.com/entries/22576178-Is-there-a-tutorial-for-PageRules-

Page Rules are found in the menu available under the gear icon in My Websites, as shown in this screenshot.

Reordering feature is available now for all customers, with the number of Page Rules set by plan type: Free domains get 3, Pro domains get 20, Business domains get 50, and Enterprise domains get a custom number.

Leading Experts Weigh in on Industry Trends at Parallels Summit 2013

Kristin Tarr — Fri, 22 Mar 2013 01:26:00 -0700

Last month we attended Parallels Summit to meet with new and existing partners, and hear from leading experts on the latest industry trends. We had a great time giving limo rides to summit attendees and hosting "Conversations with CloudFlare" at our booth.

For these conversations, CloudFlare co-founderMichelle Zatlyn sat down with 15 leading experts in the hosting and service provider industry. Their conversations were captured live and offer insight into the latest trends and news in hosting and web services.

Below are highlights and video clips from five of those conversations. We will be posting the rest of the videos next week.

Dan Havens, VP of Sales at Parallels
When it comes to the adoption of Plesk 11, Dan says it's all about the gateway. Listen in as he discusses Plesk 11 benefits, changes to the hosting space in the last few years, and what country Parallels is really big in right now...

Sam Renkema, CEO of Spam Experts
Sam knows spam. As the CEO of one of the most widely deployed packages offered in the APS catalog, Sam has seen all the trends in email spam. Sam tells us what hosting providers are most concerned about and what the newest sector of email spam looks like...

Marco Howen, CEO of LuxCloud
"We have to adapt. If we're not doing it, we won't sell" - Marco emphasizes ease of adoption and making UIs as simple as possible. Oh, and LuxCloud is a globally impacting company with an employee headcount that might surprise you...

Terry Gerstner, VP of Marketing at MobeeArt
MobeeArt is like mobile magic, for your website. MobeeArt automatically converts websites to mobile. Sound too good to be true? It's not. And they are announcing a feature that's going to make running a mobile website even better...

Michele Nelyon, CEO of Blacknight
Michele has his finger on the pulse of Internet policy. There are big issues the hosting industry will be facing over the next few years. Michele discusses these issues and what we can do to help...

The DDoS That Knocked Spamhaus Offline (And How We Mitigated It)

Matthew Prince — Wed, 20 Mar 2013 18:26:00 -0700

At CloudFlare, we deal with large DDoS attacks every day. Usually, these attacks are directed at large companies or organizations that are reluctant to talk about their details. It's fun, therefore, whenever we have a customer that is willing to let us tell the story of an attack they saw and how we mitigated it. This is one of those stories.

Spamhaus

Yesterday, Tuesday, March 19, 2013, CloudFlare was contacted by the non-profit anti-spam organization Spamhaus. They were suffering a large DDoS attack against their website and asked if we could help mitigate the attack.

Spamhaus provides one of the key backbones that underpins much of the anti-spam filtering online. Run by a tireless team of volunteers, Spamhaus patrols the Internet for spammers and publishes a list of the servers they use to send their messages in order to empower email system administrators to filter unwanted messages. Spamhaus's services are so pervasive and important to the operation of the Internet's email architecture that, when a lawsuit threatened to shut the service down, industry experts testified [PDF, full disclosure: I wrote the brief back in the day] that doing so risked literally breaking email since Spamhaus is directly or indirectly responsible for filtering as much as 80% of daily spam messages.

Beginning on March 18, the Spamhaus site came under attack. The attack was large enough that the Spamhaus team wasn't sure of its size when they contacted us. It was sufficiently large to fully saturate their connection to the rest of the Internet and knock their site offline. These very large attacks, which are known as Layer 3 attacks, are difficult to stop with any on-premise solution. Put simply: if you have a router with a 10Gbps port, and someone sends you 11Gbps of traffic, it doesn't matter what intelligent software you have to stop the attack because your network link is completely saturated.

While we don't know who was behind this attack, Spamhaus has made plenty of enemies over the years. Spammers aren't always the most lovable of individuals and Spamhaus has been threatened, sued, and DDoSed regularly. Spamhaus's blocklists are distributed via DNS and there is a long list of volunteer organizations that mirror their DNS infrastructure in order to ensure it is resilient to attacks. The website, however, was unreachable.

Filling Up the Series of Tubes

Very large Layer 3 attacks are nearly always originated from a number of sources. These many sources each send traffic to a single Internet location, effectively creating a tidal wave that overwhelms the target's resources. In this sense, the attack is distributed (the first D in DDoS -- Distributed Denial of Service). The sources of attack traffic can be a group of individuals working together (e.g., the Anonymous LOIC model, although this is Layer 7 traffic and even at high volumes usually much smaller in volume than other methods), a botnet of compromised PCs, a botnet of compromised servers, misconfigured DNS resolvers, or even home Internet routers with weak passwords.

Since an attacker attempting to launch a Layer 3 attack doesn't care about receiving a response to the requests they send, the packets that make up the attack do not have to be accurate or correctly formatted. Attackers will regularly spoof all the information in the attack packets, including the source IP, making it look like the attack is coming from a virtually infinite number of sources. Since packets data can be fully randomized, using techniques like IP filtering even upstream becomes virtually useless.

Spamhaus signed up for CloudFlare on Tuesday afternoon and we immediately mitigated the attack, making the site once again reachable. (More on how we did that below.) Once on our network, we also began recording data about the attack. At first, the attack was relatively modest (around 10Gbps). There was a brief spike around 16:30 UTC, likely a test, that lasted approximately 10 minutes. Then, around 21:30 UTC, the attackers let loose a very large wave.

The graph below is generated from bandwidth samples across a number of the routers that sit in front of servers we use for DDoS scrubbing. The green area represents in-bound requests and the blue line represents out-bound responses. While there is always some attack traffic on our network, it's easy to see when the attack against Spamhaus started and then began to taper off around 02:30 UTC on March 20, 2013. As I'm writing this at 16:15 UTC on March 20, 2013, it appears the attack is picking up again.

How to Generate a 75Gbps DDoS

The largest source of attack traffic against Spamhaus came from DNS reflection. I've written about these attacks before and in the last year they have become the source of the largest Layer 3 DDoS attacks we see (sometimes well exceeding 100Gbps). Open DNS resolvers are quickly becoming the scourge of the Internet and the size of these attacks will only continue to rise until all providers make a concerted effort to close them. (It also makes sense to implement BCP-38, but that's a topic for another post another time.)

The basic technique of a DNS reflection attack is to send a request for a large DNS zone file with the source IP address spoofed to be the intended victim to a large number of open DNS resolvers. The resolvers then respond to the request, sending the large DNS zone answer to the intended victim. The attackers' requests themselves are only a fraction of the size of the responses, meaning the attacker can effectively amplify their attack to many times the size of the bandwidth resources they themselves control.

In the Spamhaus case, the attacker was sending requests for the DNS zone file for ripe.net to open DNS resolvers. The attacker spoofed the CloudFlare IPs we'd issued for Spamhaus as the source in their DNS requests. The open resolvers responded with DNS zone file, generating collectively approximately 75Gbps of attack traffic. The requests were likely approximately 36 bytes long (e.g. dig ANY ripe.net @X.X.X.X +edns=0 +bufsize=4096, where X.X.X.X is replaced with the IP address of an open DNS resolver) and the response was approximately 3,000 bytes, translating to a 100x amplification factor.

We recorded over 30,000 unique DNS resolvers involved in the attack. This translates to each open DNS resolver sending an average of 2.5Mbps, which is small enough to fly under the radar of most DNS resolvers. Because the attacker used a DNS amplification, the attacker only needed to control a botnet or cluster of servers to generate 750Mbps -- which is possible with a small sized botnet or a handful of AWS instances. It is worth repeating: open DNS resolvers are the scourge of the Internet and these attacks will become more common and large until service providers take serious efforts to close them.

How You Mitigate a 75Gbps DDoS

While large Layer 3 attacks are difficult for an on-premise DDoS solution to mitigate, CloudFlare's network was specifically designed from the beginning to stop these types of attacks. We make heavy use of Anycast. That means the same IP address is announced from every one of our 23 worldwide data centers. The network itself load balances requests to the nearest facility. Under normal circumstances, this helps us ensure a visitor is routed to the nearest data center on our network.

When there's an attack, Anycast serves to effectively dilute it by spreading it across our facilities. Since every data center announces the same IP address for any CloudFlare customer, traffic cannot be concentrated in any one location. Instead of the attack being many-to-one, it becomes many-to-many with no single point on the network acting as a bottleneck.

Once diluted, the attack becomes relatively easy to stop at each of our data centers. Because CloudFlare acts as a virtual shield in front of our customers sites, with Layer 3 attacks none of the attack traffic reaches the customer's servers. Traffic to Spamhaus's network dropped to below the levels when the attack started as soon as they signed up for our service.

Other Noise

While the majority of the traffic involved in the attack was DNS reflection, the attacker threw in a few other attack methods as well. One was a so-called ACK reflection attack. When a TCP connection is established there is a handshake. The server initiating the TCP session first sends a SYN (for synchronize) request to the receiving server. The receiving server responds with an ACK (for acknowledge). After that handshake, data can be exchanged.

In an ACK reflection, the attacker sends a number of SYN packets to servers with a spoofed source IP address pointing to the intended victim. The servers then respond to the victim's IP with an ACK. Like the DNS reflection attack, this disguises the source of the attack, making it appear to come from legitimate servers. However, unlike the DNS reflection attack, there is no amplification factor: the bandwidth from the ACKs is symmetrical to the bandwidth the attacker has to generate the SYNs. CloudFlare is configured to drop unmatched ACKs, which mitigates these types of attacks.

Whenever we see one of these large attacks, network operators will write to us upset that we are attacking their infrastructure with abusive DNS queries or SYN floods. In fact, it is their infrastructure that is being used to reflect an attack at us. By working with and educating network operators, they clean up their network which helps to solve the root cause of these large attacks.

History Repeats Itself

Finally, it's worth noting how similar this battle against DDoS attacks and open DNS relays is with Spamhaus's original fight. If DDoS is the network scourge of tomorrow, spam was its clear predecessor. Paul Vixie, the father of the DNSBL, set out in 1997 to use DNS to help shut down the spam source of the day: open email relays. These relays were being used to disguise the origin of spam messages, making them more difficult to block. What was needed was a list of mail relays that mail serves could query against and decide whether to accept messages.

While it wasn't originally designed with the idea in mind, DNS proved a highly scalable and efficient means to distribute a queryable list of open mail relays that email service providers could use to block unwanted messages. Spamhaus arose as one of the most respected and widely used DNSBLs, effectively blocking a huge percentage of daily spam volume.

As open mail relays were shut, spammers turned to virus writers to create botnets that could be used to relay spam. Spamhaus expanded their operations to list the IPs of known botnets, trying to stay ahead of spammers. CloudFlare's own history grew out of Project Honey Pot, which started as an automated service to track the resources used by spammers and publishes the HTTP:BL.

Today, as Spamhaus's success has eroded the business model of spammers, botnet operators are increasingly renting their networks to launch DDoS attacks. At the same time, DNSBLs proved that there were many functions that the DNS protocol could be used for, encouraging many people to tinker with installing their own DNS resolvers. Unfortunately, these DNS resolvers are often mis-configured and left open to abuse, making them the DDoS equivalent of the open mail relay.

If you're running a network, take a second to make sure you've closed any open resolvers before DDoS explodes into an even worse problem than it already is.

CloudFlare Keeps TheBayLights.org Running Bright

Kristin Tarr — Wed, 13 Mar 2013 16:52:00 -0700

The Art

When you think of San Francisco, undoubtedly one bridge in particular comes to mind - The Golden Gate Bridge. This year, however, the Bay Bridge is getting its moment in the spotlight thanks to Words Pictures Ideas, a CloudFlare customer.

Words Pictures Ideas services brands and organizations in need of smarter communications. While thinking of ways to commemorate the 75th anniversary of the Bay Bridge, WPI founder Ben Davis came up with the idea to turn the West Span of the bridge into a canvas for light art.

Partnering with internationally renowned light artist Leo Villareal, Ben and the WPI team began working on what would become the world's largest LED light sculpture: The Bay Lights.

The Plan

The Bay Lights were officially unveiled on March 5, 2013. Brian VanderZanden, Lead Developer at WPI, knew there would be a surge in traffic to TheBayLights.org leading up to that day, and most likely a huge surge in traffic on the day of the unveiling. WPI has many sites on CloudFlare, including TheBayLights.org. He reached out to CloudFlare to make sure the site was ready to handle the increase in traffic.

CloudFlare suggested a few small optimizations (minification, an image that wasn't proxied because on a "grey cloud" DNS record), one useful reminder (whitelist the CloudFlare IPs), and a powerful recommendation: Cache Everything.

By default, CloudFlare will cache obviously static assets, but pass dynamic HTML through to the customer's webserver. For heavy load, on content that is not changing rapidly, full HTML pages -- or the entire site -- can be delivered from CloudFlare's global network, preserving the customer's webserver, database and other infrastructure. (Note: combined with single file purge, CloudFlare can serve as the global network for delivering a static site even with rapid changes, much as the Obama 2012 web team did.)

The Results

On the day of the unveiling, with Cache Everything turned on, TheBayLights.org saw traffic increase with a decrease on their system's resource utilization.

By mid-day, a rush in traffic caused more load than the event's peak at 8:00 pm. The graph below shows an interesting resource demand for the site pre/post cache everything:

The site saw the largest influx of traffic between 8:00-9:00 pm, but the average I/O during that hour was under 2Mb/s. By midnight traffic was back down to only 2X of baseline traffic levels.

"We began to celebrate at 9:15 pm as we were confident that the peak in site traffic had been reached and there were no issues," said Brian. "We are thrilled with the guidance and help CloudFlare offered in keeping us online during our biggest moment of the year, as well as the day to day performance and security they provide for all of our sites."

The Bay Lights will continue to shine for the next two years, creating yet another tourist stop in San Francisco. At CloudFlare, we are excited to be a part of the experience and look forward to helping keep TheBayLights.org shining online.

Go London User Group

John Graham-Cumming — Thu, 07 Mar 2013 16:45:00 -0800

We've mentioned before that we're using Go internally for projects such as Railgun (and a new DNS server and SSL infrastructure amongst other things). And we've mentioned that we are opening an office in London.

Now we're putting those two things together by sponsoring and helping to organize a new Go London User Group. The first meeting is on March 27 at Makers Academy. CloudFlare will be providing food and drink. Speakers will be announced closer to the date. Be sure to sign up as we have limited space (and if it's full please put yourself on the waiting list so we can gauge how large the interest is).

Feel free to suggest speakers and talks of interest in the comments.

We're also actively hiring Go (and other) programmers in London and San Francisco.

Load Balancing without Load Balancers

Matthew Prince — Wed, 06 Mar 2013 17:43:00 -0800

CloudFlare had an hour-long outage this last weekend. Thankfully, outages like this have been a relatively rare occurance for our service. This is in spite of hundreds of thousands of customers, the enormous volume of legitimate traffic they generate, and the barrage of large denial of service attacks we are constantly mitigating on their behalf. While last weekend's outage exposed a flaw in our architecture that we're working to fully eliminate, largely our systems have been designed to be balanced and have no single points of failure. We haven't talked much about the architecture of CloudFlare's systems but thought the rest of the community might benefit from seeing some of the choices we've made, how we load balance our systems, and how this has allowed us to scale quickly and efficiently.

Failure Isn't an Option, It's a Fact

CloudFlare's architecture starts with an assumption: failure is going to happen. As a result, we have to plan for failure at every level and design a system that gracefully handles it when it occurs. To understand how we do this, you have to understand the components of CloudFlare's edge systems. Here are four critical components we deploy at the edge of our network:

Network:CloudFlare's 23 data centers (internally we refer to them as PoPs) are connected to the rest of the world via multiple providers. These connections are both through transit (bandwidth) providers as well as other networks we directly peer with.
Router:at the edge of each of our PoPs is a router. This router announces the paths packets take to CloudFlare's network from the rest of the Internet.
Switch: within each PoP there will be one or more switches that aggregate traffic within the PoP's local area network (LAN).
Server: behind each switch there are a collection of servers. These servers perform some of the key tasks to power CloudFlare's service including DNS resolution, proxying, caching, and logging.

Those are the four components you'll find in the racks that we run in locations around the world. You'll notice some things from a typical hardware stack seem to be missing. For example, there's no hardware load balancer. The problem with hardware load balancers (and hardware firewalls, for that matter) is that they often become the bottleneck and create a single point of failure themselves. Instead of relying on a piece of hardware to load balance across our network, we use routing protocols to spread traffic and handle failure.

Anycast Is Your Friend

For most of the Internet, IP addressess correspond to a single device connected to the public Internet. In your home or office, you may have multiple devices sitting behind a gateway using network address translation (NAT), but there is only one public IP address and all the devices that sit behind the network use a unique private IP address (e.g., in the space 192.168.X.X or 10.X.X.X). The general rule on the Internet is one unique IP per devices. This is a routing scheme known as Unicast. However, it's not the only way.

There are four major routing schemes: Unicast, Multicast, Broadcast, and Anycast. Multicast and Broadcast are so-called one-to-many routing schemes. With Broadcast, one node sends packets that hit all recipient nodes. Broadcast is not widely used any longer and was actually not implemented in IPv6 (its largest contemporary use has likely been launching SMURF DDoS attacks). With Multicast, one node sends packets that hit multiple (but not all) recipient nodes that have opted into a group (e.g., how a cable company may deliver a television broadcast over an IP network).

Unicast and Anycast are one-to-one routing schemes. In both, there is one sender and one recipient of the packet. The difference between the two is that while there is only one possible destination on the entire network for a packet sent over Unicast, with Anycast there are multiple possible destinations and the network itself picks the route that is most preferential. On the wide area network (WAN) -- aka. the Internet -- this preference is for the shortest path from the sender to the recipient. On the LAN, the preferences can be set with weights that are honored by the router.

Anycast at the WAN

At CloudFlare, we use Anycast at two levels: the WAN and the LAN. At the WAN level, every router in all of CloudFlare's 23 data centers announces all of our external-facing IP addresses. For example, one of the IPs that CloudFlare announces for DNS services is 173.245.58.205. A route to that IP address is announced from all 23 CloudFlare data centers. When you send a packet to that IP address, it passes through a series of routers. Those routers look at the available paths to CloudFlare's end points and send the packet down the one with the fewest stops along the way (i.e., "hops"). You can run a traceroute to see each of these steps.

If I run a traceroute from CloudFlare's office in San Francisco, the path my packets take is:

$ traceroute 173.245.58.205
traceroute to 173.245.58.205 (173.245.58.205), 64 hops max, 52 byte packets
1  192.168.2.1 (192.168.2.1)  3.473 ms  1.399 ms  1.247 ms
2  10.10.11.1 (10.10.11.1)  3.136 ms  2.857 ms  3.206 ms
3  ge-0-2-5.cr1.sfo1.us.nlayer.net (69.22.X.X)  2.936 ms  3.405 ms  3.193 ms
4  ae3-70g.cr1.pao1.us.nlayer.net (69.22.143.170)  3.638 ms  4.076 ms  3.911 ms
5  ae1-70g.cr1.sjc1.us.nlayer.net (69.22.143.165)  4.833 ms  4.874 ms  4.973 ms
6  ae1-40g.ar2.sjc1.us.nlayer.net (69.22.143.118)  8.926 ms  8.529 ms  6.742 ms
7  as13335.xe-8-0-5.ar2.sjc1.us.nlayer.net (69.22.130.146)  5.048 ms
8  173.245.58.205 (173.245.58.205)  4.601 ms  4.338 ms  4.611 ms

If you run the same traceroute from a Linode server in London, the path my packets take is:

$ traceroute 173.245.58.205
traceroute to 173.245.58.205 (173.245.58.205), 30 hops max, 60 byte packets
1  212.111.X.X (212.111.X.X)  6.574 ms  6.514 ms  6.522 ms
2  212.111.33.X (212.111.33.X)  0.934 ms  0.935 ms  0.969 ms
3  85.90.238.69 (85.90.238.69)  1.396 ms  1.381 ms  1.405 ms
4  ldn-b3-link.telia.net (80.239.167.93)  0.700 ms  0.696 ms  0.670 ms
5  ldn-bb1-link.telia.net (80.91.247.24)  2.349 ms  0.700 ms  0.671 ms
6  ldn-b5-link.telia.net (80.91.246.147)  0.759 ms  0.771 ms  0.774 ms
7  cloudflare-ic-154357-ldn-b5.c.telia.net (80.239.161.246)  0.917 ms  0.853 ms  0.833 ms
8  173.245.58.205 (173.245.58.205)  0.972 ms  1.292 ms  0.916 ms

In both cases, the 8th and final hop is the same. You can tell, however, that they are hitting different CloudFlare data centers from hints in the 7th hop (highlighted in red below): as13335.xe-8-0-5.ar2.sjc1.us.nlayer.net suggesting it is hitting San Jose and cloudflare-ic-154357-ldn-b5.c.telia.net suggesting it is hitting London.

Since packets will follow the shortest path, if a particular path is withdrawn then packets will find their way to the next shortest available route. For simple protocols like UDP that don't maintain state, Anycast is ideal and it has been used widely to load balance DNS for some time. At CloudFlare, we've done a significant amount of engineering to allow TCP to run across Anycast without flapping. This involves carefully adjusting routes in order to get optimal routing and also adjusting the way we handle protocol negotiation itself. While more complex to maintain than a Unicast network, the benefit is we can lose an entire data center and packets flow to the next closest facility without anyone noticing and hiccup.

Anycast in the LAN

Once a packet arrives as a particular CloudFlare data center we want to ensure it gets to a server that can correctly handle the request. There are four key tasks that CloudFlare's servers perform: DNS, proxy, cache, and logging. We tend to follow the Google-like approach and deploy generic, white-box servers that can perform a number of different functions. (Incidentally, if anyone is interested, we're thinking of doing a blog post to "tour" a typical CloudFlare server and discuss the choices we made in working with manufacturers to design them.) Since servers can fail or be overloaded, we need to be able to route traffic intelligently around problems. For that, we return to our old friend Anycast.

Using Anycast, each server within each of CloudFlare's data centers is setup to receive traffic from any of our public IP addresses. The routes to these servers are announced via the border gateway protocol (BGP) from the servers themselves. To do this we use a piece of software called Bird. (You can tell it's an awesomely intense piece of networking software just by looking at one of its developers.) While all servers announce a route across the LAN for all the IPs, each server assigns its own weight to each IPs route. The router is then configured such that the route with the lowest weight is preferred.

If a server crashes, Bird stops announcing the BGP route to the router. The router then begins sending traffic to the server with the next-lowest weighted route. We also monitor critical processes on each server. If any of these critical processes fails then it can signal Bird to withdraw a route. This is not all or nothing. The monitor is aware of the server's own load as well as the load on the other servers in the data center. If a particular server starts to become overloaded, and it appears there is sufficient capacity elsewhere, then just some of the BGP routes can be withdrawn to take some traffic away from the overloaded server.

Beyond failover, we are beginning to experiment with BGP to do true load balancing. In this case, the weights to multiple servers are the same and the router hashes the source IP, destination IP, and port in order to consistently route traffic to the same server. The hash mapping table can be adjusted to increase or decrease load to any machine in the cluster. This is relatively easy with simple protocols like UDP, so we're playing with it for DNS. It's trickier with protocols that need to maintain some session state, like TCP, and gets trickier still when you throw in SSL, but we have some cool things in our lab that will help us better spread load across all the available resources.

Failure Scenarios

To understand this architecture, it's useful to think through some common failure scenarios.

Process Crash: if a core process (DNS, proxy, cache, or logging) crashes then the monitor daemon running on the server detects the failure. The monitor signals Bird to withdraw the BGP routes that are routed to that process (e.g., if just DNS crashes then the IPs that are used for CloudFlare name servers will be withdrawn, but the server will still respond to proxy traffic). With the routes withdrawn, the router in the data center sends traffic to the route with the next-lowest weight. The monitor daemon restarts the DNS server and, after verifying it has come up cleanly, signals Bird to start announcing routes again.
Server Crash: if a whole server crashes, Bird crashes along with it. All BGP routes to the server are withdrawn and the router sends traffic to the servers with the next lowest route weights. A monitor process on a control server within the data center attempts to reboot the box using the IPMI management interface and, if that fails, a power cycle from the fancy power strip (PDU). After the monitor process has verified the box has come back up cleanly, Bird is restarted and routes to the server are reinitiated.
Switch Crash: if a switch fails, all BGP routes to the servers behind the switch are automatically withdrawn. The routers are configured if they lose sufficient routes to the machines to drop the IPs that correspond to those routes out of the WAN Anycast pool. Traffic fails over for those IPs to the next closest data center. Monitors both inside and outside the affected data center alert our networking team who monitor the network 24/7 that there has been a switch failure so they can investigate and attempt a reboot.
Router Crash: if a router fails, all BGP routes across the WAN are withdrawn for the data center for which the router is responsible. Traffic to the data center automatically fails over to the next closest data center. Monitors both inside and outside the affected data center alert our networking team who monitor the network 24/7 that there has been a router failure so they can investigate and attempt a reboot.
Global Thermonuclear War: would be bad, but CloudFlare may continue to be able to route traffic to whatever portion of the Internet is left. As facilities were vaporized (starting with Las Vegas) their routers would stop announcing routes. As long as some facilities remained connected to whatever remained of the nextwork (maybe Sydney, Australia?) they would provide a path for traffic destined for our customers. We've designed the network such that more than half of it can completely fail and we'll still be able to keep up with the traffic.

It's a rare company our size that gets to play with systems to globally load balance Internet-scale traffic. While we've done a number of smart things to build a very fault tolerant network, last weekend's events prove there is more work to be done. If these are the sort of problems that excite you and you're interested in helping build a network that can survive almost anything, we're hiring.

Today's Outage Post Mortem

Matthew Prince — Sun, 03 Mar 2013 13:47:00 -0800

This morning at 09:47 UTC CloudFlare effectively dropped off the Internet. The outage affected all of CloudFlare's services including DNS and any services that rely on our web proxy. During the outage, anyone accessing CloudFlare.com or any site on CloudFlare's network would have received a DNS error. Pings and Traceroutes to CloudFlare's network resulted in a "No Route to Host" error.

The cause of the outage was a system-wide failure of our edge routers. CloudFlare currently runs 23 data centers worldwide. These data centers are connected to the rest of the Internet using routers. These routers announce the path that, from any point on the Internet, packets should use to reach our network. When a router goes down, the routes to the network that sits behind the router are withdrawn from the rest of the Internet.

We regularly will shut down one or a small handful of routers when we are upgrading a facility. Because we use Anycast, traffic naturally fails to the next closest data center. However, this morning we encountered a bug that caused all of our routers to fail network wide.

Flowspec

We are largely a Juniper shop at CloudFlare and all the edge routers that were affected were from Juniper. One of the reasons we like Juniper is their support of a protocol called Flowspec. Flowspec allows you to propagate router rules to a large number of routers efficiently. At CloudFlare, we constantly make updates to the rules on our routers. We do this to fight attacks as well as to shift traffic so it can be served as fast as possible.

This morning, we saw a DDoS attack being launched against one of our customers. The attack specifically targeted the customer's DNS servers. We have an internal tool that profiles attacks and outputs signatures that our automated systems as well as our ops team can use to stop attacks. Often, we use these signatures in order to create router rules to either rate limit or drop known-bad requests.

In this case, our attack profiler output the fact that the attack packets were between 99,971 and 99,985 bytes long. That's odd to begin with because the largest packets sent across the Internet are typically in the 1,500-byte range and average around 500 – 600 bytes. We have the maximum packet size set to 4,470 on our network, which is on the large size, but well under what the attack profiler was telling us was the size of these attack packets.

Bad Rule

Someone from our operations team is monitoring our network 24/7. As is normal practice for us, one of our ops team members took the output from the profiler and added a rule based on its output to drop packets that were between 99,971 and 99,985 bytes long. Here's what the rule (somewhat simplified and with the IPs obscured) looked like in Junos, the Juniper operating system:

+    route 173.X.X.X/32-DNS-DROP {+        match {+            destination 173.X.X.X/32;+            port 53;+            packet-length [ 99971 99985 ];+        }+        then discard;+    }

Flowspec accepted the rule and relayed it to our edge network. What should have happened is that no packet should have matched that rule because no packet was actually that large. What happened instead is that the routers encountered the rule and then proceeded to consume all their RAM until they crashed.

In all cases, we run a monitoring process that reboots the routers automatically when they crash. That worked in a few cases. Unfortunately, many of the routers crashed in such a way that they did not reboot automatically and we were not able to access the routers' management ports. Even though some data centers came back online initially, they fell back over again because all the traffic across our entire network hit them and overloaded their resources.

Sam Bowne, a computer science professor at City College of San Francisco, used BGPlay to capture the following video of BGP sessions being withdrawn as our routers crashed:

Incident Response

CloudFlare's ops and network teams were aware of the incident immediately because of both internal and external monitors we run on our network. While it wasn't initially clear the reason the routers had crashed, it was clear that it was an issue caused by an inability for packets to find a route to our network. We were able to access some routers and see that they were crashing when they encountered this bad rule. We removed the rule and then called the network operations teams in the data centers where our routers were unresponsive to ask them to physically access the routers and perform a hard reboot.

CloudFlare's 23 data centers span 14 countries so the response took some time but within about 30 minutes we began to restore CloudFlare's network and services. By 10:49 UTC, all of CloudFlare's services were restored. We continue to investigate some edge cases where people are seeing outages. In nearly all of these cases, the problem is that a bad DNS response has been cached. Typically clearing the DNS cache will resolve the issue.

We have already reached out to Juniper to see if this is a known bug or something unique to our setup and the kind of traffic we were seeing at the time. We will be doing more extensive testing of Flowspec provisioned filters and evaluating whether there are ways we can isolate the application of the rules to only those data centers that need to be updated, rather than applying the rules network wide. Finally, we plan to proactively issue service credits to accounts covered by SLAs. Any amount of downtime is completely unacceptable to us and the whole CloudFlare team is sorry we let our customers down this morning.

Parallels to Syria

In writing this up, I was reminded of the parallels to the Syrian Internet outage we reported on earlier this year. In that case, we were able to detect as the Syrian government shut down their board routers and effectively cut the country off from the rest of the Internet. In CloudFlare's case the cause was not intentional or malicious, but the net effect was the same: a router change caused a network to go offline.

At CloudFlare, we spend a significant amount of our time immersed in the dark arts of Internet routing. This incident, like the incident in Syria, illustrates the power and importance of the these network protocols. We let our customer down this morning, but we will learn from the incident and put more controls in place to eliminate problems like this in the futre.

CloudFlare London: We're hiring

John Graham-Cumming — Thu, 28 Feb 2013 17:47:00 -0800

When we talk about international expansion we're usually talking about adding data centers around the world. The last one we added was in Seoul, South Korea. And we've had a data center in London for a very long time. But now we're adding something different: people.

As CloudFlare's customer base and network have grown our need for 24 hour operations and technical support has grown. At the moment keeping things running means keeping people awake in California. With data centers in 23 locations around the world and customers in every country CloudFlare staff have to keep things humming day and night.

And so CloudFlare will expand in the next couple of months with an office in London.

We believe that London will make an ideal base for operations and technical support to complement our San Francisco office, and we can dip into the rich London talent pool to find people.

We keep our Join Our Team page updated with positions on in London. These include Technical Customer Support and Technical Operations Engineer.

Keep an eye out for new openings as we expand into London.

How to Tell How Well Railgun Is Working for Your Site

Matthew Prince — Wed, 27 Feb 2013 23:07:00 -0800

Yesterday, we announced that 30 of the world's largest hosting providers are now supporting CloudFlare's Railgun WAN optimization technology. Railgun uses delta compression to only transmit the parts of a dynamic page that have changed from one request to another. The net effect is that, on average, we can achieve a 99.6% compression ratio. In other words, what uncompressed would have taken 200 packets with Railgun we can transmit in a single packet.

This blog post is about using headers we include in responses delivered from Railgun in order to see how it is working. We've been running Railgun on CloudFlare.com for the last few months so I'll use it as an example.

Exposing the Headers

When a request is handled by Railgun, CloudFlare inserts a header with diagnostic information to track how the protocol is doing. If you want to see these headers, you'll need to use a browser that supports examining header information. Google's Chrome Browser or Apple's Safari Browser allow you to access Developer Tools (in Google, select the View > Developer > Developer Tools menu; in Safari, select the Develop > Show Web Inspector menu). In Firefox, you can installFirebug to see response headers. Microsoft's Internet Explorer makes it a bit trickier to see the response headers, but you can use a tool like Fiddler in order to expose them.

At CloudFlare, we've also made a Chrome extension for our own debugging purposes that we call Claire. When installed, it adds a small "cloud" icon to the right corner of the URL bar. If you're visiting a site that uses CloudFlare, lights up orange. Small icons under the cloud indicate whether you're using SPDY, Railgun, or IPv6 for your connection. Clicking on the icon exposes more data including information about the Railgun connection.

While Claire makes seeing the Railgun information easy, I'm going to walk through the rest of this post assuming you don't have it installed. Instead, I'll use Chrome's Developer Tools for the examples.

Story in the Headers

If you open the Developer Tools panel and click on the Network tab you'll see an interface like the one in the picture below:

Clicking on the first item in the list, which represents the dynamic HTML content that makes up the page, and then clicking on the Headers tab will show you the headers your browser sent to CloudFlare's servers as well as, if you scroll down, the response headers that your browser received back. Below is a sample of the response headers when accessing www.cloudflare.com:

There are two headers that CloudFlare is inserting in the response:

cf-railgun: e95b1c46e0 0.02 0.037872 0030 9878

cf-ray: 478149ad1570291

The second of these headers is what we call a RayID. This is a unique serial number attached to every request through the CloudFlare network, start to finish, which helps us diagnose if there's a problem at some step in our chain. If you ever have an error on your site when accessing CloudFlare, providing the RayID to our support team can help us track down the cause very quickly. The header I'm going to focus on for this post is the cf-railgun header, which I'll break down below.

The CF-Railgun Header

The CF-Railgun header has up to five codes separated by a space. In order, these codes and their corresponding values from the example above are:

Railgun Request ID: e95b1c46e0
Compression Ratio: 0.02
Origin Processing Time: 0.037872
Railgun Flags: 0030
Version Number: 9878

Breaking these down, the Railgun Request ID corresponds to an internal process number that allows us to track what connection handled a request in order to diagnose potential problems. Generally, you shouldn't need this value unless you're reporting a problem with your Railgun installation.

The Compression Ratio is more interesting in gauging how Railgun is down. It represents the size of the response after Railgun's delta compression expressed as a percentage. In the example above, the HTML returned for www.cloudflare.com is 0.02% of the size of the original that would be returned assuming no origin compression. Another way of thinking about this is the amount of data saved, which can be calculated by subtracting the Compression Ratio value from 100. In this case, 99.98% of the data that would have been required to generate www.cloudflare.com doesn't need to be transmitted because of the Railgun compression.

The Origin Processing Time represents the time, in seconds, that Railgun waits for the origin web server to generate the page. In this case, the origin server takes 0.03782 seconds from when the Railgun listener sends the request to the origin to when it responds. If this number is large, it means your web server or database may be hitting a bottleneck that is slowing down its time to render the full page.

The Railgun Flags represent how a request was processed. The simplified way of looking at the Railgun Flags is to see the 4-digit sequence as zzXz. Ignore the z's and focus on the number or letter in the X position. If it is 3,7, B or F then it means Railgun Compression is working correctly.

If there is an error of some sort, the Compression Ratio is likely to be listed as "normal" or "direct." This means that Railgun's compression was bypassed for one reason or another. The Railgun Flags help diagnose why. The Railgun Flags are a bitset and, in order to fully interpret them,, you need to use the rg-diag utility which is included with the Railgun packages. Run the utility from the command line with the -decode option. For example, to decode the Railgun Code 0038, for example, you'd run:

rg-diag -decode=0038

Which returns in:

Railgun Flag Existing origin connection reused

Railgun Flag rg-sender sent dictionary

Railgun Flag rg-listener found dictionary

This information can be useful in diagnosing potential problems with Railgun. The good news is that the Railgun protocol is designed to be resilient. If a connection fails for some reason, in most cases it will immediately roll over to a normal HTTP or HTTPS connection without the visitor seeing an error.

Finally, returning to the cf-railgun header, the final variable is the Version Number which indicates the version of the Railgun Listener software that is running on the origin server's network. The numbers aren't necessarily sequential, so having a lower number than another Railgun Listener doesn't necessarily mean your Listener is out of date.

Claire Makes It Easy

The Claire Chrome Plugin simplifies the header, leaving out the Railgun Flags and Version Number. Instead, it returns the Railgun Request ID (useful to provide to our support team if there's an issue), the amount of data saved for the particular request (derived from 100 - the Compression Ratio), and the Origin Processing Time (in seconds). Generally, this is all you should need to see whether Railgun is functioning as intended on your site.

Stay tuned. We'll post more information on tips for getting the most out of Railgun, as well as some of the design and engineering considerations that went into designing the protocol, over the coming days.

CloudFlare's Railgun: Easier Than Ever

Matthew Prince — Tue, 26 Feb 2013 08:51:00 -0800

Over the next few days we have a number of announcements regarding CloudFlare's Railgun technology. We wanted to begin, however, with what is in some ways the end: the ways in which you can take advantage of Railgun yourself. Today we're proud to announce two ways in which you can make your dynamic content faster than was ever possible before.

Do It Yourself

First, today CloudFlare is announcing version 3.3.3 of Railgun. This version has been battle tested on high-traffic sites including Imgur and 4chan. It's run billions of requests in a number of different environments through the new protocol and we're ready to push it out to the world. To make the process of installing Railgun easy, we've released RPMs for most the popular Linux and BSD variants including:

Ubuntu 12.10
Ubuntu 12.04
Ubuntu 11.10
Ubuntu 10.04
FreeBSD 9
FreeBSD 8
CentOS 6
CentOS 5
Debian 6

You can download any of these RPMs via the CloudFlare Downloads page. In addition, we've released an Amazon Machine Instance (AMI) for Amazon Web Services that you can install if you want to have you own Railgun listener. The AMI will be available soon via the AWS AMI manager.

The largest platform we're missing is Windows Server and we are working on updates to the Go runtime in order to allow us to compile for that platform and meet our quality standards. For the Windows Server users out there, stay tuned. We haven't forgotten about you.

But Wait, There's More

But that's not the really exciting part. We're extremely excited to announce that a majority of the world's leading hosting providers are now supporting CloudFlare's Railgun technology. These 30 hosting providers have already registered to be CloudFlare Optimized Hosts. That means you can enable Railgun, usually with a single click and without having to install any software or change any of your code. Within the next few days, all of the following hosts will be supporting CloudFlare's Railgun:

In most cases, if you're already hosted with one of these hosting providers, getting the benefits of Railgun is free for you. If you're using one of these hosts, look for an option in your control panel to enable Railgun. If you're not already using one of these hosts, but you want to use Railgun, you should either contact your hosting provider to become a CloudFlare Optimized Partner, or consider switching to one of the providers above.

Railgun is a revolutionary new protocol that makes dynamic web performance significantly faster and less bandwidth-intensive than was ever previously possible. Over the next few days, we'll be releasing more details about the protocol. In the meantime, we wanted to make sure you knew where you could go to get Railgun today if you're interested. Stay tuned for more.

Good Web Security News: Open DNS Resolvers Are Getting Closed

Matthew Prince — Fri, 22 Feb 2013 21:12:00 -0800

This has been a rough week in the security industry with big attacks and compromises reported at companies from Facebook to Apple. We're therefore happy to end the week with some good news: the web's open resolvers, one of the sources of the biggest DDoS attacks, are getting closed.

Sad State of Affairs

Last October, we wrote a blog post about DDoS amplification attacks. This type of attack makes up some of the largest DDoSs CloudFlare sees, sometimes exceeding 100 gigabits per second (100Gbps). The attacks use DNS resolvers that haven't been properly secured in order to "amplify" the resources of the attacker. An attacker can achieve more than a 50x amplification, meaning that for every byte they are able to generate themselves they can pummel a victim with 50 bytes of garbage data.

The problem stems from misconfigured DNS resolver software (e.g., BIND) that is setup to respond to a query from any IP address. Since DNS requests typically are sent over UDP, which, unlike TCP, does not require a handshake, an attacker can spoof a victim's IP address as the source address in a packet and a misconfigured DNS resolver will happily bombard the victim with responses.

Closing the Open Resolvers

While CloudFlare's network is very good at absorbing even these large attacks, the long term solution for the web is for providers to clean up the open resolvers running on their networks. We wanted to help with that so we engaged in a bit of name-and-shame at the end of the last blog post, listing the networks with the largest number of open resolvers. The good news is it worked: almost four months later our tests show that the number of open resolvers across the Internet is down more than 30%. The chart below shows the progress individual networks have made in cleaning up the problem.

ASN	Network	10/30/12	2/22/13	% Change
21844	THEPLANET-AS - ThePlanet.com Internet Services, In	2925	2216	-24%
3462	HINET Data Communication Business Group	2739	2213	-19%
36351	SOFTLAYER - SoftLayer Technologies Inc.	1075	781	-27%
9394	CRNET CHINA RAILWAY Internet(CRNET)	1052	774	-26%
4713	OCN NTT Communications Corporation	1044	722	-31%
45595	PKTELECOM-AS-PK Pakistan Telecom Company Limited	1030	716	-30%
4134	CHINANET-BACKBONE No.31,Jin-rong Street	970	705	-27%
33182	DIMENOC - HostDime.com, Inc.	940	638	-32%
7018	ATT-INTERNET4 - AT&T Services, Inc.	934	624	-33%
24940	HETZNER-AS Hetzner Online AG RZ	872	593	-32%
26496	AS-26496-GO-DADDY-COM-LLC - GoDaddy.com, LLC	855	560	-35%
20773	HOSTEUROPE-AS Host Europe GmbH	835	517	-38%
16276	OVH OVH Systems	803	511	-36%
13768	PEER1 - Peer 1 Network Inc.	707	421	-40%
14383	VCS-AS - Virtacore Systems Inc	596	420	-30%
32613	IWEB-AS - iWeb Technologies Inc.	585	367	-37%
23352	SERVERCENTRAL - Server Central Network	577	350	-39%
2514	INFOSPHERE NTT PC Communications, Inc.	561	341	-39%
2519	VECTANT VECTANT Ltd.	531	326	-39%
15003	NOBIS-TECH - Nobis Technology Group, LLC	521	322	-38%
22773	ASN-CXA-ALL-CCI-22773-RDC - Cox Communications Inc	484	315	-35%
6830	LGI-UPC UPC Broadband Holding B.V.	453	307	-32%
12322	PROXAD Free SAS	449	299	-33%
21788	NOC - Network Operations Center Inc.	442	295	-33%
17506	UCOM UCOM Corp.	422	293	-31%
6939	HURRICANE - Hurricane Electric, Inc.	414	284	-31%
16265	LEASEWEB LeaseWeb B.V.	407	284	-30%
3269	ASN-IBSNAZ Telecom Italia S.p.a.	402	281	-30%
29550	SIMPLYTRANSIT Simply Transit Ltd	392	271	-31%
19262	VZGNI-TRANSIT - Verizon Online LLC	390	262	-33%

Kudos

A few other organizations deserve a special shout out for helping with this effort. The great folks at Team Cymru have been tracking open resolvers and other badness online since before CloudFlare was even an idea. Their consistent efforts in this area have been awesome and we're in the process of partnering with them to help get the word out.

In addition, SoftLayer has been especially vocal and active in spearheading clean up efforts on its network. As they pointed out in a great blog post, because of the size and nature of their network, it's often difficult for them to police the configuration of software their customers run. Even so, they are actively reaching out to customers to educate them about the dangers of running open resolvers on their networks.

We greatly appreciate country CERTs/CSIRTs and various Information Sharing and Analysis Centers (ISACs) reaching out to us offering to get in touch with some of the less responsive network providers.

Going forward, we are happy to provide the IP addresses running open resolvers directly to any network provider that is interested in cleaning up their networks. If you're running a network on the list above, please don't hesitate to reach out to us and we'll get you the data you need to help with cleanup.

When the Bad Guys Name Malware After You, You Know You're Doing Something Right

Matthew Prince — Thu, 14 Feb 2013 01:29:00 -0800

CloudFlare's I'm Under Attack Mode (IUAM) is elegantly simple. When a site is under an application layer (Layer 7) distributed denial of service (DDoS) attack, the mode will return a challenge page to a visitor. The challenge requires the visitor's browser to answer a math problem which takes a bit of time to compute. Once successfully answered, the browser can request a cookie and won't be challenged again.

2 + 2 = Surprisingly Effective

IUAM has been incredibly successful at stopping Layer 7 attacks, but it's had a dirty little secret since it was first launched. While we'd suggested that the math problem the browser had to solve would be computationally complex, in reality it was incredibly simple: literally adding together two single-digit integers.

Several people over the last 6 months had written to us to let us know about this "critical vulnerability." They explained how easy it would be for an attacker to reverse engineer the math problem and create malware that could bypass the protection. Internally, we had a bet on how long it would take for some bad guy to actually do so. My money was on "never."

Good News/Bad News: I Lost the Bet

When Lee and I created Project Honey Pot back in 2004 we spent hundreds of engineering hours designing traps that were so random they were hard to identify. Even then, I secretly worried that an enterprising bad guy would recognize some pattern in the traps and be able to avoid them. We watched carefully for 9 years and no one ever took the time to do so. It was great, on one hand, since it meant that Project Honey Pot kept tracking bad guys but, on the other, it meant it was never causing them enough trouble that they'd spend the engineering effort to defeat us. Lee and I learned the lesson: don't over-engineer too early.

Which brings me back to IUAM. This morning we got word from the great folks over at ESET that they'd detected malware specifically designed to bypass CloudFlare's IUAM. Called OutFlare -- how cool is it that we have malware named after us!! -- the malware reads our IUAM page, finds the simple math problem, and calculates the answer. It is hardly rocket science, but it was actually pretty thrilling to the whole CloudFlare team that we'd been so successful at stopping bad guys that at least one of them took the time to reverse engineer this protection.

Proof of Work

Unlike me, some other engineers on CloudFlare's team had a suspicion that this day would come. They therefore had, waiting in the wings, code to increase the complexity of IUAM's challenges. The malware pulls the math equation off the page and computes the answer before posting back. The solution was easy: obfuscate the equation and run through some other tricks that make it hard to find the answer if you're not actually rendering the Javascript.

Today, after getting word that the simple version of IUAM had been reverse engineered by the OutFlare's malware, we pushed an update. If you're using IUAM there's nothing you need to do to take advantage of the new protection, we've already updated the protection rendering the OutFlare malware obsolete.

Going forward, we have plans if this scheme gets cracked. Specifically, we have an IUAM version that relies on a field of mathematics known as "proof of work" problems. These are difficult to compute answers for but easy to verify. A recent example of such a proof of work problem which has captured the imagination of much of the tech community is Bitcoin. The electronic currency requires a significant amount of computational time to find the answer to a problem, but once found each answer ("coin") is easy to verify.

In Bitcoin's case, the difficulty of the question is adjusted upward over time to compensate for increasing computing power and to control currency inflation. We can use the same premise to increase the "work" that an attacker needs to do when we detect a Layer 7 attack against a CloudFlare customer.

Arms Race? Bet on the Cloud

In these situations there's always a question of whether there will be an arms race between the bad guys writing the malware and the good guys offering protection. In this case there may be, but I like our odds in such a war. As today's example demonstrated, because CloudFlare is deployed as a service and we can update our systems to adjust to new threats in realtime we have an asymmetrical advantage. Pity the poor malware writer who now has to reverse engineer the new IUAM protection and push a code change to all his bots. If he comes up with something effective, we'll just adapt again — instantly.

The history of such arms races suggests you should bet on the cloud to win. In the spam wars, spammers and anti-spam software makers were locked in an arms race that it looked like neither would win from the mid-90s through the mid-2000s. Then something changed: new services like MXLogic, MessageLabs, CloudMark, and Postini started delivering anti-spam not as software but as a "cloud" service. Not only were these services easier to install and administer than previous anti-spam software or appliances, they could also adjust to spammers in realtime. The result has been that today these services have largely won the spam wars.

One More Thing

One more thing with regard to OutFlare. While the malware was able to read and pass the simple math challenge, that is only one layer of IUAM's protection. On the server side, CloudFlare still tracked all requests and, for devices that created a statistically high number of connections, we automatically imposed rate limits and other mitigation techniques. In other words, even without the fix we made, our customers were protected from the attack.

Thanks again to our friends at ESET for alerting us to the new OutFlare malware. We'll keep our eyes open to any new variants and, as they inevitably arise, we'll continue to adapt to ensure that all CloudFlare customers are always a step ahead of the web's nastiest threats.

Facebook Bug Redirects the Web Through Javascript Widget Error

Matthew Prince — Fri, 08 Feb 2013 05:08:00 -0800

You may have heard that Facebook took down a significant portion of the Internet today. A bug in their Facebook Connect script -- which is installed widely across many sites including CNN, MSNBC.com, New York Magazine, and many more places -- caused users to be redirected to a Facebook error page. Here's a video of what it looked like if you visited NBCNews.com:

The incident raises two good points: 1) the risk of Javascript widgets creating a "single points of failure" on your web page; and 2) the ways in which CloudFlare can help protect you from similar errors.

Widgets & SPOF

Facebook Connect works as a piece of Javascript that is embeded on pages. When the bug occurred, the Javascript effectively hijacked the page and directed it somewhere else. It may seem like installing a widget such as the Facebook button is harmless, but today's incident shows how much harm it can actually cause.

Björn Kaiser wrote a great blog post last year about the risks that embedded Javascript widgets can create, and how their failure can create a single point of failure (SPOF) on your site. In the post, he describes how you can test the embedded widgets on your page to see what would happen if any of them fail. Given that no widget provider, even Facebook, is infallible it is important to understand the risk of widget failure bringing down your site.

How CloudFlare Helps

There are two distinct ways in which CloudFlare helps protect you from Javascript widgets taking down your site. The first is via our Rocket Loader feature.

While we don't describe it this way often, Rocket Loader is effectively an on-page Javascript optimizer. It sits in front of widgets and makes sure they load as fast as possible. It also has a number of failsafes that can protect from any widget hijacking your site the way Facebook's Connect service did today. While we primarily describe Rocket Loader as a performance feature, in this role it's also very helpful for security and site availability.

The second way we protect sites from misbehaving Javascript widgets is through CloudFlare's app store. Many CloudFlare apps are Javascript widgets of one kind or another. When you install any CloudFlare app, we go through the process of making sure that the app performs well and can run asychronously. This greatly reduces the risk of an CloudFlare-installed app becoming a SPOF. Moreover, because we can install, upgrade, and remove apps centrally, if a problem like Facebook's had occurred with one of the CloudFlare apps, we could quickly remove it from pages to keep it from causing harm.

#savetheweb

Today's Facebook incident shows the risks of misbehaving Javascript widgets, but it also helps drive home the point on how CloudFlare is really building a better web. To that end, we will continue to invest in improving Rocket Loader and adding more and more apps to the CloudFlare Apps Marketplace. If you haven't turned on Rocket Loader or added an app through the CloudFlare Apps Marketplace, you now have one more reason check them both out.

New "Lucky Thirteen" SSL Vulnerabilities: CloudFlare Users Protected

Matthew Prince — Mon, 04 Feb 2013 14:26:00 -0800

CloudFlare often gets early word of new vulnerabilities before they are released. Last week we got word that today (Monday, February 4, 2013) there would be a new SSL vulnerability announced. This vulnerability follows the BEAST and CRIME vulnerabilities that have been discovered over the last 18 months. The bad news is that TLS 1.1/1.2 do not fix the issue.

The vulnerabilities are known as the Lucky Thirteen.

The good news is that our analysis of the newest vulnerability suggests that, while theoretically possible, it is fairly difficult to exploit. It is a timing attack and you'd need to create a fairly large number of connections and measure the differences in timing. That's possible, but non-trivial.

That said, at CloudFlare we want to ensure that even remote risks are fully mitigated. In this case, the good news is CloudFlare's SSL configuration is, by default, not generally vulnerable to the new attack. Specifically, because we deprioritize the vulnerable SSL cipher, it makes anyone using a modern browser invulnerable to the attack when visiting a CloudFlare-protected site over an SSL connection.

While the easiest way to ensure that your site is protected from the new vulnerability is to sign up for CloudFlare's service, if you haven't gotten around to that yet then there are some steps you should take. First, when a new version of OpenSSL is released that removes this vulnerability, which we expect will happen in the next few weeks, you should upgrade. Second, you should prioritize the RC4 cipher in your web server above others as it isn't vulnerable.

Here's the Apache SSL cipher suite configuration we'd recommend:

SSLProtocol -ALL +SSLv3 +TLSv1SSLCipherSuite ECDHE-RSA-AES128-SHA256:AES128-GCM-SHA256:RC4:HIGH:!MD5:!aNULL:!EDHSSLHonorCipherOrder on

Here's the NGINX SSL cyber suite configuration we'd recommend:

ssl_protocols               SSLv3 TLSv1 TLSv1.1 TLSv1.2;ssl_ciphers                 ECDHE-RSA-AES128-SHA256:AES128-GCM-SHA256:RC4:HIGH:!MD5:!aNULL:!EDH;ssl_prefer_server_ciphers   on;

Edge Cache Expire TTL: Easiest way to override any existing headers

Michelle Zatlyn — Fri, 01 Feb 2013 22:27:00 -0800

CloudFlare makes caching easy. Our service automatically determines what files to cache based on file extensions. Performance benefits kick in automatically.

For customers that want advanced caching, beyond the defaults, we have Cache Everything available as Page Rules. Designate a URL and CloudFlare will cache everything, including HTML, out at the edges of our global network.

With Cache Everything, we respect all headers. If there is any header in place from the server or a CMS solution like WordPress, we will respect it. However, we got many requests from customers who wanted an easy way to override any existing headers. Today, we are releasing a new feature called 'Edge cache expire TTL' that does just that.

What is Edge Cache Expire TTL?

Edge cache expire TTL is the setting that controls how long CloudFlare's edge servers will cache a resource before requesting a fresh copy from your server. When you create a Cache Everything Page Rule, you now may choose whether to respect all existing headers or to override any headers that are in place from your server. By overwriting the headers, CloudFlare will cache more content at the CloudFlare edge network, decreasing load to your server.

Common situations where you may choose to overwrite existing headers:

You expect a large surge in traffic
You are under DDOS attack
You are not sure what the headers on WordPress or your server are set to
You are using WordPress and want to easily overwrite the default settings

It is important to emphasize when you do not want to use Cache Everything. If you have any personalized information on the page like login information or credit card information, you do not want to use the Cache Everything option.What is Browser Cache Expire TTL?Browser cache expire TTL is the time that CloudFlare instructs a visitor's browser to cache a resource. Until this time expires, the browser will load the resource from its local cache thus speeding up the request significantly. CloudFlare will respect the headers that you give us from your web server, and then we will communicate with the browser based on the time selected in this drop down menu.

Using both Edge Cache Expire TTL and Browser Cache Expire TTLWhen you'd like to have CloudFlare cache your content but want your visitors to always get a fresh copy of the page, you can use the new 'Edge cache expire TTL' setting to express this differentiation. Set a value for 'Edge cache expire TTL' to how often you want the CloudFlare CDN to refresh from your server, and 'Browser cache expire TTL' to how often you want your visitors' browsers to refresh the page content. This is useful when you have a rapidly changing page but still want the benefit of the CloudFlare cache to reduce your server load.

Plan DetailsCloudFlare offers a range of edge cache expire TTLs based on plan type:

Free 2 hours
Pro 1 hour
Business 30 minutes
Enterprise as low as 30 seconds

A Pro customer may set the refetch time to 1 hour. After 60 minutes, we return to your server for a fresh copy of the resource. Business customers may lower the refetch interval to 30 minutes. Enterprise customers may set this interval as low as 30 seconds.

How to Turn It OnLogin in to your CloudFlare account and choose "Page Rules" from under the gear icon. Enter the URL that you want to Cache Everything (under Custom Caching):The edge cache server TTL option will appear:The default setting is set to "Respect all existing headers." To override this setting, choose a time from the drop down menu:

You can find more information in our knowledge base articles here and here.

Give it a try and let us know what you think.

WordPress London Meetup January 2013

John Graham-Cumming — Fri, 18 Jan 2013 09:23:00 -0800

Last night I gave a short presentation about how to use CloudFlare with WordPress sites to about 60 people attending the WordPress London Meetup. CloudFlare was happy to be sponsor of the event providing drinks, beers and lots and lots of pizza. The meetup was held at the Google Campus.

There were two talks: I was preceded by designer Laura Kalbag who talked about designing icons for WordPress sites. This is something that she made look incredibly easy using a tool called Sketch. I suspect that however good Sketch is, I'd end up drawing icons that looked awful!

My talk was about using WordPress and CloudFlare together. CloudFlare has a ton of features and I highlighted some that are of great interest to WordPress users including the CloudFlare WordPress Plugin and our integration with W3TC.

The other features that people found most interesting were:

Always Online: CloudFlare crawls the WordPress site and keeps a copy in a special cache. If the original site goes down CloudFlare serves up the most recent version from the crawler cache with a banner indicating that it is old content. This helps keep sites online when things go badly wrong.
Auto-minify: many WordPress sites have large amount of HTML, CSS and JavaScript (especially if they use lots of plugins). Auto-minify helps shrink those resources so that sites are delivered faster to web browsers.
Rocket Loader: a tool that reorganizes the loading of resources such as CSS and JavaScript to that they are downloaded to web browsers quickly by bundling them.
A new, unannounced feature that I'm calling "Help, I've gone viral!" which allows any web site owner to instantly tell CloudFlare to start completely caching a URL (overriding any caching headers set by the site) to cope with load. With this if a URL goes viral and is overloading a WordPress site it's possible to just paste in its URL and ask CloudFlare to take the load of that page. We'll be writing more about that feature when it's released.

And, of course, other CloudFlare features like Easy SSL, SPDY, and IPv6 help everyone get the latest technology onto their site quickly.

CloudFlare Heading to Parallels Summit 2013

Kristin Tarr — Thu, 17 Jan 2013 17:54:00 -0800

CloudFlare is heading to Parallels Summit in Las Vegas on Monday, February 4th to Wednesday, February 6th. We look forward to meeting and reconnecting with service providers and providing complimentary limo rides between the airport and the Caesars Palace hotel.

We recently launched a partnership with Parallels. Any Parallels service provider can increase revenue, while providing security and performance to customers' web properties and applications.

If you are already a CloudFlare Certified Partner, come introduce yourself at our booth (#113) and hear what we've been working on. If you are not a partner yet, stop by to learn more about CloudFlare and ways we can work together.

Where the CloudFlare team will be at Parallels Summit:

Monday, February 4th
8am-4pm: Limo transfers from McCarran International Airport to the Caesars Palace Hotel. Sign up now to ride in style!
4pm-7pm: Welcome Reception and Exhibit Hall opening, find CloudFlare at booth #113

*CloudFlare limo rides
*

Tuesday, February 5th
12:30pm onwards: CloudFlare is in the Exhibit Hall at Booth #113.
1pm-4pm: Conversations with CloudFlare - We will be conducting live video interviews with thought leaders within the web services industry, including founders and executives. Check out the video interviews we conducted at HostingCon 2013.
4pm-5pm: Our co-founder and CEO Matthew Prince will be speaking on the cloud portfolio panel breakout session "What's in your 2013 Cloud Portfolio?"
7pm-11pm: Come find us at the conference attendee party at PURE nightclub

*Industry leader interviews
*

Wednesdayday, February 6th
10:30am onwards: CloudFlare is in the Exhibit Hall at Booth #113.
2pm: Our co-founder and CEO Matthew Prince will be presenting in the Partners Theater Presentation.

Connect with us on Twitter during the event to find out where we are and what's coming up next:

#parallelssummit @ParallelsSummit @CloudFlare

Viva Las Vegas!

Today's System-Wide Upgrade

Matthew Prince — Fri, 11 Jan 2013 02:27:00 -0800

Today from 21:00 - 23:00 UTC CloudFlare scheduled a maintenance window. During that time, CloudFlare's interface was offline. While it was only two hours of time (and we finished a bit early, at 22:16 UTC) what went on during that window had been in the works for several months. I wanted to take a second and let you know what we just did and why.

When Dinosaurs Roamed the Web

Michelle, Lee and I started working on CloudFlare in early 2009. However, it wasn't until the beginning of 2010 that we invited the first users to sign up. Here's an early invite that went out to Project Honey Pot users. While CloudFlare's network today spans the globe, back then we only had one data center (in Chicago) and about 512 IP addresses (two /24 CIDRs).

Over the course of 2010, we built the product and continued to signup customers. It was a struggle to get our first 100 customers and, when we did, we took the whole team (at the time there were 6 of us) to Las Vegas. One of our very first blog posts was documenting that adventure. While today we regularly sign up 100 new customers an hour, we're really proud of the fact that a lot of those original customers are still CloudFlare customers today.

Over the course of the summer of 2010, about 1,000 customers signed up. On September 27, 2010, Michelle and I stepped on stage at TechCrunch Disrupt and launched the service live to the public. We were flooded with new signups, more than tripling in size in 48 hours. Our growth has only accelerated since then.

Provisioning and Accounting

One of the hardest non-obvious challenges to running CloudFlare is the accounting and provisioning of our resources across our customers sites. When someone signs up, we run hundreds of thousands of tests on the characteristics of the site in order to find the best pool to assign the site to. If a site signs up for a plan tier that supports HTTPS then we automatically issue and deploy a SSL certificate. And we spread sites across resource pools to ensure that we don't have hot spots on our network.

Originally, we were fairly arbitrary about what customers are assigned to what pool of resources. Over time, we've developed much more sophisticated systems to put new customers into the best pool of resources for them at the moment. However, the system has been relatively static: the pool a site is placed in when you first sign up generally has remained your pool over time.

Moving Sucks

Since provisioning has been relatively static, we had sites that were frozen in time. Those first 100 customers that were on CloudFlare's first IP addresses were mixed between free and paying customers. This lead to less efficient allocation of our server resources and, over time, kept us from better automating a number of systems that would better distribute load and isolate sites that were under attack from the rest of the network.

The solution was to migrate everyone from the various old systems to a new system. Lee began planning for this two months ago and stuck around the office over the holidays in order to ensure all the prep work was in place. To give you a sense of the scope, the migration script was 2,487 lines of code. And it would only be run once.

We picked a day after the holidays when all the team would be in the office. We run a global network with customers around the world so there is no quiet time during which to do system-wide maintenance, so we picked a time all our team would be on hand and fully alert. We ordered a pizza lunch for everyone and then, about an hour after lunch, began migrating everyone from various old deployments to a new, modern system.

Replacing the Engine (and Wings) of the Plane in Flight

It is non-trivial to move more than half a million websites around IP space. Sites were rebalanced to new pools of IP addresses. Where necessary, new SSL certificates were seamlessly issued. Custom SSL certificates were redeployed to new machines. From start to finish, the process took about an hour and sixteen minutes.

The process was designed to ensure that there would be no interruption in services. Unless you knew the IP addresses CloudFlare announced for your site, you likely wouldn't notice anything. And for most our customers, it went very smoothly.

We had two issues that affected a handful of customers. First, there was a conflict with some of our web server configurations that prevented a "staging" SSL environment from coming up properly. This staging environment was used as a temporary home for some sites that used SSL as they migrated from their old IP space to their new IP space. As a result, some customers saw SSL errors for about 10 minutes.

Second, a small number of customers were assigned to an IP address that had recently been under a DDoS attack and had been null routed at the routers. This null route would usually be recorded in our database, keeping sites from being assigned to the space until the null route was removed. In this case, the information wasn't correctly recorded and for a short time a small number of sites were on a pair of IP addresses that was unreachable. We removed the null route within a few minutes of realizing the mistake and the sites were again online.

Flexibility

We have known we needed to do this migration for quite some time. Now that it's done, CloudFlare's network is significantly more flexible and robust to ensure fast performance and keep attacks against one site from ever effecting any other customers.

To give you some sense of the flexibility the new system offers, here's a challenge we've faced. As CloudFlare looks to expand its network, some regions where we want to add data centers have restrictions on certain kinds of content being served. For example, in many Middle Eastern countries it is illegal to serve adult content from within their borders. CloudFlare is a reflection of the Internet, so there are adult-oriented sites that use our network. Making matters more difficult, the challenge is that what counts as an "adult" site can change over time.

The new system allows both our automated systems and our ops team to tag sites with certain characteristics. Now we can label a site as "adult" and the system automatically migrates it to a pool of resources that doesn't need to be announced from a particular region where serving the content would be illegal.

A similar use case is a site that is under attack. The new provisioning system allows us to isolate the site from the rest of the network so mitigate any collateral damage to other customers. We can also automatically dedicated additional resources (e.g., data centers in parts of the world that are at a lull of traffic based on the time of the day) in order to better mitigate the attacks. In the end, the benefit here is extreme flexibility.

We never like to take our site and API offline for any period of time, and I am disappointed we didn't complete the migration completely without incident, but overall this was a very important, surprisingly complex transition that went very smoothly. CloudFlare's network is now substantially more robust and flexible in order to continue to grow and expand as we continue on our mission to build a better web.

App: Clearspike automates search engine optimization

John Roberts — Wed, 09 Jan 2013 18:55:00 -0800

You care about your website, and you want it to be found. For many visitors, finding your website starts with search engines. Together, Google, Bing, Baidu and others are huge sources of traffic for every website.

The extra speed and security CloudFlare delivers are helpful for search engine ranking, but there are many other factors, including site content, organization and proper promotion.

The newest CloudFlare App, Clearspike automates the search engine optimization (SEO) process to help your website attract more organic search engine traffic.

We know you cared enough to make your website faster and safer. Improving your SEO is a complementary step, and we're pleased to make it easy to use the Clearspike service and tap into the expertise of the Clearspike team for additional benefits.

How it works

Like other CloudFlare Apps, Clearspike is easy to activate, with different levels of service available immediately, and no long-term commitment.

Self-Service Plan: Get custom recommendations and update website yourself. $24 / month.
Automated Plan: Use Clearspike tools to get website optimized automatically. $49 / month.
Do-It-For-Me Plan: Get Clearspike experts to optimize your website. $199 / month.

There's no tricks: the experts at Clearspike capture a wealth of experience in an easy-to-use service which makes their expertise usable and easy to apply.

At every level of service, Clearspike actively reviews your site for possible improvements, making recommendations and giving you tools to take action. The service includes keyword recommendations, page title optimizations, submission to appropriate directories, finding broken links, checking sitemaps and more. Clearspike helps you measure your progress, too, so you can see the return on your investment in SEO.

Try Clearspike now.

P.S. Clearspike made their service available to CloudFlare customers using the app development platform. CloudFlare is hiring to extend the platform.

CloudFlare: Fastest Free DNS, Among Fastest DNS Period

Matthew Prince — Mon, 07 Jan 2013 05:38:00 -0800

CloudFlare runs one of the largest networks of DNS servers in the world. Over the last few months, we've invested in making our DNS as fast and responsive as possible. We were happy to see these efforts pay off in third-party DNS test results.

The good folks at SolveDNS conduct a monthly survey of the fastest DNS providers in the world. CloudFlare has regularly been in the top-5 fastest DNS providers. This month we're up to number two, with SolveDNS's tests showing an average 4.51ms response time. That's just a hair behind number one (at 4.38ms) and almost twice as fast as number three (at 8.85ms). And, unlike most the other DNS providers in the top-10, CloudFlare's fast Anycast DNS service is provided even for our free plans.

Lest you think we're resting on our laurels, we've got a major DNS release (which we've dubbed RRDNS) scheduled for the next few months that we think will allow us to squeeze a bit more speed out of our DNS lookups. We're shooting for number one!

CloudFlare's 2012: Happy New Year!

Matthew Prince — Tue, 01 Jan 2013 02:14:00 -0800

For about half the world (and about half of CloudFlare's data centers) it's already 2013. As our team (most of whom are in San Francisco) get ready to celebrate New Year's Eve, wanted to quickly look back on CloudFlare's 2012. Here are some stats that tell the story of our last year:

Page views served by CloudFlare in 2012: 679,237,127,874
Hits served via CloudFlare's network in 2012: 3,691,532,490,107
Bandwidth served from CloudFlare's network in 2012: 765 Petabytes
Bandwidth we saved our customers in 2012: 436 Petabytes
New sites that signed up for CloudFlare in 2012: 573,177
Threats stopped by CloudFlare in 2012: 281,701,624,076
New CloudFlare data centers added in 2012: 10

Over 2012, we saw more than 720 million unique IPs connect to CloudFlare's network. Our best estimate is that behind each of those IPs there are 1.8 Internet users. In other words, we saw approximately 1.3 billion Internet users pass through CloudFlare's network in 2012. That's well over half of the Internet's total population of users.

We also saved a ton of time that those Internet users would have otherwise spent waiting for websites to load. If you add up all the time that people would have spent waiting for websites to load had CloudFlare not existed in 2012, you get more than 891 lifetimes worth of time saved. We're really proud of that.

We have a number of improvements, new features, new data centers, and other surprised lined up for 2013. From everyone at CloudFlare, Happy New Year! Here's to an even faster, safer Internet in the year ahead.

Optimizing Your Linux Stack for Maximum Mobile Web Performance

Matthew Prince — Mon, 31 Dec 2012 00:29:00 -0800

The following is a technical post written by Ian Applegate (@AppealingTea), a member of our Systems Engineering team, on how to optimize the Linux TCP stack for mobile connections. The article was originally published as part of the 2012 Web Performance Calendar. At CloudFlare, we spend a significant amount of time ensuring our network stack is tuned to whatever kind of network or device is connecting to us. We wanted to share some of the technical details to help other organizations that are looking to optimize for mobile network performance, even if they're not using CloudFlare. And, if you are using CloudFlare, you get all these benefits and the fastest possible TCP performance when a mobile network accesses your site.

We spend a lot of time at CloudFlare thinking about how to make the Internet fast on mobile devices. Currently there are over 1.2 billion active mobile users and that number is growing rapidly. Earlier this year mobile Internet access passed fixed Internet access in India and that's likely to be repeated the world over. So, mobile network performance will only become more and more important.

Most of the focus today on improving mobile performance is on Layer 7 with front end optimizations (FEO). At CloudFlare, we've done significant work in this area with front end optimization technologies like Rocket Loader, Mirage, and Polish that dynamically modify web content to make it load quickly whatever device is being used. However, while FEO is important to make mobile fast, the unique characteristics of mobile networks also means we have to pay attention to the underlying performance of the technologies down at Layer 4 of the network stack.

This article is about the challenges mobile devices present, how the default TCP configuration is ill-suited for optimal mobile performance, and what you can do to improve performance for visitors connecting via mobile networks. Before diving into the details, a quick technical note. At CloudFlare, we've build most of our systems on top of a custom version of Linux so, while the underlying technologies can apply to other operating systems, the examples I'll use are from Linux.

TCP Congestion Control

To understand the challenges of mobile network performance at Layer 4 of the networking stack you need to understand TCP Congestion Control. TCP Congestion Control is the gatekeeper that determines how to control the flow of packets from your server to your clients. Its goal is to prevent Internet congestion by detecting when congestion occurs and slowing down the rate data is transmitted. This helps ensure that the Internet is available to everyone, but can cause problems on mobile network when TCP mistakes mobile network problems for congestion.

TCP Congestion Control holds back the floodgates if it detects congestion (i.e. packet loss) on the remote end. A network is, inherently, a shared resource. The purpose of TCP Congestion Control was to ensure that every device on the network cooperates to not overwhelm its resource. On a wired network, if packet loss is detected it is a fairly reliable indicator that a port along the connection is overburdened. What is typically going on in these cases is that a memory buffer in a switch somewhere has filled beyond its capacity because packets are coming in faster than they can be sent out and data is being discarded. TCP Congestion Control on clients and servers is setup to "back off" in these cases in order to ensure that the network remains available for all its users.

But figuring out what packet loss means on a mobile network is a different matter. Radio networks are inherently susceptible to interference which results in packet loss. If pakcets are being dropped does that mean a switch is overburdened, like we can infer on a wired network? Or did someone travel from an undersubscribed wireless cell to an oversubscribed one? Or did someone just turn on a microwave? Or maybe it was just a random solar flare? Since it's not as clear what packet loss means on a mobile network, it's not clear what action a TCP Congestion Control algorithm should take.

A Series of Leaky Tubes

To optimize networks for lossy networks like those on mobile networks, it's important to understand exactly how TCP Congestion Control algorithms are designed. While the high level concept makes sense, the details of TCP Congestion Control are not widely understood by most people working in the web performance industry. That said, it is an important core part of what makes the Internet reliable and the subject of very active research and development.

To understand how TCP Congestion Control algorithms work, imagine the following analogy. Think of your web server as your local water utility plant. You've built on a large network of pipes in your hometown and you need to guarantee that each pipe is as pressurized as possible for delivery, but you don't want to burst the pipes. (Note: I recognize the late Senator Ted Stevens got a lot of flack for describing the Internet as a "series of tubes," but the metaphor is surprisingly accurate.)

Your client, Crazy Arty, runs a local water bottling plant that connects to your pipe network. Crazy Arty's infrastructure is built on old pipes that are leaky and brittle. For you to get water to them without bursting his pipes, you need to infer the capability of Crazy Arty's system. If you don't know in advance then you do a test — you send a known amount of water to the line and then measure the pressure. If the pressure is suddenly lost then you can infer that you broke a pipe. If not, then that level is likely safe and you can add more water pressure and repeat the test. You can iterate this test until you burst a pipe, see the drop in pressure, write down the maximum water volume, and going forward ensure you never exceed it.

Imagine, however, that there's some exogenous factor that could decrease the pressure in the pipe without actually indicating a pipe had burst. What if, for example, Crazy Arty ran a pump that he only turned on randomly from time to time and you didn't know about. If the only signal you have is observing a loss in pressure, you'd have no way of knowing whether you'd burst a pipe or if Crazy Arty had just plugged in the pump. The effect would be that you'd likely record a pressure level much less than the amount the pipes could actually withstand — leading to all your customers on the network potentially having lower water pressure than they should.

Optimizing for Congestion or Loss

If you've been following up to this point then you already know more about TCP Congestion Control than you would guess. The initial amount of water we talked about in TCP is known as the Initial Congestion Window (initcwnd) it is the initial number of packets in flight across the network. The congestion window (cwnd) either shrinks, grows, or stays the same depending on how many packets make it back and how fast (in ACK trains) they return after the initial burst. In essence, TCP Congestion Control is just like the water utility — measuring the pressure a network can withstand and then adjusting the volume in an attempt to maximize flow without bursting any pipes.

When a TCP connection is first established it attempts to ramp up the cwnd quickly. This phase of the connection, where TCP grows the cwnd rapidly, is called Slow Start. That's a bit of a misnomer since it is generally an exponential growth function which is quite fast and aggressive. Just like when the water utility in the example above detects a drop in pressure it turns down the volume of water, when TCP detects packets are lost it reduces the size of the cwnd and delays the time before another burst of packets is delivered. The time between packet bursts is known as the Retransmission Timeout (RTO). The algorithm within TCP that controls these processes is called the Congestion Control Algorithm. There are many congestion control algorithms and clients and servers can use different strategies based in the characteristics of their networks. Most of Congestion Control Algorithms focus on optimizing for one type of network loss or another: congestive loss (like you see on wired networks) or random loss (like you see on mobile networks).

In the example above, a pipe bursting would be an indication of congestive loss. There was a physical limit to the pipes, it is exceeded, and the appropriate response is to back off. On the other hand, Crazy Arty's pump is analogous to random loss. The capacity is still available on the network and only a temporary disturbance causes the water utility to see the pipes as overfull. The Internet started as a network of wired devices, and, as its name suggests, congestion control was largely designed to optimize for congestive loss. As a result, the default Congestion Control Algorithm in many operating systems is good for communicating wired networks but not as good for communicating with mobile networks.

A few Congestion Control algorithms try to bridge the gap by using the time of the delay in the "pressure increase" to "expected capacity" to figure out the cause of the loss. These are known as bandwidth estimation algorithms, and examples include Vegas, Veno and Westwood+. Unfortunately, all of these methods are reactive and reuse no information across two similar streams.

At companies that see a significant amount of network traffic, like CloudFlare or Google, it is possible to map the characteristics of the Internet's networks and choose a specific congestion control algorithm in order to maximize performance for that network. Unfortunately, unless you are seeing the large amounts of traffic as we do and can record data on network performance, the ability to instrument your congestion control or build a "weather forecast" is usually impossible. Fortunately, there are still several things you can do to make your server more responsive to visitors even when they're coming from lossy, mobile devices.

Compelling Reasons to Upgrade You Kernel

The Linux network stack has been under extensive development to bring about some sensible defaults and mechanisms for dealing with the network topology of 2012. A mixed network of high bandwidth low latency and high bandwidth, high latency, lossy connections was never fully anticipated by the kernel developers of 2009 and if you check your server's kernel version chances are it's running a 2.6.32.x kernel from that era.

uname -a

There are a number of reasons that if you're running an old kernel on your web server and want to increase web performance, especially for mobile devices, you should investigate upgrading. To begin, Linux 2.6.38 bumps the default initcwnd and initrwnd (inital receive window) from 3 to 10. This is an easy, big win. It allows for 14.2KB (vs 5.7KB) of data to be sent or received in the initial round trip before slow start grows the cwnd further. This is important for HTTP and SSL because it gives you more room to fit the header in the initial set of packets. If you are running an older kernel you may be able to run the following command on a bash shell (use caution) to set all of your routes' initcwnd and initrwnd to 10. On average, this small change can be one of the biggest boosts when you're trying to maximize web performance.

ip route | while read p; do ip route change $p initcwnd 10 initrwnd 10; done

Linux kernel 3.2 implements Proportional Rate Reduction (PRR). PRR decreases the time it takes for a lossy connection to recover its full speed, potentially improving HTTP response times by 3-10%. The benefits of PRR are significant for mobile networks. To understand why, it's worth diving back into the details of how previous congestion control strategies interacted with loss.

Many congestion control algorithms halve the cwnd when a loss is detected. When multiple losses occur this can result in a case where the cwnd is lower than the slow start threshold. Unfortunately, the connection never goes through slow start again. The result is that a few network interruptions can result in TCP slowing to a crawl for all the connections in the session.

This is even more deadly when combined with tcp_no_metrics_save=0 sysctl setting on unpatched kernels before 3.2. This setting will save data on connections and attempt to use it to optimize the network. Unfortunately, this can actually make performance worse because TCP will apply the exception case to every new connection from a client within a window of a few minutes. In other words, in some cases, one person surfing your site from a mobile phone who has some random packet loss can reduce your server's performance to this visitor even when their temporary loss has cleared.

If you expect your visitors to be coming from mobile, lossy connections and you cannot upgrade or patch your kernel I recommend setting tcp_no_metrics_save=1. If you're comfortable doing some hacking, you can patch older kernels.

The good news is that Linux 3.2 implements PRR, which decreases the amount of time that a lossy connection will impact TCP performance. If you can upgrade, it may be one of the most significant things you can do in order to increase your web performance.

More Improvements Ahead

Linux 3.2 also has another important improvement with RFC2099bis. The initial Retransmission Timeout (initRTO) has been changed to 1s from 3s. If loss happens after sending the initcwnd two seconds waiting time are saved when trying to resend the data. With TCP streams being so short this can have a very noticeable improvement if a connection experiences loss at the beginning of the stream. Like the PRR patch this can also be applied (with modification) to older kernels if for some reason you cannot upgrade (here's the patch).

Looking forward, Linux 3.3 has Byte Queue Limits when teamed with CoDel (controlled delay) in the 3.5 kernel helps fight the long standing issue of Bufferbloat by intelligently managing packet queues. Bufferbloat is when the caching overhead on TCP becomes inefficient because it's littered with stale data. Linux 3.3 has features to auto QoS important packets (SYN/DNS/ARP/etc.,) keep down buffer queues thereby reducing bufferbloat and improving latency on loaded servers.

Linux 3.5 implements TCP Early Retransmit with some safeguards for connections that have a small amount of packet reordering. This allows connections, under certain conditions, to trigger fast retransmit and bypass the costly Retransmission Timeout (RTO) mentioned earlier. By default it is enabled in the failsafe mode tcp_early_retrans=2. If for some reason you are sure your clients have loss but no reordering then you could set tcp_early_retrans=1 to save one quarter a RTT on recovery.

One of the most extensive changes to 3.6 that hasn't got much press is the removal of the IPv4 routing cache. In a nutshell it was an extraneous caching layer in the kernel that mapped interfaces to routes to IPs and saved a lookup to the Forward Information Base (FIB). The FIB is a routing table within the network stack. The IPv4 routing cache was intended to eliminate a FIB lookup and increase performance. While a good idea in principle, unfortunately it provided a very small performance boost in less than 10% of connections. In the 3.2.x-3.5.x kernels it was extremely vulnerable to certain DDoS techniques so it has been removed.

Finally, one important setting you should check, regardless of the Linux kernel you are running: tcp_slow_start_after_idle. If you're concerned about web performance, it has been proclaimed sysctl setting of the year. It can be enabled in almost any kernel. By default this is set to 1 which will aggressively reduce cwnd on idle connections and negatively impact any long lived connections such as SSL. The following command will set it to 0 and can significantly improve performance:

sysctl -w tcp_slow_start_after_idle=0

The Missing Congestion Control Algorithm

You may be curious as to why I haven't made a recommendation as far as a quick and easy change of congestion control algorithms. Since Linux 2.6.19, the default congestion control algorithm in the Linux kernel is CUBIC, which is time based and optimized for high speed and high latency networks. It's killer feature, known as called Hybrid Slow Start (HyStart), allows it to safely exit slow start by measuring the ACK trains and not overshoot the cwnd. It can improve startup throughput by up to 200-300%.

While other Congestion Control Algorithms may seem like performance wins on connections experiencing high amounts of loss (>.1%) (e.g., TCP Westwood+ or Hybla), unfortunately these algorithms don't include HyStart. The net effect is that, in our tests, they underperform CUBIC for general network performance. Unless a majority of your clients are on lossy connections, I recommend staying with CUBIC.

Of course the real answer here is to dynamically swap out congestion control algorithms based on historical data to better serve these edge cases. Unfortunately, that is difficult for the average web server unless you're seeing a very high volume of traffic and are able to record and analyze network characteristics across multiple connections. The good news is that loss predictors and hybrid congestion control algorithms are continuing to mature, so maybe we will have an answer in an upcoming kernel.

App: GamaSec Web Application Security and Vulnerability Scanning

Kristin Tarr — Sat, 22 Dec 2012 00:19:00 -0800

We enjoy working with companies who share a focus on website security. When GamaSec, an online web vulnerability-assessment service, inquired about ways to integrate, we were excited to make their scanning service available as a CloudFlare app, where any CloudFlare customer can easily turn on GamaSec.

GamaSec's cloud-based security scan serves as an early-warning system of defense for web operation, applications, and online information. GamaSec can be used by any website of any size and is now available to all CloudFlare customers: https://www.cloudflare.com/apps/gamasec

Vulnerability Scanning

GamaSec goes beyond signature-based tools to find more "real" vulnerabilities.

The GamaSec Application Vulnerability Scanner identifies application vulnerabilities such as Cross Site Scripting (XSS), SQL injection, and Code Inclusion, as well as site exposure risks. It also ranks threat priority, produces highly graphical, intuitive HTML reports, and indicates site security posture by vulnerabilities and threat exposure.

Benefits of GamaSec

Regular use of GamaSec's on-demand vulnerability assessment service provides the following benefits:

Fully automated scans
Easy dashboard & reporting
Web application SaaS Scanner
Update vulnerability protection
Trusted Website Security Seal
Web Application Scan via Cloud Computing

Plans, pricing and getting started

Like all CloudFlare apps, GamaSec is one-click simple, turned on in a customer's app dashboard.

There are two different plans, including Basic for $7.99 a month, per domain, and Premium for $16.99 a month, per domain, to fit the varied needs of different customers.

Visit the GamaSec app page to learn more and to get signed up!

P.S. GamaSec followed the CloudFlare app development process. CloudFlare is hiring to extend our platform.

Railgun in the real world: faster web page load times

John Graham-Cumming — Fri, 21 Dec 2012 10:50:00 -0800

In past blog posts I've described CloudFlare's Railgun technology that is designed to greatly speed up the delivery of non-cached pages. Although CloudFlare caches about 65% of the resources needed to make up a page, something like 35% can't be cached because they are dynamically generated or marked as 'do not cache'. And those 35% are often the initial HTML of the page that must be downloaded before anything else.

To solve that problem CloudFlare came up with a delta compression technique that recognizes that even dynamically-generated or personalized pages change only a little over time or between users. Railgun uses that compression technique to greatly reduce the amount of data that is sent over the Internet to CloudFlare's data centers from backend web servers. The result is faster delivery of the critical HTML that the browser must receive before it can download the rest of the page.

Testing with Railgun showed that very large compression ratios were possible and they resulted in a large speedup in page delivery. But two questions remained: "what's the effect in the real world?" and "how much difference does that make to page load time?".

We're now able to give some answers to those questions. The first hosting partner to roll out Railgun is Montreal-based Vexxhost. They gave us a sample of 51 web sites that they've enabled Railgun on and allowed us to run performance tests to see what difference Railgun makes. We decided to measure three things: how much faster the HTML is delivered, what the compression ratio is and how much page load time changes.

To get useful numbers we decided to load pages multiple times (each page was loaded 20 times with and without Railgun for a total of 40 downloads) and median values were used. Testing was done by downloading the pages from a machine in London, UK. The median round trip time between the nearest CloudFlare data center (where Railgun was running) and the origin web servers was 78ms.

HTML Delivery Speedup, Time To First Byte and Compression Ratio

On the 51 sites supplied by Vexxhost we saw a median speedup on downloading the HTML of 1.43x. To put that another way that means that the median time to download the HTML of the web pages decreased to 70% of what it was without Railgun.

Of the 51 sites 11 saw a speedup up of greater than 2x (i.e. the time to download the HTML of the web page more than halved) and for 8 of the sites the speedup was greater than 3x (i.e. the time to download the HTML of the web page was cut to a third of the original).

The median compression ratio achieved by Railgun was 0.65% (i.e. the page was reduced to 0.65% of its size). Of the 51 sites, only 9 saw a compression ratio greater than 3% (i.e. most of the pages were reduced to just a tiny percentage of their original size).

It's this huge compression that enables Railgun to speedup HTML delivery dramatically.

Another measurement to look at is Time To First Byte (how long it takes for the first byte of a page to be delivered to the browser). This is measured as the time from starting the TCP connection to the server to the moment the first byte is received from the server. Railgun has an effect on TTFB as well. The median improvement in TTFB was to drop it to 90% of the non-Railgun-accelerated value.

But HTML delivery is one thing, what's the real end-user visible effect? i.e. how does this translate to a difference in page load time.

Page Load Time

Railgun makes a difference to page load time because it accelerates the download of the initial HTML which has to occur before the rest of the page downloads. Downloading the HTML faster helps the entire page download more quickly. Here's an example of the effect of Railgun on CloudFlare's Plans page. This small test was done from the same machine in London as all the other tests. First here's the waterfall for that page without Railgun enabled.

The page load time was 1.83s. Now with Railgun enabled the page load time dropped to 1.15s because the time to download the initial HTML dropped.

Of course, that's just one test. Repeating the test 10 times with and without Railgun saw a median page load time of 1.59s with Railgun and 2.59s without (making the Railgun accelerated time 61% of the non-accelerated page load time). A similar test with CloudFlare's home page showed a median Railgun-accelerated page load time of 2.56s and without Railgun of 3.2s (i.e. Railgun makes the page load time drop to 83% of what it was).

To measure page load time on the 51 sites supplied by Vexxhost we set up PhantomJS (a headless browser that uses the WebKit for engine) on the same machine as used for the measurements above. A small script enabled us to generate HAR files of the download of entire web pages (including the JavaScript, CSS, HTML and images) and to extract the page load time (we use the 'onload' time).

These page load times include assets that are not accelerated by CloudFlare or by Railgun so they show realistic figures of how Railgun helps. Nevertheless, Railgun helps across the sites picked by Vexxhost with a median decrease in page load time to 89% of the original time. The best increase in median page load time was 56%. A small number of sites didn't see an improvement in page load time (they correspond to sites that didn't get a significant Railgun speedup because they typically only had a small amount of HTML).

A comparison of the same site downloaded via Railgun and not can be seen in these two images. The decrease in page load time is due to the decrease in time to get the initial HTML. Here's the page loading without Railgun:

And with Railgun the intial HTML load is accelerated resulting in a faster overall load time.

The difficulty with measuring page load time to see the Railgun-related improvement is that page load time is highly variable as different assets (especially from sites that are not accelerated by a CDN like CloudFlare) cause the page load time to vary enormously. To get a picture of the expected page load time improvement we can move on from measurement to estimation to check that measurements are similar the expected improvement.

Estimating the Railgun Improvement

One obvious question is to ask how much improvement Railgun can bring to a web site. To work that out you need to know two numbers: the page load time (call it p) and the time to download the initial HTML (call that h). Both values can be obtained from the Developer view in Safari or Chrome or from Firebug.

Railgun will be able to decrease time h. Using the figures above the median improvement would be 70% so you'd expect a page that takes p seconds to load to take roughly p - 0.3 * h with Railgun. In the CloudFlare example above p was 1.83s and h was 0.949s. The formula would give a Railgun page load time of 1.83 - 0.3 * 0.949 = 1.55s (the actual value 1.15s because Railgun did better than the median for that particular page).

In general, the larger the initial HTML the more Railgun can help. Very small pages won't require many round trips between the origin server and Railgun edge server, but larger pages will benefit from the delta compression. And Railgun helps when the web browser and origin server are far apart (for example, when a web site is accessed from around the world, Railgun will help eliminate the round trip time between a web surfer in one country and a web server in another).

To double check the measured performance above we ran a prediction for the sites that Vexxhost gave us. To predict the speedup generated by Railgun we first loaded each page 20 times using PhantomJS and extracted the median page load time (p) and the median time to download the initial HTML (h).

Then using the measured median speedup in the initial HTML load (see the first section above) we predicted that change in page load time by accelerating the initial HTML load and leaving all the other asset load times fixed.

The prediction showed that the median page would load in 93% of the non-Railgun-accelerated time. The measured times were 89%. As with the prediction for the speedup of a CloudFlare page, the measured times are better than the crude predictor, but both show the importance of accelerating the initial HTML load.

Conclusion

In real world tests Railgun gives a median decrease in the page load time to 89% of the non-accelerated time. That translates directly into an improved experience for the end user, and because Railgun runs everywhere in the CloudFlare network it means that page load times are improved for users wherever they are in the world.

Of course, none of this means that web page authors can be complacent about page load time. CloudFlare provides many tools to accelerate web page delivery and web page authors need to be mindful of slow assets and use tools like YSlow to make web page as fast as possible. They need to be particularly mindful of slow third-party assets (such as JavaScript libraries or Like and Share buttons loaded from other domains) as these directly affect page load time.

In fact, the greatest benefit from Railgun comes for sites that have already optimized page load time. Railgun will help drastically reduce the time taken for the already optimized page to reach our edge servers and be sent on to end users. In contrast, a page that has not been optimized may rely on tens of slow or third-party assets that must be downloaded for the page to be ready masking the effects of Railgun.

In a future post I'll look at Railgun performance when accelerating RESTful APIs. And I'll look at the effect of Railgun on subsequent page loads where static assets will be in local cache: in that case Railgun acceleration will be even more noticeable as the HTML download time will be a greater proportion of the total page load time.

Hackers love the holidays

John Graham-Cumming — Fri, 21 Dec 2012 03:41:00 -0800

This article was written by John Graham-Cumming on the CloudFlare team and originally published by VentureBeat. We're republishing it here.

Looking at the latest DDoS attack statistics from CloudFlare's network, it seems that hackers love the holidays.

Zooming in on November and December 2012 it's not hard to spot when Thanksgiving 2012 happened. Fully 1/5 of the attacks that CloudFlare saw in November and December (so far) happened on the Thursday and Friday of Thanksgiving:

In the past we've seen drops in DDoS attacks on some holidays because the home and office machines used as bots in those attacks have been turned off. For example, this year we noticed a large drop in attack activity on Earth Day (when people are encouraged to switch off their machines to save the planet). But this year's Thanksgiving attack statistics indicate that plenty of hacked machines were online through the holiday.

But what does this tell us about the coming Christmas holiday period? To answer that we can look back to December 2011. CloudFlare has DDoS data for December 11, 2011 to January 1, 2012 which shows two distinct peaks of attack activity: one just before Christmas and one just after.

So, if 2011 is a guide DDoS attackers will be taking a few days off over Christmas, but will be keeping the pressure on just before and immediately after. That's probably not a surprise as some fo the attackers will be attempting to disrupt businesses during critical periods for pre- and post-Christmas sales.

Even though there's a Christmas lull, that doesn't mean that CloudFlare staff will be letting down their guard, however. We'll be here working to ensure that whenever attacks arise and from whereever we're ready to absorb and deflect them.

It's the Most Wonderful Time of the Year...For Ecommerce Sites

Kristin Tarr — Mon, 17 Dec 2012 22:59:00 -0800

Forecasters have estimated that online holiday shopping will account for almost 25 percent of total ecommerce sales in 2012. That's more than $54 Billion dollars in online transactions. With so much shopping happening online, we thought we'd talk to one of our ecommerce customers to hear what they do to prepare their site for the busiest time of the year.

Luxury Link curates exclusive travel experiences with luxury properties around the world at insider prices. Chris Holland is the Director of Technology at Luxury Link, and has more than 16 years of web development experience. I recently spoke with Chris to learn more about Luxury Link, what he has seen over the years in the ecommerce industry, and what it's like to run an ecommerce site when the holidays hit.

Can you tell me a little about Luxury Link's story and technical background?

Since 1997, luxurylink.com has evolved from an exclusive e-mail list to exclusive online listings. Luxury Link has pioneered the web-based auction model for Luxury Travel. Our audience is extremely savvy, discerning and demanding of the greatest possible value for the most outstanding luxury vacation experiences.

While we used to have the niche to ourselves, the online travel landscape is competitive and so we are constantly working to optimize our website to make sure our visitors get the most out of the experience. Our web property experience includes everything from design to merchandising to site performance to SEO and the conversion funnel, as well as offering valuable insights to travelers while accommodating innovative marketing and product strategies. We, in the Tech Team, have our work cut out for ourselves catering to many business functions.

As the company has grown, how have your technology needs changed?

We've had to evolve beyond merely "selling online." It's no longer sufficient to put up a page clamoring "Here are 12 amazing vacations this week." We've seen travelers increasingly seeking inspiration and guidance. Finding the right vacation is a personalized and, at times, challenging process as many variables need to be juggled. While we've dramatically improved search and categorization on our site, we're just getting started. Solving these problems is less about using a specific search technology like Lucene, SphinX, SLI Systems, or Endeca and more about information architecture and accommodating a critical factor: Human curation. Everything you see on our site is an ever-evolving blend of human and machine curation. While search engines will seek out what you want, we have the added responsibility of helping visitors shape their traveling desires.

What are some tips/tricks you can offer other ecommerce site owners?

These core fundamentals really matter: performance, SEO, business intelligence, merchandising, and seasonal relevance.

For site performance, one of the tools we use is CloudFlare. To audit and monitor site speed, we use a blend of inexpensive resources such as webpagetest.org,Google PageSpeed Insights, and WatchMouse (now Nimsoft Cloud Monitor). I'll defer to your local expert to cover SEO.

We leverage Google Analytics and in-house-built event frameworks and data warehousing for various aspects of business intelligence. I'm a big fan of Tableau Software to crunch data.

For any site, and especially commerce sites, analyzing your marketing channels and respective conversion rates can uncover valuable insights: A/B testing is a very important part of this process. We've found PHP Scenario very helpful and we've integrated it into our A/B testing platform.

You might also consider giving your customers a voice by launching a community around your brand. While we've had a community on our site for some time, participation in it had died down. In 2012 we completely revamped it into "The Luxury Lounge" -- This initiative has brought about renewed interest from our loyal members in sharing their travel experiences. It's a veritable trove of great travel insights. It is positive for SEO, as well.

How has ecommerce changed in the last five years?

Online commerce has had to evolve beyond just listing and selling products, as competition and margins have become fierce. Consumers seek insights and guidance. Commerce sites featuring fresh, relevant and timely content in the form of editorial and consumer insights, tend to do better than sites that don't. In recognition of this, Google's algorithm updates have shaken things up. Incumbent sites that once merely listed products are finding themselves displaced by sites offering relevant content about those products. SEO is an exciting world where quality content is king, and this has had an impact on every commerce site I've worked on.

Do you see an increase in traffic during the holidays?

It is typically all about Q1 for the travel industry. We expect a 50% jump in traffic in January over November. While we do have plenty of capacity, CloudFlare's "always-on" feature is a nice safety net.

What do you do to prepare the site for the holidays?

We ensure our zabbix monitors are well-tuned, stick to best practices when deploying new code, don't stray away from our phones at nights, and generally do everything we can to ensure the site is running fast.

What are the "hot spots" your site visitors are looking into right now?

The top five pages and locations people are looking at include:

Ski and Snow Destinations
Caribbean
Cabo San Lucas
London
Bali
Guided Tours

How has CloudFlare impacted your site?

Our server origin is in downtown Los Angeles. We have seen a big speed difference in the average time it takes to download a dynamic web page weighing 23,000 bytes from Texas:

Without CloudFlare: 569 milliseconds
With CloudFlare: 332 milliseconds

This is for dynamic content. In other words, for this type of request, CloudFlare has to fetch the dynamic content from our system, and then pass it along to the user, every time. Going through CloudFlare for dynamic content delivery is 42 percent faster.

Overall, I believe CloudFlare is the best thing to happen to the Web in recent memory, and by extension, the Internet at large. CloudFlare's infrastructure is staggering and the architecture and pace of innovation are simply impressive. CloudFlare's offerings have an incredibly positive impact on site owners and web visitors.

CDNJS: The Fastest Javascript Repo on the Web

Matthew Prince — Wed, 12 Dec 2012 18:35:00 -0800

More than a year ago, Ryan Kirkman and Thomas Davis approached us about a project they were working on. Dubbed CDNJS, the project had a noble goal: make the world's Javascript resources load as fast as possible. They had been hosting the service on Amazon's CloudFront CDN, but as it got more popular the costs started to be significant. They approached us about whether we'd mind them using CloudFlare. We thought it was a great idea and we've been working together ever since. Today they just sent us data that shows CDNJS is the fastest Javascript repository on the Internet. More on that in a second, but first a bit about why CDNJS is so cool.

Why Do You Need a CDN for Javascript

The there are a core set of Javascript resources that are used across the web. Packages such as jQuery, Bootstrap, Backbone.js, and YUI underpin many of the web's pages. In order for these pages to load, the Javascript resources need to be downloaded. As a result, it makes sense for the resources to be on the fastest connections possible. However, that's only half the story.

The other benefit involves browser caching. If two sites use jQuery, ideally your browser only needs to download it once and then use the same code across both sites. In order to take advantage of this browser caching, both sites need to reference the same code via the same URL. As a result, it not only makes sense to reference a CDN for your Javascript code, but for you to use the same CDN as other sites are also using.

The Big Boys

Google and Microsoft have understood the benefits of having a central repository of Javascript resources and both provide their own public repositories. The challenge is that they only have a limited number of the most popular resources. Moreover, since running the repos isn't their primary job, they are slow to update as new versions of code comes out.

Everything so far is what Ryan and Thomas from CDNJS explained to us. They wanted to build a central repository for Javascript that was fast and reliable. They wanted to make sure it contained a wide range of the web's Javascript resources. They wanted to ensure that the latest versions would always be available. And they wanted to provide it to the web for free. We thought that sounded great, so we took over the job of serving the CDNJS resources from CloudFlare's global network.

Fast Wins

Today Ryan and Thomas sent us the latest data on the performance of CDNJS versus the Google and Microsoft Javascript CDNs. The results are terrific. Graphs are at the top of this post, but the here's the data: on average, CDNJS is 50% faster than the Google's Javascript CDN (100ms vs. 157ms), and more than four times as fast as Microsoft's CDN (100ms vs. 432ms). That's based on data gathered using Pingdom to download the same Javascript resource (jQuery 1.8.3 minified) from the three CDNs from multiple points around the globe over the last week.

CDNJS is also expanding beyond just Javascript. They've recently added CSS and Images for popular packages like Bootstrap. In other words, you can load the entire Bootstrap package directly from CDNJS, saving you bandwidth and ensuring it is delivered as quickly as possible. What's also great is that since CloudFlare's network supports SSL, SPDY, and IPv6 by default, these benefits also extend to CDNJS. In other words, if you're using any Javascript resources on your websites it's a no-brainer that you should be loading them from the CDNJS network.

What We Just Did to Make SSL Even Faster

Matthew Prince — Tue, 11 Dec 2012 17:24:00 -0800

A little over a month ago, we published a couple of blog posts about how we were making SSL faster. Specifically, we enabled OCSP stapling across our network. In brief, when you visit a page over HTTPS, your browser checks to see if the SSL certificate is still valid via a protocol called OCSP. Those checks can be slow so we took multiple steps to make them faster. The net effect was that, for browsers that supported OCSP stapling, visitors to HTTPS sites on CloudFlare would see about a 30% performance increase on their SSL handshakes.

That was the good news. What happened next was a number of people checked our SSL setup to validate our claims. While we had increased performance around OCSP checks, these investigations turned up a number of ways in which we weren't optimally deploying SSL. In particular, William Chan wrote a blog post looking at our SSL deployment and suggesting a number of things we could do to make it better.

We took this criticism to heart and today released an improved SSL process. Our goal is to provide the fastest, strongest SSL with the most ubiquitous browser support. Since a number of other cloud service providers are likely to face the same challenges, and since we haven't found anyone else that was automatically optimizing certificate bundles intelligently, we wanted to document what we did.

SSL: A Chain of Trust

In the simple case, SSL is easy. When your browser connects to a website over HTTPS, the site's web server returns a SSL certificate back to the browser. This certificate is used to verify the identity of the website and encrypt data exchanged between the browser and the web server.

In order to perform these functions, the certificate has to be trusted by the browser. A website's SSL certificate is issued by a Certificate Authority (CA). CAs vouch for a website's certificate as being valid. The CAs then have what are known as root certificates that are trusted by the browser. Since the CA's root is trusted by the browser, and since the CA trusts the web server's SSL certificate, by the transitive property the browser trusts the SSL certificate and a secure HTTPS connection can be established.

The challenge is that the world of SSL isn't always that clean. To begin, most CAs don't use their root certificate to directly sign the SSL certificates they issue to clients. Instead they use "intermediate" certificates. The chain of intermediate certificates can be of any arbitrary length, strung together to pass trust from the root to the eventual final certificate used by the web server. More complicated still, one intermediate certificate can have multiple parent certificates that vouch for its identity. GoDaddy, for example, has a case where their own root certificate as well as Valicert's root certificate both vouch for a GoDaddy intermediate certificate.

Long Chain = Slow Performance

While trust can pass along any length of SSL chain, the longer the chain is the more of a performance impact there is on setting up a HTTPS connection. While some of this overhead comes from validating all the certs in the chain, much also has to do with just having to transmit all the data that makes up the intermediate certificates.

As William Chan pointed out, CloudFlare was including more than we needed to and thereby overflowing the maximum amount of data per packet. This can have an especially large impact on web performance since SSL data is the first thing to be exchanged, no more data can be exchanged until after the SSL handshake takes place, so getting rid of a round trip on the SSL handshake can speed up everything else down the line.

CloudFlare's Smarter SSL Bundler

For SSL certificates we issued ourselves (such as those we create for Pro customers), this wasn't a problem. However, for custom SSL certificates, like those available for Business and Enterprise subscribers, we were not being smart about what we were including in the SSL bundle. The lowest hanging fruit in terms of reducing the size of these certificates was to remove the root certificates from the certificate bundle. There's no reason to include these since they should already be present in browsers and, even if they're not, the browser won't trust them.

We wanted to be even smarter about how we build bundles, so we spent some time developing a system that would find the shortest path between a certificate a user uploads to our system and one of the root certificates present in browsers. To do this, we needed to build a directory of the web's most common intermediate certificates. You'd think that's something someone would have assembled and published. We searched around for quite some time to find all these and didn't find it anywhere, so we created one ourselves. (PS - So no one else has to go through this same painful exercise, we're going to be publishing the directory on GitHub in the next few days and will keep it updated as we find more intermediate certificates.)

Build the Chains, Pick the Best

Today, when someone uploads a custom SSL certificate, we use our directory of intermediate certificates to build all the possible chains from the uploaded cert to a trusted browser root. We then rank these chains based on a number of factors including:

The length of the certificate chain
The ubiquity of the root certificate in browsers and other clients
The security of each step in the chain (e.g., does their Extended Key Usage include Server Authentication)
The length of the validity period of all the steps in the chain

The result is a server bundle that is small, fast and strong while having ubiquitous browser and client support. These are all optimizations that organizations concerned with performance and security should be doing by hand. What we're excited about is that we've automated this process and made it easy for anyone who wants the fastest possible SSL for their given certificate. If you were already using SSL through CloudFlare, your SSL bundle has been automatically optimized. If your site seemed a bit faster, that's why.

Going forward, in addition to releasing the directory of intermediate SSL certificates on Github, we plan on releasing our SSL bundler as a free service so you can package up your SSL certificates as efficiently as possible, even if you're not using CloudFlare. Just one more way we're working to make the web fast and safe.

App: iubenda generates fast, easy, simple privacy policies

John Roberts — Mon, 10 Dec 2012 22:16:00 -0800

Almost every website needs a privacy policy. In most countries, it's legally required, and it helps build trust with visitors. But without a lawyer on your team, writing a policy that meets your legal requirements can be challenging. What's even harder is avoiding jargon and writing a policy that's easy to read and understand.

If your site's privacy policy doesn't clear these high bars, we encourage you to try the newest CloudFlare App from iubenda. The iubenda team keeps up with worldwide policies and requirements, and has worked with lawyers to make generating a privacy policy simple. They capture your site's needs in an easy-to-read document that's easy to update without recoding your site.

The iubenda approach also radically reduces the cost: you'll have a clear policy maintained by a legal team for only $2.25/month/site.

We'd never envisioned a privacy policy App, but we're thrilled with the fit for many CloudFlare customers. Big companies have legal teams and have privacy policies, small companies (and some medium ones) need privacy policies, but certainly don't have legal teams.

We are excited to offer a service like iubenda which shares the CloudFlare ethos of bringing the tools of the Internet giants within reach of every website online.

How it works

iubenda lets website owners generate a privacy policy within seconds, beautifully designed, customized on needs and remotely maintained by a legal team. As you turn on the iubenda App, a minimal privacy policy will be added to your website as a badge attached to the lower border.

You customize and extend the privacy policy to match your practices using the iubenda step-by-step tools. Of course, you can integrate the policy in different ways, turning off the badge and presenting different integrations following iubenda's clear instructions.

We're proud to have iubenda's privacy policy service available now to all CloudFlare customers.

Extending the platform

When iubenda reached out to CloudFlare, it became clear that -- given their interesting service and technical savvy -- iubenda was a clear fit for building their App via our new packaged approach, which gives the developer more control over the resources needed.

Here's a link to iubenda's GitHub repo for their integration, following the guidelines documented here.

We encourage other interested developers to consider this approach of putting their service or website utility into the CloudFlare Apps marketplace where it can be instantly turned on by hundreds of thousands of CloudFlare customers.

Pushing Nginx to its limit with Lua

Matthieu Tourne — Sat, 08 Dec 2012 20:46:00 -0800

At CloudFlare, Nginx is at the core of what we do. It is part of the underlying foundation of our reverse proxy service. In addition to the built-in Nginx functionalities, we use an array of custom C modules that are specific to our infrastructure including load balancing, monitoring, and caching. Recently, we've been adding more simple services. And they are almost exclusively written in Lua.

I wanted to share more about how we are augmenting Nginx with new capabilities using Lua and provide some examples so you can do the same.

What is Lua?

Lua is a scripting language. Specifically, it is a full-featured multi-paradigm language with a simple syntax and semantics that resemble JavaScript or Scheme. Lua also has an interesting story to it, as it is one of the only languages from an emerging country that has had worldwide impact.

Lua has always meant to be embedded with larger systems written in other languages (like C and C++), and has thrived at staying very minimal and easy to integrate. As a result, Lua is popular within video games, security oriented software, and, more recently, Wikipedia.

Benefits of Nginx+Lua

Nginx+Lua is a self-contained web server embedding the scripting language Lua. Powerful applications can be written directly inside Nginx without using cgi, fastcgi, or uwsgi. By adding a little Lua code to an existing Nginx configuration file, it is easy to add small features. To see it yourself, at the end of this post I've included some logging code that can be added to any existing configuration.

One of the core benefits of Nginx+Lua is that it is fully asynchronous. Nginx+Lua inherits the same event loop model that has made Nginx a popular choice of webserver. "Asynchronous" simply means that Nginx can interrupt your code when it is waiting on a blocking operation, such as an outgoing connection or reading a file, and run the code of another incoming HTTP Request.

All the Lua code is written in a sequential fashion. The asynchronous logic is hidden to the Nginx+Lua programmer. If you are familiar with other event-driven webservers, that means no callbacks. In addition, Nginx+Lua is blazingly fast, leveraging the LuaJIT interpreter.

Getting Nginx+Lua installed

You can install it from source by compiling the lua-nginx-module with your existing Nginx. If you chose that path you will also need a Lua interpreter. LuaJIT-2.0.0 is recommended.

Or, you can use the tested ngx_openresty bundle. ngx_openresty comes loaded with Nginx, 3rd party modules, Lua libraries and other goodies. If you already use Nginx without 3rd party modules, from your Linux distribution for instance, you can safely swap it out with ngx_openresty. (Quick shout-out to my CloudFlare colleague Yichun Zhang who wrote ngx_openresty. Thanks, Yichun!)

Limitations

What makes Nginx, and therefore Nginx+Lua, really fast is the asychronous model and the event loop that Nginx relies on. To stay within that model, outgoing communication that is outside of Nginx has to be treated carefully. It is not recommended that you use classic LuaSocket, and instead it is recommended that you rely on the built-in ngx_lua sockets.

However, with a multitude of openresty libraries to "speak" SQL, memcached, and Redis, as well as the DNS built on top of ngx_lua sockets, this isn't really a problem in practice.

An example to try: Nginx Log Aggregation

Here is an example of how to build and run a simple log aggregator for Nginx. You can add it to any of your own existing configuration. This is the output once the aggregated logs are funneled to a time series system:

This particular example graph shows the average number of requests per second on certain nodes of the CloudFlare infrastructure.

Show me the code already!

Let's assume you already have a working webapp in Nginx, or that you use the proxy_pass directives to upstream to an Apache server.

First, add some lines in the Nginx conf to look at .lua files, and use a 1MB space of shared memory between your Nginx workers. ($prefix is relative to your Nginx install).

Next, add a little Lua snippet to calculate request_time for each request, and aggregate it into shared memory using a logging library available. Here is a simple logging library that I built.

This snippet can be used directly inline in your Nginx conf, using the log_by_lua directive.

Displaying/collecting aggregated logs

The last step to complete the example is a system to collect and/or display logs. In the full example, we set the aggregation as a separate server listening on a different port.

You should now have a functioning Nginx+Lua modification running in your environment.

Using Lua instead of custom C modules

This example showed how Lua found its way into our system at CloudFlare, but we soon realized that it wasn't limited to aggregating and printing logs. Using the same phases that Nginx has laid out for processing HTTP requests, it is becoming possible to add interesting new capabilities to Nginx, with almost as much control as a custom C module, while being pleasant and easy to write.

For instance, the access phase can be seen as a programmatic .htaccess, and even more. Whereas the content phase is where your web application would go.

Nginx+Lua has become a foundation for the work that I do at CloudFlare. As a long-time C developer, I am constantly struck by how powerful and extremely expressive Lua can be, while being simple and approachable as well.

Sometimes, simple is beautiful.

PS - We're hiring Lua programmers who are interested in working at extreme scale. Check out the Systems Engineer listing on our careers page if you're interested.

Efficiently compressing dynamically generated web content

John Graham-Cumming — Thu, 06 Dec 2012 09:19:00 -0800

I originally wrote this article for the Web Performance Calendar website, which is a terrific resource of expert opinions on making your website as fast as possible. We thought CloudFlare users would be interested so we reproduced it here. Enjoy!

Efficiently compressing dynamically generated web content

With the widespread adoption of high bandwidth Internet connections in the home, offices and on mobile devices, limitations in available bandwidth to download web pages have largely been eliminated.

At the same time latency remains a major problem. According to a recent presentation by Google, broadband Internet latency is 18ms for fiber technologies, 26ms for cable-based services, 43ms for DSL and 150ms-400ms for mobile devices. Ultimately, bandwidth can be expanded greatly with new technologies but latency is limited by the speed of light. The latency of an Internet connection directly affects the speed with which a web page can be downloaded.

The latency problem occurs because the TCP protocol requires round trips to acknowledge received information (since packets can and do get lost while traversing the Internet) and to prevent Internet congestion TCP has mechanisms to limit the amount of data sent per round trip until it has learnt how much it can send without causing congestion.

The collision between the speed of light and the TCP protocol is made worse by the fact that web site owners are likely to choose the cheapest hosting available without thinking about its physical location. In fact, the move to ‘the cloud' encourages the idea that web sites are simply ‘out there' without taking into account the very real problem of latency introduced by the distance between the end user's web browser and the server. It is not uncommon, for example, to see web sites aimed at UK consumers being hosted in the US. A web user in London accessing a .co.uk site that is actually hosted in Chicago incurs an additional 60ms round trip time because of the distance traversed.

Dealing with speed-of-light induced latency requires moving web content closer to user who are browsing, or making the web content smaller so that fewer round trips are required (or both).

The caching challenge

Caching technologies and content delivery services mean that static content (such as images, CSS, JavaScript) can be e cached close to end users helping to reduce latency when they are loaded. CloudFlare sees on average that about 65% of web content is cacheable.

But the most critical part of a web page, the actual HTML content is often dynamically generated and cannot be cached. Because none of the relatively fast to load content that's in cache cannot even be loaded before the HTML, any delay in the web browser receiving the HTML affects the entire web browsing experience.

Thus being able to deliver the page HTML as quickly as possible even in high latency environments is vital to ensuring a good browsing experience. Studies have shown that the slower the page load time the more likely the user is to give up and move elsewhere. A recent Google study said that a response time of less than 100ms is perceived by a human as ‘instant' (a human eye blink is somewhere in the 100ms to 400ms range); less than 300ms the computer seems sluggish; above 1s and the user's train of thought is lost to distraction or other thoughts. TCP's congestion avoidance algorithm means that many round trips are necessary when downloading a web page. For example, getting just the HTML for the CNN home page takes approximately 15 round trips; it's not hard to see how long latency can quickly multiply into a situation where the end-user is losing patience with the web site.

Unfortunately, it is not possible to cache the HTML of most web pages because it is dynamically generated. Dynamic pages are commonplace because the HTML is programmatically generated and not static. For example, a news web site will generate fresh HTML as news stories change or to show a different page depending on the geographical location of the end user. Many web pages are also dynamically generated because they are personalized for the end user — each person's Facebook page is unique. And web application frameworks, such as WordPress, encourage the use dynamically generate HTML by default and mark the content as uncachable.

Compression to the rescue

Given that web pages need to be dynamically generated the only viable option is to reduce the page size so that fewer TCP round trips are needed minimizing the effect of latency. The current best option for doing this is the use of the gzip encoding. On typical web page content gzip encoding will reduce the page size to about 20-25% of the original size. But this still results in multiple-kilobytes of page data being transmitted incurring the TCP congestion avoidance and latency penalty; in the CNN example above there were 15 round-trips even though the page was gzip compressed.

Gzip encoding is completely generic. It does not take into account any special features of the content it is compressing. It is also self-referential: a gzip encoded page is entirely self-contained. This is advantageous because it means that a system that uses gzipped content can be stateless, but it means that even larger compression ratios that would be possible with external dictionaries of common content are not possible.

External dictionaries increase compression ratios dramatically because the compressed data can refer to items from the dictionary. Those references can be very small (a few bytes each) but expand to very large content from the dictionary.

For example, imagine that it's necessary to transmit The King James Bible to a user. The plain text version from Project Gutenberg is 4,452,097 bytes and compressed with gzip it is 1,404,452 bytes (a reduction in size to 31%). But imagine the case where the compressor knows that the end user has a separate copy of the Old Testament and New Testament in a dictionary of useful content. Instead of transmitting a megabyte of gzip compressed content they can transmit an instruction of the form \<Insert Old Testament>\<Insert New Testament>. That instruction will just be a few bytes long.

Clearly, that's an extreme and unusual case but it highlights the usefulness of external shared dictionaries of common content that can be used to reconstruct an original, uncompressed document. External dictionaries can be applied to dynamically generated web content to achieve compression that exceeds that possible with gzip.

Caching page parts

On the web, shared dictionaries make sense because dynamic web content contains large chunks that's the same for all users and over time. Consider, for example the BBC News homepage which is approximately 116KB of HTML. That page is dynamically generated and the HTTP caching headers are set so that it is not cached. Even though the news stories on the page are frequently updated a large amount of boilerplate HTML does not change from request to request (or even user to user). The first 32KB of the page (28% of the HTML) consists of embedded JavaScript, headers, navigational elements and styles. If that ‘header block' were stored by web browsers in a local dictionary then the BBC would only need to send a small instruction saying \<Insert BBC Header> instead of 32KB of data. That would save multiple round-trips. And throughout the BBC News page there are smaller chunks of unchanging content that could be referenced from a dictionary.

It's not hard to imagine that for any web site there are large parts of the HTML that are the same from request to request and from user to user. Even on a very personalized site like Facebook the HTML is similar from user to user.

And as more and more applications use HTTP for APIs there's an opportunity to increase API performance through the use of shared dictionaries of JSON or XML. APIs often contain even more common, repeated parts than HTML as they are intended for machine consumption and change slowly over time (whereas the HTML of a page will change more quickly as designers update the look of a page).

Two different proposals have tried to address this in different ways: SDCH and ESI. Neither have achieved acceptance as Internet standards partly because of the added complexity of deploying them.

SDCH

In 2008, a group working at Google proposed a protocol for negotiating shared dictionaries of content so that a web server can compress a page in the knowledge that a web browser has chunks of the page in its cache. The proposal is known as SDCH (Shared Dictionary Compression over HTTP). Current versions of Google Chrome use SDCH to compress Google Search results.

This can be seen in the Developer Tools in Google Chrome. Any search request will contain an HTTP header specifying that the browser accepts SDCH compressed pages:

Accept-Encoding: gzip,deflate,sdch

And if SDCH is used then the server responds indicating the dictionary that was used. If necessary Chrome will retrieve the dictionary. Since the dictionary should change infrequently it will be in local web browser cache most of the time. For example, here's a sample HTTP header seen in a real response from a Google Search:

Get-Dictionary: /sdch/60W93cgP.dct

The dictionary file simply contains HTML (and JavaScript etc.) and the compressed page contains references to parts of the dictionary file using the VCDIFF format specified in RFC 3284. The compressed page consists mostly of COPY and ADD VCDIFF functions. A COPY x, y instruction tells the browser to copy y bytes of data from position x in the dictionary (this is how common content gets compressed and expanded from the dictionary). The ADD instruction is used to insert uncompressed data (i.e. those parts of the page that are not in the dictionary).

In a Google Search the dictionary is used to locally cache infrequently changing parts of a page (such as the HTML header, navigation elements and page footer).

SDCH has not achieved widespread acceptance because of the difficulty of generating the shared dictionaries. Three problems arise: when to update the dictionary, how to update the dictionary and prevention of leakage of private information.

For maximum effectiveness it's desirable to produce a shared dictionary that will be useful in reducing page sizes across a large number of page views. To do this it's necessary to either implement an automatic technique that samples real web traffic and identifies common blocks of HTML, or to determine which pages are most viewed and compute dictionaries for them (perhaps based on specialised knowledge of what parts of the page are common across requests).

When automated techniques are used it's important to ensure that when sampling traffic that contains personal information (such as for a logged in user) that personal information does not end up in the dictionary.

Although SDCH is powerful when used, these dictionary generation difficulties have prevented its widespread use. The Apache mod_sdch project is inactive and the Google SDCH group has been largely inactive since 2011.

ESI

In 2001 a consortium of companies proposed addressing both latency and common content with ESI (Edge Side Includes). Edge Side Includes work by having a web page creator identify unchanging parts of the page and then making these available as separate mini-pages using HTTP.

For example, if a page contains a common header and navigation, a web page author might place that in a separate nav.html file and then in a page they are authoring enter the following XML in place of the header and navigation HTML:

\<esi:include src="http://example.com/nav.html" "continue"/>

ESI is intended for use with HTML content that is delivered via a Content Delivery Network and major CDNs were the sponsor of the original proposal.

When a user retrieves a CDN managed page that contains ESI components the CDN reconstructs the complete page from the component parts (which the CDN will either have to retrieve, or, more likely, have in cache since they change infrequently).

The CDN delivers the complete, normal HTML to the end user, but because the CDN has access nodes all over the world the latency between the end user web browser and the CDN is minimized. ESI tries to minimize the amount of data sent between the origin web server and the CDN (where the latency may be high) while being transparent to the browser.

The biggest problem with adoption of ESI is that it forces web page authors to break pages up into blocks that can be safely cached by a CDN adding to the complexity of web page authoring. In addition, a CDN has to be used to deliver the pages as web browsers do not understand the ESI directives.

The time dimension

The SDCH and ESI approaches rely on identifying parts of pages that are known to be unchanging and can be cached either at the edge of a CDN or in a shared dictionary in a web browser.

Another approach is to consider how web pages evolve over time. It is common for web users to visit the same web pages frequently (such as news sites, online email, social media and major retailers). This may mean that a user's web browser has some previous version of the web page they are loading in its local cache. Even though that web page may be out of date it could still be used as a shared dictionary as components of it are likely to appear in the latest version of the page.

For example, a daily visit to a news web site could be speeded up if a web server were only able to send the differences between yesterday's news and today's. It's likely that most of the HTML of a page like the BBC News homepage will have remained unchanged; only the stories will be new and they will only make up a small portion of the page.

CloudFlare looked at how much dynamically generated pages change over time and found that, for example, reddit.com changes by about 2.15% over five minutes and 3.16% over an hour. The New York Times home page changes by about 0.6% over five minutes and 3% over an hour. BBC News changes by about 0.4% over five minutes and 2% over an hour. With delta compression it would be possible to turn those figures directly into a compression ratio by only sending the tiny percentage of the page that has changed. Compressing the BBC News web site to 0.4% is an enormous improvement compared to gzip's 20-25% compression ratio meaning that 116KB would result in just 464 bytes transmitted (which would likely all fit in a single TCP packet requiring a single round trip).

This delta method is the essence of RFC 3229 which was written in 2002.

RFC 3229

This RFC proposes an extension to HTTP where a web browser can indicate to a server that it has a particular version of a page (using the value from the ETag HTTP header that was supplied when the page was previously downloaded). The receiving web server can then apply a delta compression technique (encoded using VCDIFF discussed above) to send only the parts that have changed since that particular version of the page.

The RFC also proposes that a web browser be able to send the identifiers of multiple versions of a single page so that the web server can choose among them. That way, if the web browser has multiple versions in cache there's an increased chance that the server will have one of the versions available to it for delta compression.

Although this technique is powerful because it greatly reduces the amount of data to be sent from a web server to browser it has not been widely deployed because of the enormous resources needed on web servers.

To be effective a web server would need to keep copies of versions of the pages it generates in order that when a request comes in it is able to perform delta compression. For a popular web site that would create a large storage burden; for a site with heavy personalization it would mean keeping a copy of the pages served to every single user. For example, Facebook has around 1 billion active users, just keeping a copy of the HTML of the last time they viewed their timeline would require 250TB of storage.

CloudFlare's Railgun

CloudFlare's Railgun is a transparent delta compression technology that takes advantage of CloudFlare's CDN network to greatly accelerate the transmission of dynamically generated web pages from origin web servers to the CDN node nearest end user web surfers. Unlike SDCH and ESI it does not require any work on the part of a web site creator and unlike RFC 3229 it does not require caching a version of each page for each end user.

Railgun consists of two components: the sender and the listener. The sender is installed at every CloudFlare data center around the world. The listener is a software component that customers install on their network.

The sender and listener establish a permanent TCP connection that's secured by TLS. This TCP connection is used for the Railgun protocol. It's an all binary multiplexing protocol that allows multiple HTTP requests to be run simultaneously and asynchronously across the link. To a web client the Railgun system looks like a proxy server, but instead of being a server it's a wide-area link with special properties. One of those properties is that it performs compression on non-cacheable content by synchronizing page versions.

Each end of the Railgun link keeps track of the last version of a web page that's been requested. When a new request comes in for a page that Railgun has already seen, only the changes are sent across the link. The listener component make an HTTP request to the real, origin web server for the uncacheable page, makes a comparison with the stored version and sends across the differences.

The sender then reconstructs the page from its cache and the difference sent by the other side. Because multiple users pass through the same Railgun link only a single cached version of the page is needed for delta compression as opposed to one per end user with techniques like RFC 3229.

For example, a test on a major news site sent 23,529 bytes of gzipped data which when decompressed become 92,516 bytes of page (so the page is compressed to 25.25% of its original size). Railgun compression between two version of the page at a five minute interval resulted in just 266 bytes of difference data being sent (a compression to 0.29% of the original page size). The one hour difference is 2,885 bytes (a compression to 3% of the original page size). Clearly, Railgun delta compression outperforms gzip enormously.

For pages that are frequently accessed the deltas are often so small that they fit inside a single TCP packet, and because the connection between the two parts of Railgun is kept active problems with TCP congestion avoidance are eliminated.

Conclusion

The use of external dictionaries of content is a powerful technique that can achieve much larger compression ratios that the self-contained gzip method. But only CloudFlare's Railgun implements delta compression in a manner that is completely transparent to end users and website owners.

CloudFlare and Parallels to Bring Website Performance and Security to Millions of SMBs

Maria Karaivanova — Tue, 04 Dec 2012 22:01:00 -0800

In early October we quietly announced our partnership with Parallels, a global leader in hosting, cloud services enablement and desktop virtualization. Parallels makes it easier for service providers, like webhosts, to grow their business with service delivery software and a robust partner ecosystem.

Today, we are officially launching our partnership with Parallels. If you are a Parallels Service Provider and are using either Plesk or Parallels Automation, it means two things for you. First, your customers will be able to activate CloudFlare with a few clicks from their Parallels control panel. Second, you can now generate revenue by offering CloudFlare Performance Plus in addition to CloudFlare Free.

The partnership has made it super easy for all Parallels Service Providers to offer enterprise-grade performance and security to any website, especially their SMB and e-commerce customers. To get started, a hosting provider can install CloudFlare from the APS catalog. Installation takes less than 5 minutes and CloudFlare can be activated by all customers with just a few clicks.

Since our soft launch, we worked with over a hundred Parallels Service Providers who successfully integrated CloudFlare. In addition to generating revenue, partners enjoy many operational benefits such as reduced server load, bandwidth savings, protection from DDoS attacks and an automatic IPv4/6 gateway. Here's what some of them are saying about the partnership:

"As a result of our partnership with Parallels, we have been able to bring a number of innovative solutions to our customers," said Celal Ulgen, Chief Marketing Officer at SoftCom, parent company to the popular myhosting.com. "Now that CloudFlare is available in Parallels Automation, we will have another service to further improve our customers' experience."

"CloudFlare is so much more than just a content delivery network; the service optimizes web traffic, protects against online attacks, and provides real-time statistics," said Marco Houwen, CEO and co-founder of LuxCloud. "We are thrilled to be adding CloudFlare to our marketplace so that our partners can offer enhanced performance and increased security to their customers."

"The new Parallels Plesk Panel plugin is a great addition that allows for quick and easy deployment of CloudFlare on our shared web hosting, virtual and dedicated servers," said Gerardo Altman, CEO of Velocity Host. "CloudFlare allows us to get more out of every server and network connection in our cloud."

If you are a Parallels Plesk Panel service provider interested in signing up for the program, visit our Parallels page today: www.cloudflare.com/parallels

Syrian Internet access reestablished starting 1432 UTC

John Graham-Cumming — Sat, 01 Dec 2012 15:35:00 -0800

1432 UTC Syria has reestablished partial connectivity to the Internet. The following map of BGP connectivity shows Syria's 29386 network connected to multiple networks outside Syria.

This BGP map shows that connectivity with global Internet carriers PCCW and TATA has been reestiablished.

1557 UTC Syria has reconnected to Turk Telecom as well as PCCW and TATA.

The BGP map shows that with PCCW, TATA and Turk Telecom.

The following video shows the reestablishment of connectivity in two phases. At 1432 UTC Syria reconnected to PCCW and TATA; at 1557 UTC Syria reconnected to Turk Telecom.

1706 UTC Internal monitoring at CloudFlare shows that traffic is flowing from Syrian IP addresses onto the Internet. Both fixed and mobile IP addresses appear to be at least partially able to access the Internet.

2105 UTC Traffic to the CloudFlare network from Syrian IP addresses appears to have returned to levels seen prior to the shutdown. Almost immediately after the first links were reestablished we saw traffic levels jump back up.

2112 UTC There have been no routing changes for Syria since 1558 UTC. The Internet connectivity and traffic levels appear stable.

How Syria Turned Off the Internet

Matthew Prince — Thu, 29 Nov 2012 20:13:00 -0800

Today, 29 November 2012, between 1026 and 1028 (UTC), all traffic from Syria to the rest of the Internet stopped. At CloudFlare, we witnessed the drop off. We've spent the morning studying the situation to understand what happened. The following graph shows the last several days of traffic coming to CloudFlare's network from Syria.

Since the beginning of today's outage, we have received no requests from Syrian IP space. That is a more complete blackout than we've seen when other countries have been cut from the Internet (see, for example, Egypt where while most traffic was cut off some requests still trickled out).

The graph above shows two other incidents over the last week. On 25 November 2012 at approximately 0800 UTC we witnessed a 15 minute period during which Syrian traffic was cut to only 13% of normal levels. Again on 27 November 2012 at 0730 UTC, we saw a 15 minute period during which traffic dropped to only 0.2% of normal.

What Happened?

The Syrian Minister of Information is being reported as saying that the government did not disable the Internet, but instead the outage was caused by a cable being cut. Specifically: "It is not true that the state cut the Internet. The terrorists targeted the Internet lines, resulting in some regions being cut off." From our investigation, that appears unlikely to be the case.

To begin, all connectivity to Syria, not just some regions, has been cut. The exclusive provider of Internet access in Syria is the state-run Syrian Telecommunications Establishment. Their network AS number is AS29386. The following network providers typically provide connectivity from Syria to the rest of the Internet: PCCW and Turk Telekom as the primary providers with Telecom Italia and TATA for additional capacity. When the outage happened, the BGP routes to Syrian IP space were all simultaneously withdrawn from all of Syria's upstream providers. The effect of this is that networks were unable to route traffic to Syrian IP space, effectively cutting the country off the Internet.

Syria has 4 physical cables that connect it to the rest of the Internet. Three are undersea cables that land in the city of Tartous, Syria. The fourth is an over-land cable through Turkey. In order for a whole-country outage, all four of these cables would have had to been cut simultaneously. That is unlikely to have happened.

Watching the Shutdown Happen

One of our network engineers recorded the following video of network routes being withdrawn. Syrian Telecommunications (AS29386) is represented by the red dot in the middle of the video. The lines represent routes to the Syrian upstream providers.

(Video created with BGPlay by Roma Tre University)

Beginning at 1026 UTC, routes were withdrawn for PCCW. The routing shifted primarily to Turk Telekom. Routes to Telecom Italia and TATA were also withdrawn, but has less of an impact. Then, at 1028 UTC, routes were withdrawn for Turk Telekom. After that, Syria was effectively cut off from the Internet. (Note that the remaining path that appears to be present in the video is an anomaly. We have confirmed that it is not actually active.)

While we cannot know for sure, our network team estimates that Syria likely has a small number of edge routers. All the edge routers are controlled by Syrian Telecommunications. The systematic way in which routes were withdrawn suggests that this was done through updates in router configurations, not through a physical failure or cable cut.

What Syrians Were Surfing Before the Internet Was Turned Off

The last four sites on CloudFlare that received requests from Syria in the seconds before access was cut were:

fotoobook.com - a photo sharing blog
aliqtisadi.com - a Syrian news site
madinah.com - a Muslim-oriented social network
to2.xxx - a porn site (warning: not safe for work)

In other words, traffic from Syrians accessing the Internet in the moments before they were cut off from the rest of the world looks remarkably similar to traffic from any part of the world.

As we have posted about recently, we don't believe our role is to take sides in political conflicts. However, we do believe it is our mission to build a better Internet where everyone can have a voice and access information. It is therefore deeply troubling to the CloudFlare team when we see an entire nation cut off from the ability to access and report information. Our thoughts are with the Syrian people and we hope connectivity, and peace, will be quickly restored.

UPDATE: Syrian Internet access appears to be at least partially restored as of 1 December 2012 at 1432 UTC. We have confirmed both that the BGP routes are reestablished and traffic from both wired and mobile devices is flowing to CloudFlare's network. We've posted a blog post with more details: http://blog.cloudflare.com/syrian-internet-access-appears-partially-rees

Choosing a Two-Factor Authentication System

Matthew Prince — Wed, 28 Nov 2012 20:21:00 -0800

We've been thinking about how to best implement two-factor authentication to better protect our customers' accounts for quite some time now. When, about 6 months ago, my account was targeted by hackers the importance of a good account security became clear. However, as my hacking case illustrates, two-factor authentication alone is not a complete answer.

At CloudFlare, we considered a number of different ways to implement two-factor authentication. We considered building it ourselves and using Twilio, or another similar service, to send authentication codes via SMS to our customers' mobile phones. The problem with that strategy is that it passes the supposedly secure authentication code through your mobile carrier's less-than-secure network. And, again, if there's a lesson to be learned from my own hacking case it's that mobile providers' security is not always the most robust.

We also considered some sort of fob-based two-factor system. Unfortunately, these are generally very expensive and therefore prohibitive for us to offer all our customers. We also considered solutions like Google's Authenticator. It's a well thought out system, and we have a ton of respect for the Google team, but we were nervous about handing another key to identity over to a company whose primary business is search and advertising. Not to mention a bit of a bad taste after a flaw in Google's own implementation of their two-factor authentication system contributed to my hack.

TOTP: Open Authentication

The underlying algorithm used by several two-factor authentication schemes, including Google's, is open and known as the Time-based One-time Password Algorithm (TOTP). TOTP was specified by the Internet Engineering Task Force (IETF) under RFC 6238.

The mechanics of TOTP are relatively easy to understand. To begin, every TOTP user is issued a random key. Both the server and the client has a copy of this random key. TOTP assumes that both the server and the client can synchronize their clocks. When a user goes to login, the client takes the current timestamp to the previous 30-second interval. The client then combines the key and the timestamp.

This combined key and timestamp value is then run through a SHA hashing algorithm. SHA, like other cryptographic hashes, is a one-way algorithm. That the output cannot be used to derive the input. The SHA algorithm's output becomes the authentication code which the user can post to the server as part of the login process.

Since the server has the same random key for the user, and since the client and server clocks are synchronized, the server can also calculate an authentication code using the SHA algorithm. If the authentication code the server has received from the user matches the one the server derived itself then the user's identity can be confirmed.

What is powerful about this scheme is that if an attacker steals the authorization code then, within 30 seconds, it will be useless. This is typically insufficient time for the attacker to gain access to the account. This is particularly effective against phishing attacks, where an attacker convinces a user to reveal their login credentials on a fake website.

Authy

If the core algorithm for two-factor authentication is public, then the question comes down to who has the best implementation. We looked at several implementations and were particularly impressed by a company called Authy. The Authy team created a beautiful, simple, elegant app that implements TOTP. Their vision is not to create yet another app you need to install, but instead to create a single place from which you can manage all your TOTP two-factor authentication tokens.

We've been using the Authy app internally for all of our administrative systems for the last three months. The Authy team has worked with us to refine their app to make it as simple and elegant as possible. After months of our own tests, and spurred on by a phishing attack that targeted CloudFlare accounts, we decided to open up two-factor authentication as a feature for all our customers. If you're interested, you can read about how to implement it on your account with just a few easy steps.

But... I've Already Installed Google Authenticator on My Phone!

The biggest question we continue to get is why we didn't just use Google Authenticator, since a number of people already have it installed on their phones. Beyond the high-level concerns above, there were a number of technical concerns over security and ease of use that we believe made Authy a better choice.

First, with Google Authenticator if you lose your app there's no way you can revoke the app's tokens. This is probably the biggest security flaw with the Google Authenticator app. While it can be mitigated by password protecting your phone, the better solution is to allow the app to be deauthorized. Authy fixes this problem and allows you to revoke the app's token if you lose your phone. That's a big win for Authy over Google Authenticator.

Second, Google's Authenticator can get out of sync when you don't have network access, leaving you in the frustrating situation of not being able to access your account. Since all TOTP systems rely on the clock on your phone to match the clock on the server, if there's not a fairly precise match then there can be problems. I've experienced this myself when traveling and it can be frustrating. Authy has built a significant amount of logic into their app in order to keep clocks in sync even when you don't have network access.

Third, if you upgrade your phone, with Google's Authenticator you have to reestablish all your two-factor accounts from scratch. With Authy, all your accounts are synced, so when you upgrade and re-install Authy everything will be setup the way you expect it.

And there are a number of other well thought out details. Authy uses SHA-2 and 256-bit keys, where Google's Authenticator uses SHA-1 and 128-bit keys — likely not a huge deal for this application, but generally longer keys and more secure hashing protocols are better. When you wake your phone from sleep, Authy will always start with a code good for the next 30 seconds — it's a nice touch and removes the annoyance with Google's Authenticator of having to wait for the timer to expire if you don't have enough time to enter a code. And the interface is cleaner and just nicer to use than Google's.

But we get it. People don't like to have to install another app on their phones. The good news is the Authy team gets it too. They're adding support in the next few weeks for Google Authenticator tokens to their system as well. That way you can use Authy's great UI to access your Google codes through one app.

Two-factor Authentication Now Available

Kristin Tarr — Wed, 28 Nov 2012 17:05:00 -0800

With web performance and security being the core of CloudFlare, we are always looking for ways to improve not just our customers' website security, but their account security as well. Therefore, we are excited to now offer two-factor authentication for all CloudFlare accounts.

With two-factor authentication, our customers' accounts get an added layer of login security, ultimately adding another layer of security to their websites. We've been working on this feature for a while, and we are happy to announce that it's ready and available to all CloudFlare customers.

To make this feature happen, we worked with Authy, a startup who loves security too. Authy provides an easy-to-use, powerful two-factor authentication service. Their mission is to turn everyone's cell phone into a secure token. The Authy app works with iOS and Android devices and we're providing it free to all CloudFlare account holders. Here's how it works.

Easy additional account security

To turn two-factor authentication on, you simply log into your CloudFlare account, navigate to "My account" and select "two-factor authentication with Authy."

Once there, you will enter your mobile phone information and select "enable two-factor authentication."

You will then receive a text message (your provider's standard text messaging rates will apply). The text message includes a link to download the Authy app. The Authy app will ask you to enter your your mobile phone number. It will then text you a setup pin that you need to enter into the Authy app.

Once you receive your pin number via text, enter it into the Authy app. The Authy app will then be authorized and able to generate authentication tokens unique to your account. In the future, whenever you access your CloudFlare account you'll need three things: 1) your email address, 2) your password, and 3) your two-factor authentication token.

When you login to CloudFlare for the first time after enabling two-factor authentication, you will need to launch the Authy app on your phone. It will generate a unique, 7-digit authentication token. The authentication token is good for 30 seconds and then will change to a new token.

You can store your authentication for 14 days. If you login from an unrecognized device, or after your authentication expires, you'll need to open the Authy app and get your new authentication token for that device. The Authy app does not rely on you having network access, so you can retrieve your code even if your phone is not connected to the Internet.

If you ever your your phone or get a new one you can reassociate your account by following the reset instructions on Authy's website.

You don't need to enable two-factor authentication in order to continue to use CloudFlare. However, we're providing it to all CloudFlare customers free and we recommend it for everyone who wants additional account security.

Ceasefires Don't End Cyberwars

Matthew Prince — Tue, 27 Nov 2012 23:49:00 -0800

There is a significant conflict in the Middle East. As has been widely reported, along with the physical confrontation between the Israelis and Palestinians, there have been widespread cyber attacks. These cyber attacks have been launched against both sides in this conflict. At CloudFlare we have found ourselves in the unusual position of protecting websites of both Israeli and Palestinian organizations on the front lines. Among others, our customers include Israeli government sites as well as numerous Palestinian organizations.

The conflict that is going on right now may be the first true cyberwar. While previous conflicts have included the use of cyber attacks by one side or the other, in this case supporters of both sides appear to be launching cyber offensives. At CloudFlare, we've been caught in the cross fire. That's allowed us a unique vantage point to report on what we're seeing.

We've been following news about the conflict and monitoring the attacks against sites on both sides for the last week. On November 21, 2012 at 19:00 (GMT) a ceasefire was announced. The large scale physical attacks appear to have largely stopped along with the ceasefire. We wanted to see what happened to cyber attacks.

When Physical Attacks Stop, Cyber Attacks Start

Quite the opposite of stopping, there was a significant increase in cyber attacks against both sides websites that coincided with the ceasefire. The following chart aggregates data from a number of sites on both sides of the conflict. The dotted line about 3/4 of the way along the timeline indicates the point of time the ceasefire was declared. We have intentionally obscured whether the attacks were targeting sites supporting Israel or Palestine, but I can say that we saw significant upticks in attacks targeting both sides in the conflict.

This graph focuses specifically on what are known as Layer 7 attacks. These are application-layer attacks, and different than some of the Layer 3/4 attacks we have discussed before. Layer 7 attacks tend to be smaller in volume but often harder to defend against using traditional DDoS scrubbing services. CloudFlare's service is able to absorb these attacks and ensure that only legitimate requests are sent to a web server.

It is important to be clear. Nothing we've seen allows us to make a claim toward the attribution of the source of these attacks. CloudFlare's network is like a flack jacket, not like a machine gun. We shield sites from the attacks we see, but we don't spend a lot of time trying to determine the motives of the attackers. It is not correct to say that this data proves one side is attacking the other. In fact, third party organizations like Anonymous, which are not directly affiliated with Palestinians, have claimed responsibility for many of the attacks targeting Israeli sites, and several "vigilante hackers," who are not directly affiliated with Israel, have claimed responsibilityfor attacks against some Palestinian sites.

The Politics of Being a Proxy

We've received criticism from supporters on both sides asking how we can be supporting the other. To be clear, we are not supporting either side. Resolving the difficult political questions of a conflict like this is way above our pay grade. We are proud, however, that in spite of withering cyber attacks CloudFlare has kept both sides' websites online.

The Internet is one of the greatest inventions in human history because it allows anyone to reach a global audience. CloudFlare's goal is to power a better Internet. While that will inherently mean we will increasingly find ourselves in difficult situations like this one, we will continue to be guided by the principle that it is not our role to decide whether one idea or another is correct, but instead to ensure that all ideas can find equal footing online.

SEO and your website

Kristin Tarr — Mon, 19 Nov 2012 21:26:00 -0800

*We get a lot of questions from our customers about CloudFlare and how we impact SEO. So when SEO.com signed up for CloudFlare, I thought it would be a great opportunity to talk to an expert to get the scoop on all things SEO. I was fortunate enough to connect with Derek Perkins, Vice President of Technology at SEO.com. With more than 12 years of industry experience, Derek provided his insight on SEO in general, debunked some of the myths out there, and gave us his take on what really works, and what doesn't, when it comes to SEO and your site.
*

CF - What are the top three tips you can offer for website owners looking to improve their SEO?

DP - Step one - use WordPress and WordPress SEO. For most website owners, a Content Management System is key. The WordPress platform would be my first choice, you don't have to deal with structure of a Wordpress site to make sure it's easily searchable and findable. WordPress ranks high, especially if you activate WordPress SEO by Yoast. A combination of those two alone put you a long way ahead of where smaller business are. Correct structure is a good thing.

Step two - Focus on great content. Sporadic posting is never going to yield a tangible output. The more Google changes their algorithms, the more likely you will get ranked lower if you're not posting often. Just posting frequently however isn't enough. Content has always risen to the top of rankings, and as search engines mature, they are continuing to increase the signal to noise ratio. Posting great content regularly is the key to SEO success.

Step three - find good website hosting that will be elastic. Great content that gets picked up on TechCrunch, Digg, Reddit - any viral site - is going to see heavy spikes in traffic. If you're on a cheap hosting plan you often won't be able to scale to meet demands.

Actually, one of the first things I do is recommend CloudFlare. I love the CDN and scalability, it takes load off of the server so you don't have to worry so much about load spikes.

CF - What are some of the misperceptions with SEO?

DP - A big misperception about SEO is the idea that you have to somehow change your writing or write things that are for search engines instead of humans. That's not the case. A lot of people also say you have to have unnaturally high keyword density and that's the only way you're going to rank. That's not good, it's harmful. Google sees that as if you're writing it specifically for SEO. Search engines try to read content as if they were human. If the content doesn't flow well or read well for a human, chances are it's not going to read well for a search engine or spider.

SEO is all about having good content. Write content you and others would like to read. It is more likely be shared socially, bringing more people to your site, and more people will link back to your site, growing your online presence.

When people think of SEO they tend to focus on the 10 percent that's the little tweaks that SEO companies can do for you, whereas the bulk of the value comes from writing good content.

CF - Web properties care a lot about SEO, what are some good resources for site owners looking to better their SEO rank?

DP - There are a number of places on the web that have good SEO forums. We have a link on our own site,www.SEO.com/forums, that links to a number of the best forums out there. Another great resource is SEOmoz, it's a great place for website owners to start.

CF - What is one thing site owners should be doing to improve their SEO, but probably aren't?

DP - Number one thing that people don't do - they don't have any sort of targets. Content is king, but a lot of it is knowing what specific content is going to be most valuable to them. You can write about two different things that are both interesting, exciting and relevant to your audience, but one is relevant to maybe 10 searches a month, whereas one is relevant to 10,000 searches a month. Having an idea of what pages or blog posts or keywords you're targeting with each will help you tailor the content.

CF - What are some of the things site owners do that might negatively impact SEO?

DP - Picking the wrong page titles and/or having a malformed HTML structure. There's a lot of SEO weight on title and header tags, you need to have your page title similar to whatever is in your H1 tag. A lot of site owners out there don't have header tags or have them set-up correctly. Even if they have a good title, the title might not be in the HTML header tag. The title and headers help both search engines and more importantly humans identify the focus of the page.

CF - How has your industry changed in the last five years?

DP - I think SEO has a bit of a stigma because of old tactics that people used to use. You used to be able to immediately rank for SEO by using tricks like white text on a white background and various other tactics to gain the system. Even just recently Google has released things like Penguin, making it harder and harder to game the system. It changes how SEO agencies function, shifting the focus from link building to strategic content driven approaches. That has driven a proliferation of socially shareable content like infographics.

CF - You've seen the Google vs Bing commercial search challenge. The commercial claims people choose Bing 2 to 1 over Google. Do you think that's right? What are your thoughts?

DP - Personally, I occasionally use Bing, but I tend to go back to Google. I took the test myself and Bing won 3 to 2, but I felt like the stripped down result pages weren't a perfect test.

CF - Google, Yahoo! and Bing are huge competitors in the search space. What are your thoughts on each? Do any of them stand out as being front runners in the near future?

DP - Bing has been gaining ground, Yahoo!'s results are powered by Microsoft, so Bing and Yahoo! will both show same results. The big two are definitely Google and Bing. You can't ignore Bing when you're tracking rankings, but they are definitely playing second fiddle to Google at this point.

For a long time Google has provided the best search rankings. Whether or not Bing has closed the gap on search, they have an uphill battle. People are used to Google, it's becoming part of the English language. I doubt that anyone outside of Microsoft headquarters has ever said "I don't know the answer, let me go Bing it."

CloudFlare...in his own words

I've been using CloudFlare for over a year now. I had a personal website calledsoccerreviews.com. I built it up to have significant traffic and I was really having server load issues, in addition to having been hacked twice. Because of that, security and scalability was very important to me.

I tried CloudFlare on that site and I now have it on 30 other sites. I have yet to have any of those sites compromised, which has been fantastic.

Once I joined SEO.com, I put us on CloudFlare. One feature I really like about CloudFlare is Rocket Loader. It combines all my javascript files and speeds them up, and they aren't all being downloaded separately, decreasing download time.

As for the impact to SEO, bounce rate plays a very important role in how Google does their rankings - they see that as a human factor in SEO. If someone immediately jumps back to Google, it's obviously not a good human source. A fast site that's always online is sure to help your rankings with lower bounce rates, and having CloudFlare helps to make this possible.

Do you want to work with Go?

John Graham-Cumming — Sun, 18 Nov 2012 00:35:00 -0800

It's no secret that CloudFlare has adopted Go for some production systems; we've written about our use of Go in the past. But over time it's become clear to us that Go is an important language for the sort of high-performance, highly-concurrent software we have to write. And the Go package library contains pretty much everything we need to write small, fast programs (and write them quickly).

So, Go has become an important part of CloudFlare Engineering.

And because of that we are actively hiring for people who know Go or want to learn it. We'll soon be open sourcing some of our Go programs and want to find more people to work on our Go code base.

We're currently using Go for PKI tools, our Railgun web optimizer, a new high-performance DNS server and a curl-like tool for SPDY. And we've hosted the GoSF meetup in the past.

So, if you're interested in writing Go code, contributing to Golang itself and having your code go into production against billions of page views per day, get in touch.

The many sites of CloudFlare

Kristin Tarr — Fri, 09 Nov 2012 19:32:00 -0800

Each day I get to trade notes with CloudFlare customers. I'm constantly amazed by the diversity of businesses that use the service from around the world. I wanted to share some stories from some of our customers about their experience on CloudFlare.

Sporting Events
The San Francisco Marathon

The San Francisco Marathon is a popular race. With 24,000 runners competing in five events, the San Francisco Marathon has runners of all ages, from all over the globe. The website is used for registration, to organize runners during the actual event weekend and to track each runners race time.

"I really love how simple it was to set up and use. CloudFlare is an easy to use CDN service with offerings at all levels," said said Laura Baalman, an independent IT consultant to the San Francisco Marathon website. "I would recommend anyone with a website, especially those that get big changes in traffic, to give it a try." Read full case study here.

Social Media Storify

Storify helps its users tell stories by curating social media. Storify sees huge spikes of traffic during major events around the world, like Hurricane Sandy or the U.S. presidential election.

"Thanks to services like CloudFlare, we can scale Storify to more than 24 million story views per month with only 3 engineers," said Xavier Damman, co-founder and CEO of Storify.

During Hurricane Sandy, CloudFlare saved Storify more than 75 million requests and over 470 GB of bandwidth.

"Having CloudFlare save these requests has enabled us to stay online and keep up with the surges in traffic due to significant news sharing during Hurricane Sandy," said Xavier. Read full blog post here.

eCommerce Runa Tea

Runa produces guayusa tea sourced from the Ecuadorian Amazon. Guayusa balances as much caffeine as one cup of coffee with twice the antioxidants of green tea creating. The drink has gained popularity around the world, including among the CloudFlare team. It is always well-stocked in our kitchen.

"We love CloudFlare and recommend it to anyone looking for amazing results," said Anna Premo Director of Marketing at Runa.

Search Engine Optimization (seo.com

Seo.com is an experienced search marketing firm dedicated to making websites more visible online and profitable.

"I've been using CloudFlare for over a year now. I had a personal website which began to see significant traffic and I was really having server load issues. In addition, the site had been hacked twice. Because of that, security and scalability was important to me," said Derek Perkins, VP of Technology at SEO.com. "I tried CloudFlare on that website and I now have it on 30 other sites, including seo.com. I have yet to have any of those sites compromised, which has been fantastic."

Live, online customer support
Zopim

Zopim is an award winning, cloud based live chat platform that makes it easy for businesses to deliver fast customer service online. Businesses who sign up with Zopim are fanatics when it comes to delivering customer wow. More than 50,000 businesses use Zopim Live Chat to chat with their online customers everyday. Zopim is based in Singapore.

"Thanks to CloudFlare's CDN, our chat widget began loading extremely fast no matter where it was being loaded from," said Qing. "By caching and serving the static images on our widget, CloudFlare accelerated the byte load and reduced the widget loading time by at least 50%. Even our customers could feel the significant improvement in speed." Read full case study here.

Russian Photographer
Ekaterina Dokuchaeva

Ekaterina Dokuchaeva is a talented photographer from Russia with her own unique style and creativity. Despite the fact that Ekaterina is just 21 years old, she has already done exquisite art creations and continues to work hard in the photo-industry.

"There's an introduction song playing on the backround of the landing page. This song made page load speed very slow and visitors constantly complained that the site was not fast enough," said Dokuchaev Konstantin, who manages the site. "Once I enabled CloudFlare's CDN and compression features to our website, the page load time immediately reduced to three seconds in all browsers. It was great solution of our problem. I wasn't expecting such a noticable change!"

Do you have a story you'd like to share about CloudFlare? I'd like to hear it and feature you on our blog! testimonials@cloudflare.com

How to choose a payment platform: a CloudFlare + Braintree meetup

Kristin Tarr — Thu, 08 Nov 2012 21:45:00 -0800

We are looking forward to co-hosting a meetup next week on Wednesday, November 14, with Braintree, the fastest growing payments platform for online and mobile commerce.

This meetup will:

Answer all of your questions about what to look for as you decide which payments provider is best for you
Debunk payment myths
Give insight into what the future of payments may hold

The speakers will give an overview of various payment solutions for different business types and stage of growth. From startups to SMBs - this meetup will answer your questions and give guidelines for best practices in choosing an online payment service.

Braintree speakers will be Charity Kittler, who directs training at Braintree, and Jenna Wyer, VP of Sales for Braintree.

There will be pizza, beer and good times, hope to see you there!

Doors open at 6:30PM, presentation starts at 7PM.

RSVP for the Braintree and CloudFlare meetup

CloudFlare London Meetup

John Graham-Cumming — Wed, 07 Nov 2012 11:06:00 -0800

We're having a CloudFlare London Meetup tomorrow night (Thursday, November 8). Three CloudFlare employees will be there: Dane, John and Ian. Come have a beer and exchange ideas with the CloudFlare crew at the Old Coffee House on Beak St.

You don't need to be a CloudFlare customer to take part, but sign up here.

Why Google Went Offline Today and a Bit about How the Internet Works

Tom Paseka — Tue, 06 Nov 2012 09:09:00 -0800

Today, Google's services experienced a limited outage for about 27 minutes over some portions of the Internet. The reason this happened dives into the deep, dark corners of networking. I'm a network engineer at CloudFlare and I played a small part in helping ensure Google came back online. Here's a bit about what happened.

At around 6:24pm PST / 02:24 UTC (5 Nov. 2012 PST / 6 Nov. 2012 UTC), CloudFlare employees noticed that Google's services were offline. We use Google Apps for things like email so when we can't reach their servers the office notices quickly. I'm on the Network Engineering team so I jumped online to figure out if the problem was local to us or global.

Troubleshooting

I quickly realised that we were unable to resolve all of Googles services — or even reach 8.8.8.8, Googles public DNS server — so I started troubleshooting DNS.

$ dig +trace google.com

Here's the response I got when I tried to reach any of Google.com's name servers:

google.com.                172800        IN        NS        ns2.google.com.google.com.                172800        IN        NS        ns1.google.com.google.com.                172800        IN        NS        ns3.google.com.google.com.                172800        IN        NS        ns4.google.com.;; Received 164 bytes from 192.12.94.30#53(e.gtld-servers.net) in 152 ms;; connection timed out; no servers could be reached

The fact that no servers could be reached means something was wrong. Specifically, it meant that from our office network we were unable to reach any of Googles DNS servers.

I started to look at the network layer, see if that's where the problems lay.

PING 216.239.32.10 (216.239.32.10): 56 data bytesRequest timeout for icmp_seq 092 bytes from 1-1-15.edge2-eqx-sin.moratelindo.co.id (202.43.176.217): Time to live exceeded

That was curious. Normally, we shouldn't be seeing an Indonesian ISP (Moratel) in the path to Google. I jumped on one of CloudFlare's routers to check what was going on. Meanwhile, others reports from around the globe on Twitter suggested we weren't the only ones seeing the problem.

Internet Routing

To understand what went wrong you need to understand a bit about how networking on the Internet works. The Internet is a collection of networks, known as "Autonomous Systems" (AS). Each network has a unique number to identify it known as AS number. CloudFlare's AS number is 13335, Google's is 15169. The networks are connected together by what is known as Border Gateway Protocol (BGP). BGP is the glue of the Internet — announcing what IP addresses belong to each network and establishing the routes from one AS to another. An Internet "route" is exactly what it sounds like: a path from the IP address on one AS to an IP address on another AS.

BGP is largely a trust-based system. Networks trust each other to say which IP addresses and other networks are behind them. When you send a packet or make a request across the network, your ISP connects to its upstream providers or peers and finds the shortest path from your ISP to the destination network.

Unfortunately, if a network starts to send out an announcement of a particular IP address or network behind it, when in fact it is not, if that network is trusted by its upstreams and peers then packets can end up misrouted. That is what was happening here.

I looked at the BGP Routes for a Google IP Address. The route traversed Moratel (23947), an Indonesian ISP. Given that I'm looking at the routing from California and Google is operating Data Centre's not far from our office, packets should never be routed via Indonesia. The most likely cause was that Moratel was announcing a network that wasn't actually behind them.

The BGP Route I saw at the time was:

tom@edge01.sfo01> show route 216.239.34.10                          inet.0: 422168 destinations, 422168 routes (422154 active, 0 holddown, 14 hidden)+ = Active Route, - = Last Active, * = Both216.239.34.0/24    *[BGP/170] 00:15:47, MED 18, localpref 100                      AS path: 4436 3491 23947 15169 I                    > to 69.22.153.1 via ge-1/0/9.0

Looking at other routes, for example to Google's Public DNS, it was also stuck routing down the same (incorrect) path:

tom@edge01.sfo01> show route 8.8.8.8 inet.0: 422196 destinations, 422196 routes (422182 active, 0 holddown, 14 hidden)+ = Active Route, - = Last Active, * = Both8.8.8.0/24         *[BGP/170] 00:27:02, MED 18, localpref 100                      AS path: 4436 3491 23947 15169 I                    > to 69.22.153.1 via ge-1/0/9.0

Route Leakage

(Image Credit: The Simpsons)

Situations like this are referred to in the industry as "route leakage", as the route has "leaked" past normal paths. This isn't an unprecedented event. Google previously suffered a similar outage when Pakistan was allegedly trying to censor a video on YouTube and the National ISP of Pakistan null routed the service's IP addresses. Unfortunately, they leaked the null route externally. Pakistan Telecom's upstream provider, PCCW, trusted what Pakistan Telecom's was sending them and the routes spread across the Internet. The effect was YouTube was knocked offline for around 2 hours.

The case today was similar. Someone at Moratel likely "fat fingered" an Internet route. PCCW, who was Moratel's upstream provider, trusted the routes Moratel was sending to them. And, quickly, the bad routes spread. It is unlikely this was malicious, but rather a misconfiguaration or an error evidencing some of the failings in the BGP Trust model.

The Fix

The solution was to get Moratel to stop announcing the routes they shouldn't be. A large part of being a network engineer, especially working at a large network like CloudFlare's, is having relationships with other network engineers around the world. When I figured out the problem, I contacted a colleague at Moratel to let him know what was going on. He was able to fix the problem at around 2:50 UTC / 6:50pm PST. Around 3 minutes later, routing returned to normal and Google's services came back online.

Looking at peering maps, I'd estimate the outage impacted around 3–5% of the Internet's population. The heaviest impact will have been felt in Hong Kong, where PCCW is the incumbent provider. If you were in the area and unable to reach Google's services around that time, now you know why.

Building a Better Internet

This all is a reminder about how the Internet is a system built on trust. Today's incident shows that, even if you're as big as Google, factors outside of your direct control can impact the ability of your customers to get to your site so it's important to have a network engineering team that is watching routes and managing your connectivity around the clock. CloudFlare works every day to ensure our customers get the optimal possible routes. We look out for all the websites on our network to ensure that their traffic is always delivered as fast as possible. Just another day in our ongoing efforts to #savetheweb.

Update:Tuesday, November 6 11:00am PST

Moratel says the issue was caused by an unexpected hardware failure, causing this abnormal condition. This was not a malicious attempt. Moratel immediately shutdown the BGP peering with Google after contact was made while the hardware failure was being looked into.

Thanks for reading all the way to the end. If you enjoyed this post, take a second to learn more about CloudFlare or nominate us for the 2012 Crunchie Award for Best Technical Innovation.

CloudFlare's Global Reach

Matthew Prince — Fri, 02 Nov 2012 20:19:00 -0700

CloudFlare is based in San Francisco, California, USA but we serve a global audience. Every minute of every day we send and receive traffic from nearly all of the world's networks. Beyond traffic to our service, our customers also come from around the world. To better understand where our customers come from, Algin Martin on our customer support team pulled data from a day of logins to CloudFlare.com in order to produce the map at the top of this post.

The results show CloudFlare's customers literally circle the globe, with a relatively even distribution between the Americas, Europe, and Asia. Turns out that making sure your website is fast and safe is a universal problem that resonates regardless of where you are in the world.

None of this surprised Jenn on our team who, in just the last 6 weeks, sent t-shirts to CloudFlare customers in 67 different countries. And she's sending out more every day. Thanks to everyone around the world who have helped make CloudFlare a global community. We're all working together to build a better Internet.

CloudFlare Works with GlobalSign to Make SSL Faster Across the Web

Matthew Prince — Thu, 01 Nov 2012 16:34:00 -0700

Earlier this week we announced how CloudFlare enabled OCSP stapling in order to improve our customers' SSL performance. OCSP stapling is awesome and improves SSL performance by as much as 30%. However, it is limited to browsers that support OCSP stapling and only benefits CloudFlare's customers. So, until every browser vendor updates to support OCSP stapling and until every website uses CloudFlare, we wanted to see if we could do something else to improve SSL performance across the web.

GlobalSign Partnership

CloudFlare has worked with GlobalSign since we first launched in September 2010. Prior to that we surveyed nearly every certificate authority in an effort to find one that was forward thinking enough to support what we needed. GlobalSign has been a terrific partner and is shaking up what has been a commodity industry.

Several months ago, GlobalSign approached us to talk about SSL performance. Their goal was simple: become the fastest SSL provider on the Internet. As I've written about before, whenever you visit a website over a HTTPS connection your browser has to perform a check to see if the certificate has been revoked. Depending on your browser, these checks are either over the CRL (Certificate Revocation List) or OCSP (Online Certificate Status Protocol) protocol. In either case, they require a request be sent back to the certificate authority and to get a response before content is downloaded. In other words, CRL and OCSP requests inherently slow down HTTPS performance.

The amount that these checks slow down performance varies depending on the certificate authority. On average, across the industry, a typical OCSP or CRL response time can be 500ms. That's half a second. In other words, every time you visit a site over HTTPS, you waste half a second waiting for the SSL check to complete. Talking with GlobalSign we realized we could do something about that.

Now Saving 1.5 Years Worth of Time a Day

This morning we officially announced our work with GlobalSign to make their CRL and OCSP requests the fastest on the Internet. GlobalSign's SSL checks (OCSP and CRL GET and POST requests) are now served from our cache across CloudFlare's global infrastructure. The results have been awesome. The requests that previously averaging around 500ms are now under 100ms. At GlobalSign's scale, that means we're now saving the web about a year and a half of time every day that people would have otherwise spent waiting for web pages to load. That's crazy.

This improvement accrues to sites using GlobalSign SSL certificates, regardless of whether the sites themselves are running on CloudFlare's network. Getting more sites using SSL is critical for increasing web security and promoting new performance protocols like SPDY. If you are choosing a CA, typically a commodity decision, now there's a good reason to pick GlobalSign over the other choices: they will ensure your site is as fast as possible over HTTPS. Put simply, GlobalSign is now the fastest certificate authority in the world, and nearly 3x as fast as Symantec/Verisign.

CloudFlare's mission is to power a faster, safer Internet so working with GlobalSign to make SSL as fast as possible has been a perfect fit. Our hope is that other certificate authorities will follow GlobalSign's lead and spend the time to optimize their SSL checks for optimal performance. As an added bonus, we've also helped GlobalSign be the first certificate authority to have their SSL checks be available over IPv6. This is all part of our efforts to help build a better Internet. As we like to tweet: #savetheweb.

What Happens When a Hurricane Hits the Web

Matthew Prince — Wed, 31 Oct 2012 01:04:00 -0700

(Photo credit: Hy Chalmé/Instagram)

Now that Hurricane Sandy has passed and the flood waters have begun to recede, we wanted to recap what we saw over the last 24 hours across the CloudFlare network.

CloudFlare's Infrastructure

Our network is designed to survive hurricanes and other natural disasters, so we were confident even if some of our data centers that were in the hurricane's path failed, traffic would immediately be transferred to the next closest facility. That said, our preference is always that all our data centers remain online and able to continue to serve traffic.

Yesterday morning our ops team met to plan for the potential loss of our facilities in Newark, NJ, which we refer to by the airport code EWR, and potentially Ashburn, VA, which we refer to by the airport code IAD. Our equipment is located in an Equinix facility in both locations and we confirmed that they had taken steps to ensure their systems were tested and as hurricane-ready as they could be.

Data centers are setup so that, if power from the grid is disrupted, they switch to stored backup power until generators can kick in. In EWR, power is stored in what are, effectively, a series of car batteries. Enough power is stored in the batteries that the data center can continue to run without a new source of power for several minutes. The diesel generators are setup to kick in within that time period, usually less than a minute after a power failure is detected. The generators are intended to be able to power the facilities indefinitely so long as there is sufficient fuel. Most of the data centers from which we operate worldwide are considered "critical infrastructure" and, during an emergency, they are second in line, behind only hospitals, for delivery of diesel fuel.

The generators at our EWR facility would get a test as the storm passed overhead. At 01:31 EDT, several hours after the storm had made landfall, we received notice EWR had lost grid power. As designed, power was immediately transferred to the batteries and then to the generators. The incident description read: "Equinix IBX reported a utility power disturbance and transferred customer loads to generator power. No customers have been impacted and Site Staff reports that sufficient fuel supplies are available. Next update will be when a significant change to the situation occurs." Our systems continued to run and we did not detect any power surge or interruption.

At 08:32 EDT, we received notice that one of the EWR generators had failed: "Equinix IBX reports customer loads are on generator power, however they have a loss of redundancy do to the failure of generator 4. Engineers are investigating the issue. Next update will be when a significant change to the situation occurs." Data centers are designed for redundancy, so losing a single generator would not cause a power loss. Our systems continued to function as normal and the functional generators continued to power our equipment throughout the day. At 19:09 EDT, 11 and a half hours after the generator originally kicked in, we received notice that grid power had been restored: "Equinix IBX AMFO reports that utility power has been restored."

Elsewhere on the Internet

While we were fortunate that all of CloudFlare's facilities stayed online, other data centers and networks experienced issues. Around 02:10 EDT, our network ops team noticed a change in routing from traffic that usually transited via Level(3)'s Yellow/Atlantic Crossing-2 (AC-2) undersea cable. The cable runs from Bude, United Kingdom to Bellport, New York. While routing changed, it did not impact our customers and our network routed around the problem. We later confirmed with other network operators that AC-2 had experienced a failure.

Several regional data centers experienced outages which caused interruption to their customers' sites. In some cases, our customers had their origin data centers knocked offline. When this happens, CloudFlare's Always Online functionality kicks in and continues to serve a static version of the site until the origin is restored. The graph below illustrates the deviation from normal of websites that have triggered Always Online. At the height of the storm, beginning around 22:30 EDT and lasting until 00:30 EDT, we were 2.5 standard deviations above normal in terms of the sites on our network whose origin servers were offline but we were serving static copies of their sites.

One thing that was somewhat surprising is that traffic to our EWR and IAD data centers dropped less than 1% versus normal operations on a regular Monday night. We had speculated that with power outaged affecting a large number of homes and businesses throughout the Northeastern United States, traffic to the data centers would have been more impacted. Our speculation is that while fewer people may have been online, those that still had connectivity were glued to their computers and surfing more than usual.

Everyone at CloudFlare's thoughts are with the people of the Northeastern United States as they begin the process of cleaning up from this extremely destructive storm. Thanks to the police, fire fighters, rescue workers, and the teams on the ground in the region that kept the lights on and allowed us to continue to operate from the region uninterrupted.

Deep Inside a DNS Amplification DDoS Attack

Matthew Prince — Tue, 30 Oct 2012 07:54:00 -0700

A few weeks ago I wrote about DNS Amplification Attacks. These attacks are some of the largest, as measured by the number of Gigabits per second (Gbps), that we see directed toward our network. For the last three weeks, one persistent attacker has been sending at least 20Gbps twenty-four hours a day as an attack against one of our customers.

That size of an attack is enough to cripple even a large web host. For CloudFlare, the nature of our network means that the attack, which gets diluted across all of our global data centers, doesn't cause us harm. Even from a cost perspective, the attack doesn't end up adding to our bandwidth bill because of the way in which we're charged for wholesale bandwidth.

We buy a lot of bandwidth and we pay for the higher of our ingress (in-bound) or egress (out-bound) averaged over a month. Since we act as a caching proxy, under normal circumstances egress always exceeds ingress. When there's an attack, the two lines get closer together but rarely is an attack large enough to add to our overall bandwidth costs.

Given that the latest attack wasn't impacting us or any of our customers, we decided to let it run for a while and see what we could learn.

Amplification Attacks

DNS Amplification Attacks are a way for an attacker to magnify the amount of bandwidth they can target at a potential victim. Imagine you are an attacker and you control a botnet capable of sending out 100Mbps of traffic. While that may be sufficient to knock some sites offline, it is a relatively trivial amount of traffic in the world of DDoS. In order to increase your attack's volume, you could try and add more compromised machines to your botnet. That is becoming increasingly difficult. Alternatively, you could find a way to amplify your 100Mbps into something much bigger.

The original amplification attack was known as a SMURF attack. A SMURF attack involves an attacker sending ICMP requests (i.e., ping requests) to the network's broadcast address (i.e., X.X.X.255) of a router configured to relay ICMP to all devices behind the router. The attacker spoofs the source of the ICMP request to be the IP address of the intended victim. Since ICMP does not include a handshake, the destination has no way of verifying if the source IP is legitimate. The router receives the request and passes it on to all the devices that sit behind it. All those devices then respond back to the ping. The attacker is able to amplify the attack by a multiple of how ever many devices are behind the router (i.e., if you have 5 devices behind the router then the attacker is able to amplify the attack 5x, see the diagram below).

SMURF attacks are largely a thing of the past. For the most part, network operators have configured their routers to not relay ICMP requests sent to a network's broadcast address. However, even as that amplification attack vector has closed, others remain wide open.

DNS Amplification

There are two criteria for a good amplification attack vector: 1) query can be set with a spoofed source address (e.g., via a protocol like ICMP or UDP that does not require a handshake); and 2) the response to the query is significantly larger than the query itself. DNS is a core, ubiquitous Internet platform that meets these criteria and therefore has become the largest source of amplification attacks.

DNS queries are typically transmitted over UDP, meaning that, like ICMP queries used in a SMURF attack, they are fire and forget. As a result, their source attribute can be spoofed and the receiver has no way of determining its veracity before responding. DNS also is capable of generating a much larger response than query. For example, you can send the following (tiny) query (where x.x.x.x is the IP of an open DNS resolver):

dig ANY isc.org @x.x.x.x

And get back the following (gigantic) response:

; <<>> DiG 9.7.3 <<>> ANY isc.org @x.x.x.x
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5147
;; flags: qr rd ra; QUERY: 1, ANSWER: 27, AUTHORITY: 4, ADDITIONAL: 5

;; QUESTION SECTION:
;isc.org.                        IN        ANY

;; ANSWER SECTION:
isc.org.                4084       IN        SOA          ns-int.isc.org. hostmaster.isc.org. 2012102700 7200 3600 24796800 3600
isc.org.                4084       IN        A            149.20.64.42
isc.org.                4084       IN        MX           10 mx.pao1.isc.org.
isc.org.                4084       IN        MX           10 mx.ams1.isc.org.
isc.org.                4084       IN        TXT          "v=spf1 a mx ip4:204.152.184.0/21 ip4:149.20.0.0/16 ip6:2001:04F8::0/32 ip6:2001:500:60::65/128 ~all"
isc.org.                4084       IN        TXT          "$Id: isc.org,v 1.1724 2012-10-23 00:36:09 bind Exp $"
isc.org.                4084       IN        AAAA         2001:4f8:0:2::d
isc.org.                4084       IN        NAPTR        20 0 "S" "SIP+D2U" "" _sip._udp.isc.org.
isc.org.                484        IN        NSEC         _kerberos.isc.org. A NS SOA MX TXT AAAA NAPTR RRSIG NSEC DNSKEY SPF
isc.org.                4084       IN        DNSKEY       256 3 5 BQEAAAAB2F1v2HWzCCE9vNsKfk0K8vd4EBwizNT9KO6WYXj0oxEL4eOJ aXbax/BzPFx+3qO8B8pu8E/JjkWH0oaYz4guUyTVmT5Eelg44Vb1kssy q8W27oQ+9qNiP8Jv6zdOj0uCB/N0fxfVL3371xbednFqoECfSFDZa6Hw jU1qzveSsW0=
isc.org.                4084       IN        DNSKEY       257 3 5 BEAAAAOhHQDBrhQbtphgq2wQUpEQ5t4DtUHxoMVFu2hWLDMvoOMRXjGr hhCeFvAZih7yJHf8ZGfW6hd38hXG/xylYCO6Krpbdojwx8YMXLA5/kA+ u50WIL8ZR1R6KTbsYVMf/Qx5RiNbPClw+vT+U8eXEJmO20jIS1ULgqy3 47cBB1zMnnz/4LJpA0da9CbKj3A254T515sNIMcwsB8/2+2E63/zZrQz Bkj0BrN/9Bexjpiks3jRhZatEsXn3dTy47R09Uix5WcJt+xzqZ7+ysyL KOOedS39Z7SDmsn2eA0FKtQpwA6LXeG2w+jxmw3oA8lVUgEf/rzeC/bB yBNsO70aEFTd
isc.org.                4084       IN        SPF          "v=spf1 a mx ip4:204.152.184.0/21 ip4:149.20.0.0/16 ip6:2001:04F8::0/32 ip6:2001:500:60::65/128 ~all"
isc.org.                484        IN        RRSIG        NS 5 2 7200 20121125230752 20121026230752 4442 isc.org. oFeNy69Pn+/JnnltGPUZQnYzo1YGglMhS/SZKnlgyMbz+tT2r/2v+X1j AkUl9GRW9JAZU+x0oEj5oNAkRiQqK+D6DC+PGdM2/JHa0X41LnMIE2NX UHDAKMmbqk529fUy3MvA/ZwR9FXurcfYQ5fnpEEaawNS0bKxomw48dcp Aco=
isc.org.                484        IN        RRSIG        SOA 5 2 7200 20121125230752 20121026230752 4442 isc.org. S+DLHzE/8WQbnSl70geMYoKvGlIuKARVlxmssce+MX6DO/J1xdK9xGac XCuAhRpTMKElKq2dIhKp8vnS2e+JTZLrGl4q/bnrrmhQ9eBS7IFmrQ6s 0cKEEyuijumOPlKCCN9QX7ds4siiTIrEOGhCaamEgRJqVxqCsg1dBUrR hKk=
isc.org.                484        IN        RRSIG        MX 5 2 7200 20121125230752 20121026230752 4442 isc.org. VFqFWRPyulIT8VsIdXKMpMRJTYpdggoGgOjKJzKJs/6ZrxmbJtmAxgEu /rkwD6Q9JwsUCepNC74EYxzXFvDaNnKp/Qdmt2139h/xoZsw0JVA4Z+b zNQ3kNiDjdV6zl6ELtCVDqj3SiWDZhYB/CR9pNno1FAF2joIjYSwiwbS Lcw=
isc.org.                484        IN        RRSIG        TXT 5 2 7200 20121125230752 20121026230752 4442 isc.org. Ojj8YCZf3jYL9eO8w4Tl9HjWKP3CKXQRFed8s9xeh5TR3KI3tQTKsSeI JRQaCXkADiRwHt0j7VaJ3xUHa5LCkzetcVgJNPmhovVa1w87Hz4DU6q9 k9bbshvbYtxOF8xny/FCiR5c6NVeLmvvu4xeOqSwIpoo2zvIEfFP9deR UhA=
isc.org.                484        IN        RRSIG        AAAA 5 2 7200 20121125230752 20121026230752 4442 isc.org. hutAcro0NBMvKU/m+2lF8sgIYyIVWORTp/utIn8KsF1WOwwM2QMGa5C9 /rH/ZQBQgN46ZMmiEm4LxH6mtaKxMsBGZwgzUEdfsvVtr+fS5NUoA1rF wg92eBbInNdCvT0if8m1Sldx5/hSqKn8EAscKfg5BMQp5YDFsllsTauA 8Y4=
isc.org.                484        IN        RRSIG        NAPTR 5 2 7200 20121125230752 20121026230752 4442 isc.org. ZD14qEHR7jVXn5uJUn6XR9Lvt5Pa7YTEW94hNAn9Lm3Tlnkg11AeZiOU 3woQ1pg+esCQepKCiBlplPLcag3LHlQ19OdACrHGUzzM+rnHY50Rn/H4 XQTqUWHBF2Cs0CvfqRxLvAl5AY6P2bb/iUQ6hV8Go0OFvmMEkJOnxPPw 5i4=
isc.org.                484        IN        RRSIG        NSEC 5 2 3600 20121125230752 20121026230752 4442 isc.org. rY1hqZAryM045vv3bMY0wgJhxHJQofkXLeRLk20LaU1mVTyu7uair7jb MwDVCVhxF7gfRdgu8x7LPSvJKUl6sn731Y80CnGwszXBp6tVpgw6oOcr Pi0rsnzC6lIarXLwNBFmLZg2Aza6SSirzOPObnmK6PLQCdmaVAPrVJQs FHY=
isc.org.                484        IN        RRSIG        DNSKEY 5 2 7200 20121125230126 20121026230126 4442 isc.org. i0S2MFqvHB3wOhv2IPozE/IQABM/eDDCV2D7dJ3AuOwi1A3sbYQ29XUd BK82+mxxsET2U6hv64crpbGTNJP3OsMxNOAFA0QYphoMnt0jg3OYg+AC L2j92kx8ZdEhxKiE6pm+cFVBHLLLmXGKLDaVnffLv1GQIl5YrIyy4jiw h0A=
isc.org.                484        IN        RRSIG        DNSKEY 5 2 7200 20121125230126 20121026230126 12892 isc.org. j1kgWw+wFFw01E2z2kXq+biTG1rrnG1XoP17pIOToZHElgpy7F6kEgyj fN6e2C+gvXxOAABQ+qr76o+P+ZUHrLUEI0ewtC3v4HziMEl0Z2/NE0MH qAEdmEemezKn9O1EAOC7gZ4nU5psmuYlqxcCkUDbW0qhLd+u/8+d6L1S nlrD/vEi4R1SLl2bD5VBtaxczOz+2BEQLveUt/UusS1qhYcFjdCYbHqF JGQziTJv9ssbEDHT7COc05gG+A1Av5tNN5ag7QHWa0VE+Ux0nH7JUy0N ch1kVecPbXJVHRF97CEH5wCDEgcFKAyyhaXXh02fqBGfON8R5mIcgO/F DRdXjA==
isc.org.                484        IN        RRSIG        SPF 5 2 7200 20121125230752 20121026230752 4442 isc.org. IB/bo9HPjr6aZqPRkzf9bXyK8TpBFj3HNQloqhrguMSBfcMfmJqHxKyD ZoLKZkQk9kPeztau6hj2YnyBoTd0zIVJ5fVSqJPuNqxwm2h9HMs140r3 9HmbnkO7Fe+Lu5AD0s6+E9qayi3wOOwunBgUkkFsC8BjiiGrRKcY8GhC kak=
isc.org.                484        IN        RRSIG        A 5 2 7200 20121125230752 20121026230752 4442 isc.org. ViS+qg95DibkkZ5kbL8vCBpRUqI2/M9UwthPVCXl8ciglLftiMC9WUzq Ul3FBbri5CKD/YNXqyvjxyvmZfkQLDUmffjDB+ZGqBxSpG8j1fDwK6n1 hWbKf7QSe4LuJZyEgXFEkP16CmVyZCTITUh2TNDmRgsoxrvrOqOePWhp 8+E=
isc.org.                4084       IN        NS           ns.isc.afilias-nst.info.
isc.org.                4084       IN        NS           ams.sns-pb.isc.org.
isc.org.                4084       IN        NS           ord.sns-pb.isc.org.
isc.org.                4084       IN        NS           sfba.sns-pb.isc.org.

;; AUTHORITY SECTION:
isc.org.                4084       IN        NS           ns.isc.afilias-nst.info.
isc.org.                4084       IN        NS           ams.sns-pb.isc.org.
isc.org.                4084       IN        NS           ord.sns-pb.isc.org.
isc.org.                4084       IN        NS           sfba.sns-pb.isc.org.

;; ADDITIONAL SECTION:
mx.ams1.isc.org.        484        IN        A            199.6.1.65
mx.ams1.isc.org.        484        IN        AAAA         2001:500:60::65
mx.pao1.isc.org.        484        IN        A            149.20.64.53
mx.pao1.isc.org.        484        IN        AAAA         2001:4f8:0:2::2b
_sip._udp.isc.org.      4084       IN        SRV          0 1 5060 asterisk.isc.org.

;; Query time: 176 msec
;;SERVER: x.x.x.x#53(x.x.x.x)
;; WHEN: Tue Oct 30 01:14:32 2012
;; MSG SIZE  rcvd: 3223

That's a 64 byte query that resulted in a 3,223 byte response. In other words, an attacker is able to achieve a 50x amplification over whatever traffic they can initiate to an open DNS resolver. Note, ironically, how the effectiveness of the attack based on the size of the response is made worse by the inclusion of the huge DNSSEC keys -- a protocol designed to make the DNS system more secure.

Open DNS Resolvers: Bane of the Internet

The key term that I used a couple times so far is "open DNS resolver." The best practice, if you're running a recursive DNS resolver is to ensure that it only responds to queries from authorized clients. In other words, if you're running a recursive DNS server for your company and your company's IP space is 5.5.5.0/24 (i.e., 5.5.5.0 - 5.5.5.255) then it should only respond to queries from that range. If a query arrives from 9.9.9.9 then it should not respond.

The problem is, many people running DNS resolvers leave them open and willing to respond to any IP address that queries them. This is a known problem that is at least 10 years old. What has happened recently is a number of distinct botnets appear to have enumerated the Internet's IP space in order to discover open resolvers. Once discovered, they can be used to launch significant DNS Amplification Attacks.

Returning to our little 20Gbps DDoS attack that has been ongoing for the last three weeks, the map below illustrates the DNS resolvers we've seen launching attacks by country. You can mouse over each country to see the number of open resolvers operating in each. In total, we've seen 68,459 unique open resolvers participating in the attack.

The US dominates the list, but that is largely skewed by the number of networks present within the country. Per capita, the worst country is Taiwan where the country's HINET is the second largest source of open resolvers of any network in the world. We've also published a list of the top networks from which we're seeing abused open DNS resolvers. Below is a sample of the top-ten worst offenders:

# of Open Resolvers	AS Number	Network Name
3359	45595	PKTELECOM-AS-PK Pakistan Telecom Company Limited
2992	3462	HINET Data Communication Business Group
1431	9394	CRNET CHINA RAILWAY Internet(CRNET)
1403	21844	THEPLANET-AS - ThePlanet.com Internet Services, Inc.
1323	4134	CHINANET-BACKBONE No.31, Jin-rong Street
1120	36351	SOFTLAYER - SoftLayer Technologies Inc.
1112	4713	OCN NTT Communications Corporation
1039	26496	AS-26496-GO-DADDY-COM-LLC - GoDaddy.com, LLC
980	7018	ATT-INTERNET4 - AT&T Services, Inc.
852	32613	IWEB-AS - iWeb Technologies Inc.
--------------------------	---------------	------------------------------------------------------

Wonder why there's been an increase in big DDoS attacks? It's in large part because the network operators listed above have continued to allow open resolvers to run on their networks and the attackers have begun abusing them. While we're reluctant to publish the list of the actual IP address of the open resolvers for fear that they may be misused, if you are one of the operators of one of the networks listed above, we're happy to share data with you in order to help you get your network cleaned up. Organizations such as Team Cymru publish more extensive lists and also work with network operators to get their networks cleaned up.

If you are running an open recursor, you should close it now. Leaving it open means you will continue to aid in these attacks. If you're running BIND, you can include one or more of the following in your configuration file in order to limit attackers abusing your network:

// Disable recursion for the DNS service
//options {    recursion no;};

// Permit DNS queries for DNS messages with source addresses
// in the 192.168.1.0/24 netblock. The 'allow-query-cache'
// options configuration can also be used to limit the IP
// addresses permitted to obtain answers from the cache of
// the DNS server. Substitute with your own network range.
//options {    allow-query {192.168.1.0/24;};};

CloudFlare itself is designed to automatically learn from the traffic to our network, whether the traffic is good or bad. While this size of attack would be crippling for most networks, it has been relatively trivial for us to identify the sources of the attack, route them so they don't affect any of our customers, and study their behavior over the last three weeks. Now that we've enumerated the sources of the attack, we've begun to null route the traffic upstream to fully neuter the attack.

We will continue to work with the networks listed above in order to get their networks cleaned up. And, as new threats emerge, we'll continue to share information on them in order to ensure the Internet can remain fast and safe.

OCSP Stapling: How CloudFlare Just Made SSL 30% Faster

Matthew Prince — Mon, 29 Oct 2012 17:54:00 -0700

This week CloudFlare is announcing several things we're doing to significantly improve the performance of SSL. Too few sites are secured with SSL. One of the reasons sites don't implement SSL is that it can slow down web performance. One of the less frequently discussed, but most significant, performance hits to SSL is the OCSP/CRL check. These checks make up 30% or more of the HTTPS overhead. That's painful.

The best solution to speed up OCSP/CRL performance is something called OCSP Stapling. CloudFlare is committed to making the Internet faster and safer so we just enabled OCSP Stapling network wide in order to speed up all HTTPS connections and making the decision to secure a site with SSL a no-brainer. So what is the OCSP/CRL check? Why does it slow down page loads so significantly? And how have we eliminated this performance tax on HTTP connections with OCSP Stapling? Read on to find out.

The Revocation Overhead

To support secure web connections over HTTPS a website must have a SSL certificate. SSL certificates are issued by what is known as a Certificate Authority (CA). SSL certificates are issued for a period of time during which they will be trusted by browsers. If, however, a SSL certificate is stolen or compromised in some way before it expires, sites need a way to revoke the certificate so it will no longer be trusted.

The OCSP and CRL are the two protocols used to revoke certificates. CRL, which stands for Certificate Revocation List, is the older and cruder of the two protocols. When a CA receives a CRL request from a browser, it returns a complete list of all the certificates that CA manages that have been revoked. The browser then needs to parse the list and determine if the certificate of the visited site has been revoked.

With OCSP, the browser sends the certificate for the site in question to the CA. The CA then returns good, revoked, or unknown for the particular certificate. OCSP is generally preferable because less data needs to be sent and there's less overhead from the browser having to parse the CRL response. While every browser handles the revocation check process differently, generally modern browsers prefer OCSP to CRL checks.

Revocation Checks: 30%+ of SSL Slowness

Regardless of whether the browser performs an OCSP or CRL check, the check adds significant overhead. To give you a sense, the following connection flow is taken from this post on SSL overhead on mobile devices:

DNS (1334ms)
TCP handshake (240ms)
SSL handshake (376ms)
Follow certificate chain (1011ms)
DNS to CA (300ms)
TCP to CA (407ms)
OCSP to CA #1 (598ms)
TCP to CA #2 (317ms)
OCSP to CA #2 (444ms)
Finish SSL handshake (1270ms)

The red portions in the list above (steps 5 - 9) represent the overhead required for the revocation check requests. Add up the time for each step and you'll see that over 30% of the SSL overhead comes from checking whether the certificate has been revoked. And, unfortunately, this check is not done in parallel. In most browsers, until the revocation check is complete, the browser won't begin downloading any additional content. In other words, the OCSP check is blocking on content delivery and inherently adds a significant amount of time to the request. Painful.

Stapling OCSP for the Win

The key to speeding up OCSP is to get rid of the requests that go back to the CA. Rather than needing to request the OCSP response from the CA directly, the OCSP response can be included in the initial SSL handshake (step 3 in the example above). In this sense, the OCSP response is "stapled" to the initial SSL handshake. While it seems like this approach would be less secure, the response is signed by the CA's root certificate so the browser can verify its authenticity even if it is not delivered directly from the CA's OCSP server.

While OCSP Stapling makes a ton of sense, it unfortunately hasn't previously been widely supported by web servers. Part of the problem is that it often requires a significant technical investment by web administrators. While that investment may not make sense for many individual sites, CloudFlare sits in a unique position to enable OCSP Stapling for a large number of sites in one fell swoop. So that's what we just did.

At CloudFlare, our mission is to make the web faster and more secure. Inherent to this mission is eliminating the performance penalty of SSL connections so as many sites as possible will support secure HTTPS connections. Our SSL performance was already best of class, now it's even faster. If you're already a CloudFlare customer with SSL enabled, your HTTPS performance is now about 30% faster than it was last week. If you're not yet a CloudFlare customer but you want to make sure your SSL performance it as fast as possible, it only takes about 5 minutes to sign up.

Stay tuned this week for more announcements on how we're helping improve SSL performance for the whole web.

Sandy, Meet CloudFlare

Matthew Prince — Sun, 28 Oct 2012 20:27:00 -0700

There is a large hurricane in the Atlantic Ocean which is likely to come on-shore in the Eastern United States late tomorrow. Dubbed Sandy, the current track of the hurricane has it making landfall around Philadelphia, Pennsylvania. High winds and rain from the hurricane will likely impact two of CloudFlare's data centers: Ashburn, VA and Newark, NJ.

In both locations, the vendors that run the buildings have taken steps to ensure that service is not interrupted. Specifically:

Backup generators have been tested, fuel tanks are full, levels verified, and backup fuel vendors have been placed on standby in the event of extended power interruption.
Special arrangements have been made to make sure staff is available on site as necessary to maintain operability standards.
Hotel rooms near the sites have been secured, cots are in the facilities, MREs, and other emergency supplies, are available should the situation become extreme.

That is comforting, but what is more comforting is that the CloudFlare network is designed to have no single points of failure. If connectivity is lost in any of our data centers -- due to a hurricane, earthquake, or someone just tripping over a power cable -- traffic automatically fails over to the next closest data center.

If your website is hosted in a region that is likely to be impacted by Sandy, signing up for CloudFlare can help ensure that your site stays online. Even if the effects of the hurricane are significant for the region, CloudFlare will continue to serve the static portions of your site via our Always Online feature.

We've been through hurricanes before with Irene about a year ago. We're keeping a close eye on the hurricane and will post updates to @CloudFlareSys and our System Status page if there is any impact to our Ashburn or Newark facilities. However, you can rest assured that even if there is a regional interruption, it won't impact the availability of CloudFlare's service.

The Pumpkin Lady Carves Again

Kristin Tarr — Fri, 26 Oct 2012 19:17:00 -0700

With Halloween around the corner, I caught up with one of our customers, the husband and wife team behind Pumpkinlady.com. If you haven't been to their website, it offers a range of pumpkin carving stencils and accessories for the holiday season. The website has gained a lot of popularity over the last few years. Lisa Berberette (aka, The Pumpkin Lady) has been featured on Good Morning America and the Home Shopping Network, not to mention she's carved pumpkins for Oprah Winfrey and Martha Stewart. I sat down with Jack Berberette (aka, The Pumpkin Hubby) to talk about their business and what it's like to run a seasonal website.

CF: How long have you been running the websitepumpkinlady.com?

JB: This is our 15th season! In 1998 we noticed that there really weren't a lot downloadable carving patterns on the web, so Lisa created a bunch of patterns and I designed the site as a place for people to grab some cool freebies. We had no idea that the site would take off the way it did.

CF: What are some of the perks of running a seasonal website? What are some of the challenges?

JB: Running a seasonal site is cool in that it's easy to focus on a particular niche and we can relax a bit outside of the busy season. The downside is that we only have a three month income window. We're in the process now of working on a non-seasonal specific line of artwork that will hopefully fill in the nine month gap.

CF: Is there anything you do to prepare your website for the busy holiday season?

JB: Over the years, more and more pumpkin carving sites have popped up. Now competition is a good thing, but it does present search engine ranking challenges (this past year even more so with Google's algorithm change). So, in March I started researching Google "ranking rules" and how we fit in compared to our competitors....from there I started making measured tweaks to optimize SEO.

Aside from search engine ranking, running a high volume, seasonal Wordpress site has presented some interesting challenges. I am constantly looking for ways to reduce CPU usage, database calls and bandwidth. From custom code tweaks in the e-commerce backend, optimizing caching to CDN, there is quite a bit to prepare for each year. But I'm a nerd and like that sort of thing.

CF: What advice would you give to other small businesses running a seasonal site?

JB: Content generation for a seasonal site is easy, creating a great visitor experience is the challenge. Seasonal sites, by their very nature, are highly competitive. "Big Box" sites have huge budgets to hone in on that market. So, to set yourself apart, make sure that your visitors feel that they are the single most important thing that has ever happened to your business. It doesn't matter if they don't purchase anything, only spend a buck or place a huge order...the customer wants to know that you care about them and their experience they had on your site. If you can convey this type of conviction, you will definitely have an edge over the world of auto-responders and canned replies.

CF: Any tips or tricks you'd like to share on using CloudFlare?

JB: Yes...don't ever stop using it, and if you aren't using CloudFlare, start right now! As far as tips go, I pretty much use the default settings as I have found that these work best with our site and its plugins. I do recommend activating the Google Analytics settings.

In his own words...

Without a doubt, the single most significant performance and security enhancement our site has ever experienced has been the incredible impact CloudFlare has made. In just the past 30 days, CloudFlare has saved our site from over 21 million requests and over 200GB of bandwidth. Knowing that CloudFlare has my back with great support, resource savings and enhanced site security makes my job as a site owner so much easier. If you aren't using this incredible service, then you are missing out on something great.

\~ Jack Berberette (www.pumpkinlady.com)

WSJ: CloudFlare Named Most Innovative Internet & Networking Company, Second Year in a Row

Matthew Prince — Tue, 16 Oct 2012 01:25:00 -0700

We just got word from the Wall Street Journal that CloudFlare was named the Most Innovative Internet & Networking Company of 2012. This is the second year in a row that CloudFlare has won this award. Given the pace of development of new technologies on the Internet, that is high praise for the innovations our team has brought to market over the last year.

In announcing the award, the WSJ specifically called out three technologies we launched over the last year: Rocket Loader, Railgun, and the Automatic IPv6 Gateway.

Rocket Loader

Rocket Loader helps ensure that scripts and other resources don't slow down page load times. Too often, third party scripts like buttons, widgets, and ads can block a page from loading. These problems are exacerbated on mobile devices. Since mobile bandwidth is limited, and connections are fickle, needing to open connections to multiple third party services can seriously degrade mobile performance.

Rocket Loader's innovative approach fetches multiple third party objects through a single connection to CloudFlare's network. Instead of needing to open new requests to each service, a device only needs to establish a single connection to CloudFlare's network. Our servers, which do not have the same constraints, can then open connections to all the services needed to download the objects. The net result is pages load faster and aren't blocked if an ad or Twitter button fails to load quickly.

Rocket Loader is available to all CloudFlare customers for free. We've found it often improves page load performance an additional 30% over the other benefits delivered by CloudFlare's systems, with the biggest performance gains seen on mobile devices.

Railgun

The caching model for the Internet is based on files. CloudFlare caches static files (images, CSS, Javascript) at our edge and, in doing so, saves approximately 70% of the average websites' bandwidth and requests. While CloudFlare works well with dynamic content, since the file changes it is impossible for it to be cached.

CloudFlare spent a year studying the properties of the dynamic web. In the process, we realized that while many sites are dynamic, the amount they actually change is very small: less than 5% every 24 hours. Railgun changes the caching model of the Internet to allow only the portions of a file that have changed to be sent across the network.

Railgun was inspired by CloudFlare's continued expansion of data centers. As CloudFlare's network grew to include 23 data centers, the latency from a browser to our network shrank. However, the latency from our network back to our customers' web servers grew. Business and Enterprise customers, as well as customers on CloudFlare Optimized Hosting Partners, can use Railgun to achieve a 99.6% compression ratio when sending data to our network. That means what used to take 200 packets can now be sent in a single packet and, more practically, even highly dynamic sites can, for the first time, be as fast as if the data center were next door regardless of where a visitor is surfing from.

Automatic IPv6 Gateway

One of the most vexing challenges the Internet faces is the move from IPv4 to IPv6. The IPv4 protocol was designed to only accommodate approximately 4 billion devices simultaneously connected to the network at any given time. As we close in on this theoretical limit we are literally running out of addresses. While it's hard to imagine, that means the seemingly limitless Internet is running out of space.

The solution is a new protocol — IPv6 — which can support a vastly larger number of connected devices. Unfortunately, IPv6 networks cannot interoperate with IPv4 networks meaning the Internet has been faced with a giant chicken-or-egg problem. Web surfers are reluctant to upgrade to IPv6 because there's no IPv6 web content, and web publishers are reluctant to switch because there are not IPv6 web surfers.

CloudFlare realized we could help solve this problem by providing our Automatic IPv6 Gateway. The free service allows anyone to maintain their existing IPv4 infrastructure and be available on the IPv6 web just by signing up for CloudFlare. While that is great for existing customers, the bigger opportunity is actually with new entrants to the Internet. As it becomes more expensive to launch an IPv4 infrastructure, CloudFlare will allow new Internet publishers to put their content on IPv6 but still reach legacy IPv4 surfers.

CloudFlare's Automatic IPv6 Gateway was so successful that CloudFlare doubled the number of IPv6 websites available the day we launched the product. Today, CloudFlare provides IPv6 connectivity to more websites than any other provider.

Not Slowing Down

CloudFlare's team comes to work every day excited to invent the future of the Internet. Our goal is nothing short of rebuilding a faster, safer, smarter web. We are honored to be recognized by the Wall Street Journal for the second year in a row as one of the world's most innovative companies. And we're hard at work on the next technologies in our continuing efforts to build a better Internet.

You can read more about CloudFlare's award on the Wall Street Journal's website:

http://online.wsj.com/article/SB10000872396390444024204578046911546099812.html

CloudFlare and StopBadware partner to make the Web a better place

Kristin Tarr — Fri, 05 Oct 2012 16:45:00 -0700

CloudFlare and StopBadware go way back. Even before CloudFlare was founded, our founders had been working with the StopBadware team to help make the Web a safer, better place for everyone. When CloudFlare introduced its phishing protection this summer, StopBadware was a great sounding board as we developed our remediation and notification process.

Today, we are excited to announce we have partnered with StopBadware to formalize this cooperation. StopBadware makes the Web safer through the prevention, mitigation, and remediation of badware websites. Their work protects people and organizations from becoming victims of viruses, spyware, scareware, and other badware.

As a StopBadware partner, CloudFlare will participate in forums, events and educational programs to help further protect websites all over the world. CloudFlare is providing its Business-level service, with extended performance and security, to stopbadware.org andbadwarebusters.org. We look forward to working with StopBadware and other partners in making the Web a better place!

Visit the StopBadware partner page to learn more about the program, or listen in to CloudFlare co-founder, Michelle Zatlyn, as she discusses StopBadware with executive director Maxim Weinstein.

CloudFlare Partners with Parallels To Bring Web Performance and Security to 10 Million SMBs

Maria Karaivanova — Thu, 04 Oct 2012 23:45:00 -0700

Parallels makes it easier for service providers, like webhosts, to grow their business with service delivery software and a robust partner ecosystem. More than 9,000 service providers use Parallels to deliver thousands of applications and cloud services to 10 million small and medium businesses (SMBs) in 130 countries.

Today, we are excited to announce the CloudFlare and Parallels partnership. CloudFlare is available to all Parallels Plesk Panel 11 and 10.4 service providers through the APS catalog. It takes less than 5 minutes to install and comes in twoplans: CloudFlare Free and CloudFlare Performance Plus. We've made it super easy for all Parallels Service Providers to offer enterprise-grade performance and security to their customers. With just a few clicks, website owners can activate CloudFlare from their Parallels Panel.

Parallels Service Providers get many operational benefits; reduced server load, bandwidth savings, protection from DDoS attacks and an automatic IPv4/6 gateway. For the first time, we've also enabled every Parallels Service Provider to resell CloudFlare to their customers and generate revenue.

"The new CloudFlare Parallels Panel plugin allows for quick and easy deployment," said Gerado Altman, National Partner & Reseller Manager at Velocity Host. "We would love to keep CloudFlare as our own secret weapon but really the more users who are on the network the greater the power CloudFlare has to wield against malicious attacks and help all those who use their services."

Our goal at CloudFlare is to bring premium performance and security services to every website. Today, over 500,000 websites use CloudFlare, ranging from blogs to SMBs to ecommerce websites to governments to Fortune 500 companies. Through our partnership with Parallels, we will be able to deliver performance and security to even more websites around the world. We're thrilled to be working with the Parallels team.

If you are a Parallels Plesk Panel service provider, sign up for the program here: www.cloudflare.com/parallels

Common FAQs:

In addition to Parallels Panel, Parallels also offers Parallels Automation. We are finalizing our integration into the Parallels Automation software and expect to release it later this year.

If you host with a hosting provider that uses Parallels Plesk Panel and they haven't enabled CloudFlare yet, encourage them to check out the program.

CloudFlare is only available on Parallels Plesk Panel 10.4 and Parallels Plesk Panel 11. If you are using an earlier version of Parallels Plesk Panel, then CloudFlare is not available through this partnership.

Check out our live interview with Parallels Sr. Director/ISV and SaaS Alliance, Alex Danyluk at HostingCon 2012 as he discusses our new integration.

Happy Second Birthday CloudFlare!

Matthew Prince — Thu, 27 Sep 2012 21:55:00 -0700

Michelle, Lee and I started working on CloudFlare back in early 2009. It took about a year and a half for us to fully bake the idea, hire a team, write the code, build the start of a network, sign up beta customers to kick the tires, and then finally release CloudFlare to the public. On September 27, 2010, CloudFlare launched and so we think of today as our birthday. We're posting this two years to the minute from the moment CloudFlare went live to the public. Here's the video of our launch presentation:

Whirlwind

The last two years have been a whirlwind. Two years ago, we proudly announced on stage that we had 1,000 websites using CloudFlare during its initial year and a half of beta. Today, we have more than half a million and regularly sign up several thousand a day. Two years ago, we had seen 6 million unique IPs connect to our network. Today, we see about 600 million unique IPs connecting every month. Two years ago, we had powered 50 million page views. Today, we've powered more than half a trillion.

There is a lot of technology and infrastructure that has gone into building CloudFlare — we've literally added a new data center a month for every month since we launched, a pace we plan to continue. However, the real key to our scaling as quickly as we have has been CloudFlare's incredible team. There were eight of us that launched on stage two years ago. It was an incredible group. What I'm proud of is that, as the company has grown, we've continued to attract people who reflect that original team: great engineers who are also really good people.

I don't know of any other company in the world that handles a billion page views per month per employee — we handle two billion. The challenge and opportunity of working at CloudFlare is that when you push a line of code it affects over 200 million people in the next 24 hours. If you want to make an impact and learn how to work at real scale, there is no better place today than CloudFlare.

Everyone Loves Presents!

So what do we want for our birthday? Just to continue to grow and help build a better Internet. You can help. If you're already a CloudFlare customer, take a second to sign up another one of your websites, add a CloudFlare badge, or to tell your friends and colleagues about us. And, the best present of all, if CloudFlare seems like a place you'd thrive, then check out our careers page to apply to join our team.

Last year, we were surprised to learn that Google shared today as their birthday. We're fond of Google and see a lot of what we're doing as following in their footsteps. As we said when we launched, CloudFlare is bringing the resources previously reserved for the Internet giants to the rest of the web. Echoing Google, our mission is to build a faster, safer, better web. Two years in, we're well on our way.

CloudFlare protects Einstein's brain!

Kristin Tarr — Wed, 26 Sep 2012 00:20:00 -0700

CloudFlare protects a lot of websites - from eCommerce websites to blogs to SMBs to government sites, and everything in between. And now we can say we protect Einstein's brain through an interactive iPad app from the National Museum of Health + Medicine Chicago.

Thanks to the NMHMC, neuroscientists, researchers, educators and the general public now have access to Albert Einstein's brain via a new iPad app that will allow its users to examine the Nobel Prize-winning physicist's neuroanatomy as if they were sitting in front of a microscope.

The NMHMC Harvey App provides access to the Harvey Collection, from the National Museum of Health and Medicine, prepared by Dr. Thomas Harvey, the pathologist who performed the autopsy on Albert Einstein. The collection includes microscopic neuroanatomical slides prepared from approximately 170 areas of Einstein's brain and brainstem. Users can explore these ultra-high-resolution microscopic slide images at a variety of magnification levels, extending from low magnification all the way to cellular detail, just like with a physical microscope, all from their own network-connected iPads.

The iPad app launched today and we are thrilled that the NMHMC chose CloudFlare to help make sure their app was fast, secure and available for their new users. The iPad app has received a lot of media coverage around the world including USA Today, Venture Beat and The Sun, and with CloudFlare's help, the launch of the app has gone off without a hitch.

The NMHMC Harvey App is based on the VScope virtual microscope system from the National Museum of Health and Medicine Chicago. For more information about the app, visit http://nmhmchicago.net/harvey/

CloudFlare’s Newest App Partner: Prosperent’s ProsperLinks

Kristin Tarr — Tue, 25 Sep 2012 17:57:00 -0700

What if there was an automated system that could analyze all of your page content, determine what your users are talking about and deliver highly targeted product references directly in your content? Say hello to Prosperent's ProsperLinks.

CloudFlare is excited to announce our newest app, Prosperent's ProsperLinks. Their affiliate tool is a very simple way to earn extra revenue on your content without diminishing your readers' experience.

When enabled and configured, it instantly analyzes your page content and turning product references into affiliated links. Additionally, ProsperLinks automatically affiliates existing links in your content. You get paid when your readers make a purchase!

What is Prosperent?

Prosperent is an advertising company that specializes in retail and product advertising. With over 3,000 merchants including Zappos.com, 6pm, Overstock.com, Backcountry, REI, Nordstrom's and thousands of other retailers, Prosperent makes the best products available through their advertising products, and improves profitability for publishers.

User Experience

With ProsperLinks, you are in control of the user experience. Choose from single underline, double underline, or links with a price comparison hover box. This allows you to maximize revenue while minimizing the impact to your visitors.

Making Money

ProsperLinks is free to install and 70% of revenue will be paid to the website owner. CloudFlare will pay publishers directly, on a net-60 day basis. For example, revenue accrued in March will be paid by the end of May. Payments are made at the end of each month.

How to Launch a 65Gbps DDoS, and How to Stop One

Matthew Prince — Mon, 17 Sep 2012 22:17:00 -0700

Yesterday I posted a post mortem on an outage we had Saturday. The outage was caused when we applied an overly aggressive rate limit to traffic on our network while battling a determined DDoS attacker. In the process of writing it I mentioned that we'd seen a 65Gbps DDoS earlier on Saturday. I've received several questions since that all go something like: "65Gbps DDoS!? Who launches such an attack and how do you defend yourself against it?!" So I thought I'd give a bit more detail.

What Constitutes a Big DDoS?

A 65Gbps DDoS is a big attack, easily in the top 5% of the biggest attacks we see. The graph below shows the volume of the attack hitting our EU data centers (the green line represents inbound traffic). When an attack is 65Gbps that means every second 65 Gigabits of data is sent to our network. That's the equivalent data volume of watching 3,400 HD TV channels all at the same time. It's a ton of data. Most network connections are measured in 100Mbps, 1Gbps or 10Gbps so attacks like this would quickly saturate even a large Internet connection.

At CloudFlare, an attack needs to get over about 5Gbps to set off alarms with our ops team. Even then, our automated network defenses usually stop attacks without the need of any manual intervention. When an attack gets up in the tens of Gigabits of data per second, our ops team starts monitoring the attack: applying filters and shifting traffic to ensure the attacked customer's site stays online and none of the rest of our network is affected.

So You Want to Launch a DDoS

So how does an attacker generate 65Gbps of traffic? It is highly unlikely that the attacker has a single machine with a big enough Internet connection to generate that much traffic on its own. One way to generate that much traffic is through a botnet. A botnet is a collection of PCs that have been compromised with a virus and can be controlled by what is known as a botnet herder.

Botnet herders will often rent out access to their botnets, often billing in 15 minute increments (just like lawyers). Rental prices depend on the size of the botnets. Traditionally, email spammers purchased time on botnets in order to send their messages to appear to come from a large number of sources. As email spam has become less profitable with the rise of better spam filters, botnet herders have increasingly turned to renting out their networks of compromised machines to attackers wanting to launch a DDoS attack.

To launch a 65Gbps attack, you'd need a botnet with at least 65,000 compromised machines each capable of sending 1Mbps of upstream data. Given that many of these compromised computers are in the developing world where connections are slower, and many of the machines that make up part of a botnet may not be online at any given time, the actual size of the botnet necessary to launch that attack would likely need to be at least 10x that size. While by no means unheard of, that's a large botnet and using all its resources to launch a DDoS risks ISPs detecting many of the compromised machines and taking them offline.

Amplifying the Attacks

Since renting a large botnet can be expensive and unwieldy, attackers typically look for additional ways to amplify the size of their attacks. The attack on Saturday used one such amplification technique called DNS reflection. To understand how these work, you need to understand a bit about how DNS works.

When you first sign up for an Internet connection, your ISP will provide you with a recursive DNS server, also known as a DNS resolver. When you click on a link, your computer sends a lookup to your ISP's DNS resolver. The lookup is asking a question, like: what is the IP address of the server for cloudflare.com? If the DNS resolver you query knows the answer, because someone has already asked it recently and the answer is cached, it responds. If it doesn't, it passes the request on to the authoritative DNS for the domain.

Typically, an ISP's DNS resolvers are setup to only answer requests from the ISP's clients. Unfortunately, there are a large number of misconfigured DNS resolvers that will accept queries from anyone on the Internet. These are known as "open resolvers" and they are a sort of latent landmine on the Internet just waiting to explode when misused.

DNS queries are usually sent via the UDP protocol. UDP is a fire-and-forget protocol, meaning that there is no handshake to establish that where a packet says it is coming from actually is where it is coming from. This means, if you're an attacker, you can forge the header of a UDP packet to say it is coming from a particular IP you want to attack and send that forged packet to an open DNS resolver. The DNS resolver will reply back with a response to the forged IP address with an answer to whatever question was asked.

To amplify an attack, the attacker asks a question that will result in a very large response. For example, the attacker may request all the DNS records for a particular zone. Or they may request the DNSSEC records which, often, are extremely large. Since resolvers typically have relatively high bandwidth connections to the Internet, they have no problem pumping out tons of bytes. In other words, the attacker can send a relatively small UDP request and use open resolvers to fire back at an intended target with a crippling amount of traffic.

Mitigating DNS Reflection Attacks

One of the great ironies when we deal with these attacks is we'll often get an email from the owner of the network where an open resolver is running asking us to shut down the attack our network is launching against them. They're seeing a large number of UDP packets with one of our IPs as the source coming in to their network and assume we're the ones launching it. In fact, it is actually their network which is being used to launch an attack against us. What's great is that we can safely respond and ask them to block all DNS requests originating from our network since our IPs should never originate a DNS request to a resolver. Not only does that solve their problem, but it means there's a smaller pool of open resolvers that can be used to target sites on CloudFlare's network.

There have been a number of efforts to clean up open resolvers that are currently active. Unfortunately, it is slow going and the default installation of many DNS clients still has them open by default. While we actively reach out to the worst offenders to protect our network, to protect the Internet generally there will need to be a concerted effort to clean up open DNS resolvers.

In terms of stopping these attacks, CloudFlare uses a number of techniques. It starts with our network architecture. We use Anycast which means the response from a resolver, while targeting one particular IP address, will hit whatever data center is closest. This inherently dilutes the impact of an attack, distributing its effects across all 23 of our data centers. Given the hundreds of gigs of capacity we have across our network, even a big attack rarely saturates a connection.

At each of our facilities we take additional steps to protect ourselves. We know, for example, that we haven't sent any DNS inquiries out from our network. We can therefore safely filter the responses from DNS resolvers: dropping the response packets from the open resolvers at our routers or, in some cases, even upstream at one of our bandwidth providers. The result is that these types of attacks are relatively easily mitigated.

What was fun to watch was that while the customer under attack was being targeted by 65Gbps of traffic, not a single packet from that attack made it to their network or affected their operations. In fact, CloudFlare stopped the entire attack without the customer even knowing there was a problem. From the network graph you can see after about 30 minutes the attacker gave up. We think that's pretty cool and, as we continue to expand our network, we'll get even more resilient to attacks like this one.

Post Mortem: What Yesterday's Network Outage Looked Like

Matthew Prince — Sun, 16 Sep 2012 19:40:00 -0700

Yesterday, around 16:36 GMT, we had an interruption to our network services. The interruption was caused by a combination of factors. First, we had an upstream bandwidth provider with some network issues that primarily affected our European data centers. Second, we misapplied a network rate limit in an attempt to mitigate a large DDoS our ops team had been fighting throughout the night.

CloudFlare is designed to make sites faster, safer and more reliable so any time an incident on our network causes any of our customers' sites to be unreachable it is unacceptable. I wanted to take some time to give you more of a sense of exactly what happened and what our ops and engineering teams have been working on since we got things restored in order to protect our network from incidents like this in the future.

Two Visible Events

The graph at the top of this post is the aggregate traffic across CloudFlare's eight European data centers. The green section represents traffic that is inbound to our network, the blue line represents traffic that is outbound from our network. Inbound traffic includes both requests that we receive from visitors to our customers' sites as well as any content that we pull from our customers' origin servers. Since we cache content on our network, the blue line should always be significantly above the green one.

Two things you notice from the graph: the big green spike around 13:30 GMT and the fall off in the blue line around 16:30 GMT. While the two were almost 3 hours apart, they are in fact related. Here's what happened.

Limited Network and a Nasty Attack

One of our upstream network providers began having issues in Europe so we routed traffic around their network, which concentrated more traffic than usual in some of our facilities in the region. These incidents happen all the time and our network is designed to make them invisible to our customers. Around 13:00 GMT, a very large DDoS attack was launched against one of our customer's websites. The initial attack was initially a basic layer 4 attack — sending a large amount of garbage traffic from a large number of infected machines to the site. The attack peaked at over 65 Gbps of traffic, much of that concentrated in Europe.

This attack is represented by the big green spike in the graph above. It was a lot of traffic, but nothing our network can't handle. We're pretty good at stopping simple attacks like this and, by 13:30 GMT, that attacker had largely stopped with that simple attack vector. During that time, the attack didn't affect any other customers on our network. Nothing so far is atypical for a normal day at CloudFlare.

Mitigation and a Mistake

The attacker switched to trying other vectors over the next several hours. While we have automated systems to deal with many of these attacks, the size of the attack was sufficient that several members of our ops team were monitoring the situation and manually adjusting routing and rules in order to ensure the customer under attack stayed online and none of the rest of the network was impacted.

Around 16:30 GMT the attacker switched vectors again. Our team implemented a new rate limit. The rate limit was supposed to apply only to the affected customer, but was misapplied to a wider number of customers. Because traffic was already concentrated in Europe more so than usual, the misapplied network rate limit impacted a large number of customers in the region. There was some spillover to traffic to our facilities in North America and Asia Pacific, however the brunt of the outage was felt in Europe.

As you can see from the graph, both inbound and outbound traffic fell off almost entirely in the region. We realized our mistake and reverted the rate limit. In some cases, the rate limit also affected BGP announcements that setup routes to our network. The spikes on the graph you see over the next hour are from the network routing rebalancing in the region.

Sanity Checks

We have been working to automate more and more of our attack mitigation. For most manual changes that could affect our network we have sanity checks in place to ensure mistake don't make it into production. The events of today have exposed one more place we need to put such checks in place. Our team worked yesterday to build additional sanity checks to protect against something similar to this happening again in the future.

CloudFlare has grown very quickly because we provide a service many websites need in a way that is affordable and easy for anyone to implement. Core to what we do is ensuring the uptime and availability of our network. We let many of our customers down yesterday. We will learn from the outage and continue to work toward implementing systems that ensure our network is among the most reliable on the Internet.

What It's Like to Launch at TechCrunch Disrupt

Matthew Prince — Mon, 10 Sep 2012 17:35:00 -0700

CloudFlare launched almost exactly 2 years ago at the first TechCrunch Disrupt SF. It was an incredible experience for us and we owe a significant amount of our success to the stage Disrupt provided us. Since then, we've rolled out 23 data centers (one per month since launch), added more than half a million customers' websites, and powered nearly half a trillion page views through the CloudFlare network. It's been quite a two years.

Living Vicariously

One of the most rewarding things for us has been getting to help other companies as they launched at the four Disrupt events since: New York 2011, San Francisco 2011, Beijing 2011, and New York 2012. CloudFlare's bread and butter is keeping sites running fast and stable even during huge bursts in traffic. And, if launching at Disrupt does one thing, it's deliver a huge burst of traffic. By our count, we've helped about 25% of the Battlefield companies over the last two years ensure their sites stay online even under the crushing load a Disrupt launch brings.

As Disrupt San Francisco 2012 gets started, I thought it would be cool to reach out to some of the standout companies that launched a year ago in order to give you a behind the scenes peek at what it's like to launch there. Tony Gauda, CEO of Bitcasa (Battlefield Finalist), Rebecca Woodcock, CEO of CakeHealth (Battlefield Finalist), and Jevon MacDonald CEO of GoInstant (acquired by Salesforce) were all standouts from last year's competition. Their sites were also all on CloudFlare for the launch, so we have the actual log data on what their servers saw. They agreed to let me tell a bit about their experience and share details from their traffic logs in order to prepare companies in the Battlefield for what to expect.

What to Expect if You're Launching

To get a sense of what Battlefield companies can expect, we aggregated the log data of 20 companies that were on CloudFlare's network when they launched. While the exact numbers vary, here's what happens to their websites:

Over the three days at TC Disrupt, expect your site to get an average of a 3x to 10x surge in traffic.
For the average company that doesn't make the finals, expect between 10,000 - 20,000 page views per day during TC Disrupt, about 20% of that traffic concentrated during the hour that you're on stage.
If you have a consumer appeal you can expect to get more traffic than if you are business or enterprise focused.
If you effectively leverage social media, you can use the Disrupt audience to further amplify the traffic to your site.
Traffic builds over all three days of the competition, even if you don't make the finals. The largest spike for most companies occurs when TechCrunch publishes the article about your company (usually shortly after you're on stage).
For companies that do make the finals, traffic can reach hundreds of requests per second during the time you are on stage and when the winners are announced.

Nerves and Details

You only get to launch once. Doing so in front of a live, tech savvy audience of 3,000, not to mention untold numbers of people watching via the live stream, is nerve wracking. There are horror stories every competitor hears, like when RedBeacon, during Disrupt's predecessor TC50 event, had their site completely crash during their presentation. It's comforting to remember that it didn't hurt them much and they went on to win their year of the competition. I remember getting dinner with my co-founders Lee Holloway and Michelle Zatlyn the night before our launch at a little Italian restaurant in SOMA called La Bricola. CloudFlare still wasn't fully working, and we all planned to work late into the night, but for a few minutes the three of us took a break to have dinner and some wine. Our toast that evening: "Please just don't let the servers melt."

Just about every company as they get ready to launch at Disrupt has the same concern. "Leading up to TechCrunch Disrupt, we were super nervous. We didn't know what would happen with our site when we stepped on stage," recounted Jevon MacDonald from GoInstant. Below is a screenshot from the CloudFlare control panel showing exactly what did happen to GoInstant's site. GoInstant launched on stage September 13 and traffic to the site continued to climb, peaking at over 20,000 page views on the 14th, as the company demoed the product and Disrupt attendees checked it out. The dashboard shows traffic from big press hits in Forbes, PC Magazine, and ZDNetlater in the month, but they were dwarfed by the traffic that Disrupt sent the site."You're always nervous that something will go wrong or something will crash," Rebecca Woodcock from CakeHealth recounted, "but you need to focus on your presentation and telling the story of the product you've built." CakeHealth launched on Monday and was one of the seven companies to make it to the Disrupt Battlefield Finals. Traffic to the site continued to grow over the three days. In total, CakeHealth's site received more than 2 million hits in less than 72 hours. When Rebecca was giving her final pitch the site was seeing several hundred requests per second.Bitcasa was another of the seven Disrupt Battlefield Finalists and saw an even bigger spike in traffic. "We had a couple EC2 instances for our website," explained Tony Gauda from Bitcasa. "We'd spent our time focusing on ensuring the backend and the product were built out, we hadn't worried about building out our web infrastructure." Bitcasa had a consumer appeal and saw an even bigger spike in traffic. Over the three days of the conference, 85,318 people sign up for their beta. "Disrupt sent the initial traffic of influencers to our site and we then encouraged them to tell their friends about Bitcasa via social media. The result was beyond what we could have ever imagined."

Good Luck to Everyone!

Everyone I've talked to who has launched at Disrupt remembers the experience extremely fondly. Whether you win or lose, once you've been on that stage you will forever be a part of the TechCrunch family, and that's something that pays dividends for your company for years to come. At CloudFlare, we've been proud to help a number of Disrupt companies with their launch. This year is no different. We've gotten a sneak peak at a few that will be launching over the next few days at Disrupt SF 2012 and, while we're sworn to strict secrecy, I can say two things: 1) It's going to be a great competition; and 2) I've already made my prediction hinting at who I think the winner will be.

To everyone launching over the next two days, best of luck from the CloudFlare team. The best advice I can offer is this: this is a once in a lifetime experience. Enjoy it!

SmartErrors: Decrease Your Site's Bounce Rate

joshuamotta — Wed, 05 Sep 2012 18:08:00 -0700

You may add SmartErrors to your site here.

Visitors dread them, publishers fear them and we've all seen them. Page Not Found errors (also known as 404s) are a common occurrence on the web where content is constantly created and discarded. However, few things increase a site's bounce rate faster. Even cutesy errors — which may garner a frustrated smile—don't actually help a visitor find what they're looking for.

Imagine our horror to discover that over 1% of all page views that pass through CloudFlare's global network lead to a 404 error! We knew that something had to be done.

Your Enemy: the 404 Error

A web site's server will typically generate a "404 - Not Found" page when a visitor to a site tries to access a page that doesn't exist for one reason or another. These errors typically occur when someone mistypes a URL on your site, when there's a broken link from another page or site, when a page that previously existed is moved or removed, or if there is an error when a search engine indexes your site. What's worse is that they can be impossible to control, as many old links come from external sites.

The 404 error: not a pretty enemy

404s can be problematic for site owners as they:

Are a lost opportunity to engage with a visitor
Increase bounce rates (when users click back and navigates away)
May lead to SEO (search engine optimization) penalization

Your Mission, Should You Choose to Accept It

Introducing SmartErrors: a beta feature that can be enabled with a single click, and that dynamically transforms dead-end error pages into an intelligent site search. SmartErrors uses keywords from the referring link that produced the 404 to find relevant content and discourage your site's visitors from using the back button and leaving your site to look elsewhere.

*SmartError's Substantially Increases Visitor Engagement *

In our testing across hundreds of CloudFlare sites we've measured average visitor engagement rates of over 12%, with some sites as high as 75-80%. What this means is that, on average, 12% of SmartErrors resulted in deeper engagement with a site. It really does make a difference!

SmartErrors can also help your site's SEO (search engine optimization). Dead-end pages can rob your site of so called "link juice." We've marked all links that point off your site as rel="nofollow" and designed the page so that both search crawlers and actual visitors stay on your site.

Who Gets SmartErrors?

SmartErrors can be enabled for any active CloudFlare site from the app store. To learn more about how SmartErrors works, check out the detailed app description here.

With SmartErrors we're excited to tackle one of the web's most frequently seen errors as we continue our mission of building a faster, safer and smarter Internet.

Alex Danyluk from Parallels on integrating CloudFlare - “What you guys are offering is awesome”

Kristin Tarr — Tue, 28 Aug 2012 23:41:00 -0700

Alex has 25 years experience in B2B hosting, web and video conferencing, mobile, voice, VoIP, call center, Internet, and data network products and services. Currently, Alex is the Sr. Director ISV & SaaS Alliances at Parallels.

CloudFlare recently integrated into the Parallels APS catalog and we couldn't be more excited. We are currently in the beta phase of our partnership, and we are looking forward to our official launch later this fall.

"Application Packaging Standard is a really quick way to connect CloudFlare to thousands of service providers around the world," said Alex. "We're really excited that CloudFlare is now integrated. What you guys are offering for service providers is awesome."

Parallels was founded over 10 years ago and has more than 9,000 partners around the world. Today, they are present in 130 countries around the world.

"We help service providers use our software to deliver cloud services," said Alex. "We are making it easy for them to deliver a lot of solutions to the market."

At HostingCon 2012,CloudFlare co-founderMichelle Zatlyn sat down with 28 leading experts in the hosting industry. Their conversations were captured live and offer insight into the latest trends and news in hosting.

Turning "I'm Under Attack" into "I'm Doing Some Good"

John Graham-Cumming — Tue, 28 Aug 2012 19:44:00 -0700

CloudFlare's I'm Under Attack mode allows our customers to, at the click of a button, tell us that they are experiencing an attack and enable automatic protection. It works by slowing down visits to the web site that's under attack and performing extra work to identify malicious visitors. When enabled, visitors to the site suffering an attack see a web page like this:

These checks take about 5 seconds to perform, and during that time the visitor's (or attacker's) web browser could be performing other work. Part of the verification takes the form of JavaScript that CloudFlare delivers to the browser. Currently, that JavaScript only performs the verification checks, but it could do more. After the checks the visitor is forwarded on to the web site.

In the past, many distributed computing efforts have harnessed the power of machines across the Internet to do collaborative work. The SETI@Home project looks for extraterrestrial life, Folding@Home looks at protein folding to help research into drugs and diseases, and GIMPS is looking for particular prime numbers. Wikipedia has a long list of such projects.

We think that I'm Under Attack mode version 2.0 could be an "I'm Doing Some Good" mode by including a distributed computation in the JavaScript that's delivered as part of dealing with attacks. The project would need to be able to broken down into chunks that run 5 seconds at a time, and be written in JavaScript. It could be run across all web sites that are under attack and in the browsers of legitimate and attacking users potentially using the resources of evil doers for a good purpose.

The end users wouldn't see any difference from the way I'm Under Attack Mode works today, but a little bit of compute power that's not being used while checks for malicious behavior are made could be put to good use. Put together, many thousands of machines could be working on a distributed computing project without any effort on the part of end users. And without any extra impact on web site owners.

The hard question to answer is... which project?

Rather than come up with our own ideas we'd like to throw this open to the community for suggestions. The best (and most implementable) solution will be picked by CloudFlare and implemented to start turning a bad situation into a good one.

Make suggestions in the comments below.

Dax Moreno from PEER 1 on acquisitions, expansions and announcements

Kristin Tarr — Tue, 28 Aug 2012 01:34:00 -0700

Dax has more than 10 years of experience in the technology industry. Currently the General Manager of ServerBeach, which is part of PEER 1 Inc., Dan has spent time in sales, support, business development, and leadership roles at various technology companies.

PEER 1 Hosting is one of the world's leading server and cloud hosting providers. We caught up with Dax at HostingCon 2012 to get the latest on recent acquisitions and announcements from PEER 1.

"We just announced the acquisition of a new company into the PEER 1 family, it's called NetBenefit," said Dax.

NetBenefit is based in Nice, France and doubled PEER 1's UK presence. The international acquisition is instrumental in reaching a new market for PEER 1 and is their first foray into continental Europe.

The acquisition is just one of many new announcements PEER 1 will be making this year.

"We're going to have some big announcements coming up in November at CloudExpo," said Dax. " We're going to be talking a little bit about ServerBeach and ZuniCore, our cloud offering, and some other PEER 1 changes."

Listen in to hear more about the NetBenefit acquisition and other announcements from PEER 1.

At HostingCon 2012, CloudFlare co-founder Michelle Zatlyn sat down with 28 leading experts in the hosting industry. Their conversations were captured live and offer insight into the latest trends and news in hosting.

Jared Ewy from name.com talks community management, culture and doing what you love

Kristin Tarr — Mon, 27 Aug 2012 21:36:00 -0700

Jared is the online "face" of name.com. He tweets (a lot) and keeps the branded conversation going on Facebook and all other social media platforms.

We caught up with Jared to learn about name.com and find out exactly what a community manager does.

"We are GoDaddy, but cooler," said Jared. "Domains, websites and hosting, we pride ourselves on ridiculous customer support."

When it comes to community management Jared says name.com is always trying new things, listening to people, giving things a chance and just seeing what works. He says it's about creating that conversation.

"It's ever evolving, but my basic thing is create that conversation," said Jared on his Twitter use. "10 years ago you didn't have the opportunity like you do now, take advantage of that."

As far as creating a culture, Jared believes you can't micromanage and you need to give everyone an opportunity to go beyond their scope of work. He also believes it's important to like your job and what you're working on.

"Do what you love," said Jared.

At HostingCon 2012,CloudFlare co-founderMichelle Zatlyn sat down with 28 leading experts in the hosting industry. Their conversations were captured live and offer insight into the latest trends and news in hosting.

David Gardner on new acquisitions and announcements

Kristin Tarr — Fri, 24 Aug 2012 21:56:00 -0700

Endurance International Group is a leading provider of hosting, domains and online solutions for small and medium-sized businesses. EIG serves more than two million customers through 40 distinct brands.

David Gardner, Sr. Project Manager at EIG, sat down with us to talk about EIG's recent acquisitions and newest announcements.

"Endurance acquired HostGator and that's something we're really excited about," said David. "They're a huge brand, they've got a lot of customers that love the service and we are really excited to have that under our business."

David touched on a few other exciting announcements including Bluehost's recent partnership with CloudFlare, their University Program that provides hosting services targeted for educators and students, and the acquisition of HostGator allowing them to offer dedicated servers and VPS to their Bluehost customers.

Tune in to hear other exciting developments with EIG.

At HostingCon 2012, CloudFlare co-founder Michelle Zatlyn sat down with 28 leading experts in the hosting industry. Their conversations were captured live and offer insight into the latest trends and news in hosting.

Joe Brinkman discusses the new CMS: Cloud, Mobile, Social

Kristin Tarr — Thu, 23 Aug 2012 23:17:00 -0700

Joeis a co-founder and technical fellow at DotNetNuke. He is an avid technologist with a broad spectrum of experience in a variety of development languages and technologies.

DotNetNuke is the leading Web Content Management Platform (or CMS) for Microsoft, powering over 700,000 production web sites worldwide. The flexible DNN open source CMS platform also functions as a web application development framework.

DNN just recently launched a new social platform. We caught up with Joe to hear the latest on DNN and their social offering.

"We just launched DotNetNuke 6.2, which is what we're calling the social CMS for business," said Joe. "We include a lot of the same types of functionality that you might find in a Jive or a Yammer or a Chatter, integrated directly into the core CMS platform."

"It's more than just social," said Joe. "It's social CMS."

DNN has seen a very strong uptake in the new social offering from their customers.

They have also seen a strong growth around cloud, mobile and social, the "new CMS." DNN has invested a lot of time and energy into these three verticals and are looking forward to growing the company in those areas.

"Those three together position us pretty uniquely in the CMS space," said Joe.

Listen in to hear more on DNN and the new CMS.

At HostingCon 2012, CloudFlare co-founder Michelle Zatlyn sat down with 28 leading experts in the hosting industry. Their conversations were captured live and offer insight into the latest trends and news in hosting.

Maxim Weinstein - Infected computers can compromise a website

Kristin Tarr — Thu, 23 Aug 2012 01:21:00 -0700

Maxim has been leading StopBadware since 2007. He serves on several malware-related working groups, speaks regularly at conferences, and serves as a subject matter expert for journalists. In 2009, he was recognized by SC Magazine as one of the year's "information security luminaries."

"StopBadware is a non-profit anti-malware organization," said Maxim. "We focus on protecting people from malicious websites and we work with both industry partners and with consumers and small businesses whose sites have been hacked."

Maxim says there are three things people can do to be more prepared against computer and website attacks.

"There's a lot of things people can do to be more prepared. One is basic web maintenance, keeping your software up to date. Whether it's WordPress or plugins or various software you may be using to host your site or manage different parts of your site," said Maxim.

Keeping passwords secure is also a must. It's important to use different passwords for websites and all other login sites.

The third thing is to keep your computer clean. Maxim explains how a bad computer contributes to getting hacked.

"Most of the sites that get hacked are everyday consumer and small business sites," said Maxim.

Tune in to hear more on how you can protect your computer and websites from getting hacked.

Seoul, Korea: CloudFlare's 23rd Data Center

Matthew Prince — Wed, 22 Aug 2012 23:13:00 -0700

We just turned on our latest data center in Seoul, Korea. It's our 23rd data center and the last of the 9 new cities as part of this latest data center expansion. Seoul expands our presence in Asia to five data centers for the Asia-Pacific region: Seoul, Tokyo, Hong Kong, Singapore and Sydney.

Seoul has invested heavily in making the Internet fast for web surfers in the region. Seoul was named one of the most connected cities in the world, with one of the highest penetrations of broadband in the world and 100Mbps commercial Internet connections available for as little as US$30/month.

Fast Strikes Back

Fast connections are great within the country, they often bottleneck when they have to travel across the over-subscribed trans-oceanic cables. Given that Korean traffic needs to pass over trans-oceanic cables to reach the rest of the Internet (unless there's a transit provider through North Korea we're not aware of) those fast connections can come to a crawl if they have to leave the country.

While fast Internet connections have a lot of positives, one of the challenges of fast, fat pipes is that they can be used to launch significant DDoS attacks. In Seoul, there has been a marked increase in large attacks launched from the country's super-speed infrastructure.

Given all of this, CloudFlare is a perfect fit for Seoul. Since we cache content for sites on CloudFlare's network locally in Seoul those fast connections can remain fast end-to-end. Moreover, since we're good at mitigating DDoS attacks, Korean businesses faced with attacks from increasingly powerful local botnets can sign up for CloudFlare, stop the attacks, and still maintain the performance that the country has come to expect.

Global Domination

This is the last of the nine new cities we're launching as part of this latest expansion effort, but our ops and networking teams are already hard at work planning our next expansion wave. We know there are regions of the world that are in need of some CloudFlare love: South America, India, Africa, and China. All of them have their own unique challenges, but we're hard at work finding ways to overcome them.

CloudFlare's goal is to build a better web for everyone. With each new data center we get closer to building a network that will allow us to achieve that goal. Stay tuned because there's a lot more to come.

Shridhar Luthria gives inside look at one of the world’s largest web service providers

Kristin Tarr — Wed, 22 Aug 2012 19:27:00 -0700

Shridhar is ResellerClub's business head and leads all business activities and initiatives. He has been steering the team since inception and has helped the company grow from a start-up to the world's most popular platform for domain names.

Shridhar most likely wins the longest distance traveled award at HostingCon 2012, traveling 20 hours from India to Boston. We sat down with him at the conference to hear the latest on ResellerClub.

"ResellerClub is one of the largest web service providers in the world today," said Shridhar. "From a domain registration standpoint, we're one of the largest wholesale focused registrars. We are always in the top five fastest growing registrars."

Growing quickly, ResellerClub is beginning to focus on hosting products, including shared hosting, reseller hosting and launching VPS and dedicated servers.

Part of their growth is also international, expanding to different parts of the world.

"We've seen tremendous amounts of growth in India, it's a buzzword," said Shridhar.

From a percentage standpoint, the U.S. is still one of the strongest markets out there in terms of absolutes, but ResellerClub is seeing traction in China, Turkey, Brazil and Russia as well.

At HostingCon 2012, CloudFlare co-founder Michelle Zatlyn sat down with 28 leading experts in the hosting industry. Their conversations were captured live and offer insight into the latest trends and news in hosting.

Gabriel Fontaine on keeping customers happy and what’s new at Verio

Kristin Tarr — Wed, 22 Aug 2012 01:20:00 -0700

Verio is the recognized industry leader in delivering online business solutions to SMBs worldwide. Distributed through its network of OEM and viaVerio channel partners, Verio's solutions provide web hosting, application hosting and SaaS applications that enable SMBs to drive online success.

We talked with Gabriel, Sr. Product Manager at Verio, to find out what Verio is all about and what new initiatives they are announcing.

"Verio is a wholly owned subsidiary of NTT Communications. We've been in the business for over sixteen years, offering shared web hosting, VPS, and SaaS products," said Gabriel.

Verio serves mostly small businesses, but they do have a reseller channel. Their biggest channel is telcos and OEMs worldwide. They have a large presence in Latin America, Europe and Japan.

Within their customer base, they've seen a few changes that they are striving to address.

"Primarily customers want easy to use, turn key solutions, especially small businesses who don't want to have to spend too much time figuring out how to use or configure product," said Gabriel. "We try to make sure we find and deliver solutions that are turn key and easy."

Tune in to hear more on Verio's customer solutions and recently launched Cloud offering, Cloud End.

Warsaw, Poland: CloudFlare's 22nd Data Center

Matthew Prince — Tue, 21 Aug 2012 15:40:00 -0700

Hot on the heels of launching our new data center in Stockholm, Sweden we just turned up our newest facility in Warsaw, Poland. That makes 22 data centers worldwide, and 8 covering Europe. Warsaw will help improve performance in Northeastern Europe and Russia. It will also decrease load on the Frankfurt data center, which has become one of our busiest.

CloudFlare chooses the location of our data centers based on where we see the greatest demand for our services. We also try and provision facilities so we can fail one out if there is a networking or other service issue and continue to provide strong service to the region.

Europe

While, for this round of expansion, this is the end of new cities in Europe, we have a few more European plans for the short and long term. In the short term, we're going to more than double the size of our London facility. London was previously constrained because the amount of power we could reasonable provision restricted the amount of equipment we could locate there. We've secured space in a new facility without the restrictive power caps and will be expanding London to match the our largest facilities elsewhere in Europe.

Longer term, our network team has begun plans for our next expansion. In Europe, the locations that appear likely to make the short list are known internally as the M's: Moscow, Milan, Madrid. We're also considering locations in Southeastern Europe including potentially Sofia and, if we can resolve some regulatory concerns, Istanbul.

Aside from Europe, we have one more data center — our 23rd and the last of this expansion round — to launch in Asia. More on our plans for the rest of the globe when when we post about that launch next week.

Stockholm, Sweden: CloudFlare's 21st Data Center

Matthew Prince — Tue, 21 Aug 2012 02:48:00 -0700

CloudFlare's data center expansion continues. If you're keeping score at home, before today we had 20 data centers worldwide. In order of when they went online, they are: Chicago, Ashburn (Virginia), San Jose (California), Amsterdam, Tokyo, Los Angeles, Newark (New Jersey), Hong Kong, Paris, Dallas, Frankfurt, Singapore, London, Miami, Sydney, Atlanta, Seattle, Toronto, Vienna, and Prague. Today we add our 21st in Stockholm, Sweden.

I've never been to Stockholm and, while I've heard the country is beautiful, my impressions of Sweden have largely been colored by the country's famous fish and chef — which are delicious and amusing, respectively. From a networking perspective, Stockholm serves as the hub of the Nordic countries in Europe and we expect it will decrease ping times throughout much of northern Europe.

Speaking of ping times, Sweden is also home to Pingdom, one of CloudFlare's most popular app partners. The Swedes care about web performance, so we're excited to be making the Internet faster for the region. And I'm hopeful now that we have a data center there, I'll have a reason to visit Stockholm and no longer think about the Swedish Chef when I think Sweden.

Finally, I'm personally particularly excited about Stockholm because Terry let me press the key to commit the final updates to the router and send traffic to the new data center. Woot!

Elya McCleave from SoftCom talks customer experience and support

Kristin Tarr — Tue, 21 Aug 2012 00:45:00 -0700

Elya has over 10 years of extensive experience in the web hosting industry. As the VP of Customer Care she is responsible for all aspects of SoftCom's Customer Support including management of all the internal and external teams and vendors.

Founded in 1997, SoftCom Inc. is an industry leading provider of Cloud Hosting and Business Communication services to more than 2.5 Million users with support of 26 languages in 140 countries worldwide.

Elya is incredibly focused on initiatives to improve their customer's experiences.

"One of our initiatives I would definitely like to point out is our customer onboarding process," said Elya. "It's a three-stage process, we onboard the customers as soon as the order is submitted. Within 10 minutes a call is being placed to welcome them aboard."

Two more calls are placed within the next week, just to be sure customers are having a painless, easy onboarding experience.

At HostingCon, Elya was excited to meet with companies to partner with.

"We are looking for partnerships to enhance the customer experience," said Elya.

When it comes to dealing with the customers directly, Elya welcomes to opportunity and even gets to practice her Russian.

"I think the executive team has to be involved and connect with the customer base," said Elya.

At HostingCon 2012, CloudFlare co-founder Michelle Zatlyn sat down with 28 leading experts in the hosting industry. Their conversations were captured live and offer insight into the latest trends and news in hosting.

Updating Policies

Matthew Prince — Mon, 20 Aug 2012 22:39:00 -0700

Back in late 2009, CloudFlare's service began to take shape and our website first went online. While in the early days I had contributed to CloudFlare's early code, we quickly hired engineers to join Lee's team who were far smarter than I. That left me to turn my attention to another area of the site more appropriate for a recovering lawyer: our Terms of Service and Privacy Policy.

Generally, these documents have held up pretty well since December 5, 2009 when we first published them. However, today we're making some updates to address some issues that have come up over the last two years. I wanted to take the time to walk through the changes here so everyone is clear why we made the updates we have.

Apps

Many of the changes to the Terms of Service and Privacy Policy are the result of CloudFlare's Apps Marketplace. From early in our history, we realized we had an opportunity to help webmasters install services to enhance their sites. Oliver Roup, a friend of mine from business school, approached up about allowing CloudFlare's users to automatically incorporate the service of a company he'd started: Viglink. Viglink's service automatically adds an affiliate code to appropriate links on your site so you can make money when people click on a link and then go on to purchase something.

It seemed like a no-brainer that we offer Viglink as an option to our users. We always thought it would be a service that people could turn on or off, but I wanted to make sure our Terms of Service included the possibility that if someone had the service on then affiliate codes could be added. I included the following sentence in our terms: "[CloudFlare may] Add tracking codes or affiliate codes to links that do not previously have tracking or affiliate codes." That has, over time, caused endless confusion, customer service inquiries, and even conspiracy theories.

We're building a platform that, through Apps, can allow you to update your site in a wide number of ways. While we want to acknowledge that, we also want to make something clear: it is always your choice as to what apps are enabled. As a result, we updated this key section to now read:

You retain full copyrights in any materials served through CloudFlare. Depending on the features you select or Apps you enable, CloudFlare may modify the content of your site. For example, CloudFlare may detect any email addresses and replace them with a script in order to keep it from being harvested, or CloudFlare may insert code to improve page load performance or enable a Third Party App. Depending on the features you enable, you acknowledge CloudFlare may:

Intercept requests determined to be threats and present them with a challenge page.
Add cookies to your domain to track visitors, such as those who have successfully passed the CAPTCHA on a challenge page.
Add script to your pages to, for example, add services, Apps, or perform additional performance tracking.
Other changes to increase performance or security of your website.

CloudFlare will make it clear whenever a feature will modify your content and, whenever possible, provide you a mechanism to allow you to disable the feature.

We've made updates elsewhere to also reflect that we allow you to install third party apps. For example, our Privacy Policy now acknowledges that you should check the Terms of Service and Privacy Policies of these app providers since they may be different from CloudFlare's. The idea of the Apps Marketplace is something that really came into focus after our initial launch, so it's appropriate now for us to update our policies to account for it.

Abuse

Section 11 of our old Terms of Service included a long list of things that, if you did on our network, we could terminate you for. The history of this section is that I searched a number of other major services to see what they had prohibited and then included just about everything that had ever been listed. This list was largely pulled from hosting providers and similar sites that actually hosted content.

This list may be appropriate for a hosting service, but it isn't as appropriate for a network provider. CloudFlare is much more akin to a network provider. People also interpreted the list as if it was self-executing computer code. Someone would find a site that told people how to build a grenade, or whatever, and write to us saying we had to terminate them. We, on the other hand, saw the list as reasons we could terminate people, not reasons we must terminate them.

Given the confusion the list created we simplified it. Today our policy remains as it was before, just without the list. If you're using CloudFlare in a way we deem inappropriate we will, at our sole discretion, terminate your use of the CloudFLare network. As I've written about before, our general position is that CloudFlare is building a better Internet and it's not our role to determine what content should or should not be allowed to be published. That said, if you're using our network solely as a file locker, distributing malware or phishing, or otherwise causing per se harm then we will terminate use.

We also updated our abuse process to reflect what we've learned about running an abuse desk in front of hundreds of thousands of websites. What we learned was that as our technical defenses improved, hackers turned to abusing our abuse process to determine the identity of sites on our network. That, effectively, was a mechanism to bypass our technical protections. Our new abuse process allows legitimate rights holders to file complaints that we relay to the owners of sites with alleged violations without compromising the technical protections we offer our customers.

Miscellaneous Other Cleanup

There was a lot of other cruft in our terms that we cleaned up. For example, we previously included the following paragraph:

You are granted a limited, revocable, and nonexclusive right to create a hyperlink to any non-password protected directories, so long as the link does not portray CloudFlare, its affiliated websites, or its services in a false, misleading, derogatory, or otherwise offensive matter. You may not use any of CloudFlare's proprietary graphics or trademarks as part of the link without express written permission.

While most Terms of Service you'll find around the Internet include such paragraphs, they really are silly. We've deleted the paragraph so you can go ahead and link to our site, even if what you say is false, misleading, derogatory and offensive.

When we first started CloudFlare we also had something called the Automated Setup Tool that would login to your DNS provider and Registrar and make the changes for you if you gave us your username and password. While it was very cool and made the signup process even faster than it is today, we decided it was a very bad security practice to ask for people's username/password for a third party service. Much like we got rid of the Automated Setup Tool, we've now gotten rid of the section that covered how it worked. (Section 6 is now about Apps.) We also now provide software (e.g., mod_cloudflare and Railgun) so the terms were updated in various places to include that.

While I'm a recovering lawyer, I'm not a big believer that the legal system is the best way to resolve disputes. As a result, we added an arbitration clause. Should a dispute arise in the future, it seems like a more civilized way to resolve it. We also had some problems with machine translated versions of the Terms of Service containing oddities. As a result, we added a section to make it clear that the English version of the terms is the one that is controlling. We also moved from Palo Alto, CA to San Francisco, CA more than a year ago so we finally updated the jurisdiction information.

That's the gist of the updates. For those who are interested, we'll keep the old versions of the Terms of Service and Privacy Policies available for a few months. While I'm sure we'll have to make additional updates to the Terms of Service and Privacy Policies in the future as we learn more about running a global network, I am confident that we will continue to operate as we always have: respecting our publishers and their visitors' privacy, operating a responsible network, and working toward building a faster, safer, smarter web for everyone.

The forecast is cloudy says Ben Cherian

Kristin Tarr — Mon, 20 Aug 2012 18:38:00 -0700

Ben is a serial entrepreneur who loves playing in the intersection of business and technology. Ben is currently the Chief Strategy Officer of Midokura, which brings disruptive cloud networking technology to the world. His last role was as the GM of Emerging Technologies at DreamHost, where he ran the cloud business unit. Before that, Ben ran a cloud-focused managed services company.

During HostingCon, we were able to catch up with Ben to see what his thoughts are of the hosting and cloud space. First question, what is the cloud?

"This is how you explain it (the cloud) to your mom: Infrastructure-as-a-service is akin to going to Home Depot to build your furniture. Platform-as-a-service is like going to to Ikea. Software as a service is much more of a polished service, like going to Crate and Barrel or a furniture store, get it done and over with," said Ben, describing the services that make up the cloud.

"Every hosting provider has a cloud solution," said Ben. "Unfortunately, all the marketing has gotten in the way and the word ‘cloud' doesn't mean anything anymore."

There are a lot of exciting trends that Ben is seeing in cloud hosting, consumer products turning enterprise, and opportunities for major growth.

"I think there's going to be a lot more cloudy hosting providers coming out over time, over the next few years. That's pretty interesting I think," said Ben.

Ben also thinks there will be huge growth for enterprise offerings and startups who will fill a need not currently being met in the cloud space.

"There's a lot of space available for people to blow-up," said Ben.

Hear the rest of Ben's predictions in the full interview clip above.

At HostingCon 2012, CloudFlare co-founder Michelle Zatlyn sat down with 28 leading experts in the hosting industry. Their conversations were captured live and offer insight into the latest trends and news in hosting.

Spectators of the Cloud - Kenny Li from Cloud Spectator

Kristin Tarr — Fri, 17 Aug 2012 23:52:00 -0700

Kenny Li is co-founder and VP of Operations at Cloud Spectator. As a recognized leader in applied cloud intelligence, Cloud Spectator works together with enterprises and service providers in the cloud space. Cloud Spectator provides comprehensive industry reports, pricing comparisons, benchmarking software, as well as customized services.

"We actually provide market metrics into the cloud industry," said Kenny. "We realize there are a lot of reports out there and there are a lot of analysts out there, but no one gets down to the nitty gritty."

Cloud Spectator investigates and reports on what cloud users are really looking for. They use a variety of open source benchmark tests to provide data and peg it to pricing.

Looking forward, Kenny is excited to attend expos like HostingCon and learn more about providers and customers.

"The main goal of these expos is not only to network with cloud providers but ultimately understand the end user," said Kenny. "We want to be able to understand what cloud users are looking for so we can optimize our products to help them."

Tune in to hear more on Cloud Spectator and how they are like the "Consumer Reports" of Cloud Providers.

At HostingCon 2012, CloudFlare co-founder Michelle Zatlyn sat down with 28 leading experts in the hosting industry. Their conversations were captured live and offer insight into the latest trends and news in hosting.

“We provide a time machine for websites” - David Moeller from CodeGuard

Kristin Tarr — Thu, 16 Aug 2012 21:28:00 -0700

CodeGuard is led by CEO and co-founder David Moeller, a seasoned manager and entrepreneur with two successful exits in the last five years. Launched publicly at TechCrunch Disrupt – NYC in May 2011, CodeGuard provides cloud backup for websites, in addition to monitory and undo/restore capability.

We sat down with David at HostingCon (and started a friendly Nerf war with his staff) to find out what CodeGuard is up to these days.

"CodeGuard is a daily, automatic website backup tool," said David. "For anyone who has a website on a shared host or VPS dedicated, who has access, we do FTP, SFTP, MySQL and WordPress backup."

CodeGuard gives website owners the tools to easily fix hacks and revert changes, allowing website owners to turn back time on any damage done to their site.

"We provide a time machine for websites," said David.

CodeGuard has many different types of customers, including lawyers, doctors, florists and fitness experts.

"It's really a mix of anyone who has a website, but they don't have full-time web IT staff," said David.

In a sense, CodeGuard is like the superhero for these website owners who don't have the capabilities or staff to run their sites or fix problems that occur.

"We came out of the gates thinking people would really want us because they are so scared of being hacked and wanting to fix it and roll back, but what we've found is that people really enjoy us for just having a tool to manage this asset they have, which is their content," said David.

Saturday Night Fever: Layer 7 attacks against CloudFlare sites

John Graham-Cumming — Thu, 16 Aug 2012 10:58:00 -0700

Recently, I've taken a look at DDoS attacks against CloudFlare sites at the IP level and the source of those attacks. The worst time for those DDoS attacks is the Wednesday Witching Hour and because of source IP address forgery most of the attacks seem to come from Mars. But layer 7 attacks, where the attacker actually connects to our hardware using TCP and makes apparently valid HTTP requests are another matter: their source is traceable because of the need to establish a TCP connection.

Layer 7 attacks are in some ways the simplest attacks: an attacker performs lots of HTTP requests hoping to overwhelm the target server. To the target these requests look perfectly valid and have to be serviced. That uses up resources on the target server and either causes it to slow down or crash. CloudFlare's automatic system monitor unusual spikes in HTTP traffic and automatically deal with HTTP DoS attacks (often with a little help from our staff). At the same time the systems gather statistics about attacks.

Looking at our attack statistics we see a layer 7 DoS attack against a CloudFlare site 95.5% of the time. Those attacks come from just 0.05% of the IP addresses we see connecting to our network. There's virtually no rest for the systems (and people!) that deal with these attacks. The attacks come in the form of floods of HTTP requests made to the site that the attacker wants to knock off line. CloudFlare's systems record the IP addresses of the machines making layer 7 attacks because the address cannot be forged and are useful for filtering purposes.

Although the attacks come all the time the worst day is Saturday. The following chart shows the number of unique IP addresses use in layer 7 DoS attacks by day of the week for the period January to August 2012.

Focusing on the largest attacks shows the same trend with an uptick on Saturdays and layer 7 DoS attackers seeming to take a bit of a break on Sundays.

Looking at the time of day shows that attacks are occurring 24 hours a day with only a slight dip in the overall number of attacks around 0700 UTC (the middle of the night in California).

But focussing on the largest attacks reveals a pattern with which our team is familiar. The largest layer 7 attacks come during the night in California (around midnight, 0800 UTC) and then again at around 1800 UTC (just when the folks who've been up half the night fighting attacks are coming into work).

So whether it's night in California, or in Europe, the layer 7 DoS attackers keep the team busy.

The trend across the year shows some intriguing, and dramatic, dips in layer 7 DoS activity. The dips in the chart are around the following dates: January 30, February 21 (Mardi Gras), March 20 (attackers recovering from St. Patrick's Day?), April 22 (did attackers take Earth Day off, or did people switch off their home machines making botnets smaller for a day?), May 29 (Memorial Day weekend), June 28 (just before July 4).

The overall trend month on month is up. For the first 6 months of 2012 we say a 10% increase in layer 7 DoS attacks but a 21% increase in large layer 7 attacks. Statistics for lower level DDoS attacks show a slight decline. Attackers appear to be switching to layer 7 attacks to take sites offline.

Since the source of layer 7 attacks is known it's possible to look at where attacks originate (or at least where the machines performing the attack are). Most of these machines will be zombies taking part in botnets. The top five countries that attack CloudFlare sites are: 18.34% from US, 11.47% from China, 7.88% Turkey, 6.96% Brazil, 6.55% Thailand.

Focussing on the US the biggest networks that attack CloudFlare sites are: Verizon Online, Comcast, AT&T, Cox, Cablevision and Charter. That's consistent with the fact that attackers use botnets of machines connected to home broadband connections for their attacks.

Of course, at CloudFlare we spend a great deal of time defending against these attacks (both automatically and with tools like I'm under attack mode). And we've successfully defended small and large sites (such as the Eurovision Song Contest) against all layers of attack.

CloudFlare's mission is all about making sure our customers' web sites stay alive.

Ryan Kiskis from Basekit talks business, customers and growth trends

Kristin Tarr — Thu, 16 Aug 2012 00:44:00 -0700

Ryan is the Vice President of Product and Business Development at Basekit. Having been in the tech industry for almost a decade, Ryan is an experienced professional at both startups and Fortune 500 companies, with an MBA and engineering degrees.

We caught up with Ryan at HostingCon where we talked about Basekit, where he's seeing growth in the industry, and what their customers are saying.

"We have a platform for small businesses to get online. At the very basic part of that it is getting a website and getting a template that is relevant to what customers are doing," said Ryan. "We try to make that as slick and easy as possible."

Basekit provides customers with everything they need to create a professional website in an easy to use, completely customisable way. No coding required.

"The biggest thing, the reason they are picking Basekit, is that there's a ton of flexibility there," said Ryan.

Basekit's customers span from photography sites to ecommerce sites to small businesses of all kinds.

"It's a wild and wooly mix of small businesses out there," said Ryan.

CloudFlare meetup in Hong Kong

Kristin Tarr — Wed, 15 Aug 2012 21:50:00 -0700

CloudFlare's Network Engineer, Tom Paseka, will be in Hong Kong at the end of August. During his time there, we wanted to host a meetup for CloudFlare users and anyone interested in web performance and security.

The event will include a short presentation on what's new at CloudFlare, features we've added, and a preview of what's on the roadmap. Afterwards, Tom will answer any questions that you may have.

If you're a CloudFlare user, come introduce yourself to Tom. If you're not a CloudFlare customer, but interested in web performance and security, then you are welcome to join as well. We look forward to meeting our users and friends from China!

Meetup Details
Saturday, August 25
5:00pm
Hong Kong (exact location TBD)
RSVP & More information: http://www.meetup.com/CloudFlare-Meetups/events/77779312/

Note: We are still confirming our venue. If you have a suggestion, please let us know.

Vid Luther from ZippyKid “We can’t stop growing and we don’t want to”

Kristin Tarr — Wed, 15 Aug 2012 17:57:00 -0700

Vidis the founder and CEO of ZippyKid, the leader in WordPress hosting. Vid has taken the work he did at Third Party Code (the company he founded before ZippyKid), and productized it into ZippyKid.

ZippyKid focuses purely on WordPress, specifically hosting, securing, and scaling popular WordPress websites. Vid is a self proclaimed technology addict and is passionate about its implementation in everyday life.

We connected with Vid at HostingCon, where he shared with us their latest big news.

"We just raised money from 500 Startups and some of the founders of Rackspace and even Jason Seats, the founder of Slicehost," said Vid. "The company has just been growing like crazy."

ZippyKid's latest customers include Revlon and Estee Lauder, not to mention countless other small businesses and single-practice lawyers and doctors.

Looking ahead, ZippyKid plans to build more partnerships and hire, hire, hire.

"We are going to focus on more partnerships with people who provide really excellent tools for our customers," said Vid.

The San Antonio-based company is excited to build out their fast-growth company.

"We can't stop growing and we don't want to," said Vid.

At HostingCon 2012,CloudFlareco-founderMichelle Zatlynsat down with 28 leading experts in the hosting industry. Their conversations were captured live and offer insight into the latest trends and news in hosting.

Prague, Czech Republic: CloudFlare's 20th Data Center

Matthew Prince — Tue, 14 Aug 2012 23:53:00 -0700

We just turned up our newest data center (#20) in Prague, Czech Republic. This comes hot on the heels of new data centers in Vienna, Toronto, Seattle, Atlanta, and Sydney. Vienna, which came online last week, is already handling a high volume of traffic from Eastern Europe. Prague will take some of this load, decrease the traffic to Frankfurt and Amsterdam, and improve performance for web visitors in the area around the Czech Republic. Our network map will be updated soon to show all 20 of our data center locations worldwide.

Tom & Terry

At the same time we turned on CloudFlare's Vienna facility, we also enabled a new network provider for in-bound traffic. CloudFlare makes wide use of Anycast. The protocol allows routers in multiple different facilities to announce the same IP addresses. The network then chooses the route that is shortest when transiting the Internet. This both helps us ensure visitors reach the data center that is closest to them and also allows us to better mitigate DDoS and other attacks.

The challenge of Anycast is the network routing. It takes significant work to make sure that traffic for a particular region stays in the right place and that connections don't "flap" between multiple facilities. Tom and Terry, who head our network ops team, have been working hard over the last few weeks to ensure that as we continue to add more data centers and transit providers our network remains rock solid. Tom even wore an appropriate t-shirt to celebrate today's launch of Prague. "Czech" it out below.

We've got 3 more data centers (two more in Europe and one in Asia) we'll be turning up soon. Stay tuned!

Marco Houwen, LuxCloud CEO, has big visions for the cloud computing industry

Kristin Tarr — Tue, 14 Aug 2012 22:51:00 -0700

Marco, together with his partners LuxConnect and Datacenter Luxembourg, founded LuxCloud S.A. in April 2010. He has strong expertise in co-location, hosting, load-balancing, managing of e-commerce platforms and all associated managed services.

There are a lot of exciting things happening at LuxCloud and we were lucky enough to get an opportunity to hear about them from Marco himself.

LuxCloud is a market leading provider of cloud computing services, allowing companies to quickly launch and profitably deliver the cloud services demanded by small- and medium-sized businesses.

"We are positioning ourselves between the software and the channel," said Marco. "We are taking out the difficult part which is automation of the whole sales process."

LuxCloud is also ramping up to expand internationally and bring on twice as many employees as they have today.

"Right now we are setting up operations in the U.S. and Singapore, hopefully we will be going into South America by 2013," said Marco.

When asked about his time at HostingCon this year, Marco had big thoughts on what he was seeing in the cloud computing industry.

"We have to become aware that we are driving the world. The whole economy of the world with the cloud and SaaS services. That's a huge responsiblity that I see a lot of people taking on. It's thrilling."

Mladen Stojanovic from Atomia on hosting trends, challenges and what’s next

Kristin Tarr — Tue, 14 Aug 2012 00:39:00 -0700

An entrepreneur and big believer in automation, Mladen is the CEO and lead project manager of the Atomia platform. He is also the CEO of web development company Troxo and the creative mobile development company Dear Future Astronaut NY. Having been in charge of numerous projects for large web hosts, Mladen has a deep understanding and thorough knowledge of the hosting and DNS industry.

We caught up with Mladen at HostingCon to hear what Atomia is all about, what challenges they've overcome and what they are looking to do next.

"We do control panel automation, billing, domain and DNS," said Mladen. "We are like a full platform for hosting companies."

Once a host installs Atomia, they can start selling shared hosting and VPS right away. Atomia makes it easier for hosting companies to get started and serve their customers.

As for hosting trends going forward, Mladen thinks there's going to be a focus on usage-based types of hosting.

"We're thinking of supporting, as much as possible, a new type of packaging where the end user will pay as few or nothing at the beginning and just pay what they used in the end," said Mladen.

Having just launched in 2011, Atomia is focused on getting their name out there and getting more exposure. They offer something new in hosting and provide high reliability. We are excited to see a new company like this who focuses on making life better for hosting providers.

At HostingCon 2012, CloudFlare co-founder Michelle Zatlyn sat down with 28 leading experts in the hosting industry. Their conversations were captured live and offer insight into the latest trends and news in hosting.

Dallas Kashuba from DreamHost "Having personality helps build the brand and have a differential"

Kristin Tarr — Fri, 10 Aug 2012 19:39:00 -0700

Dallas started DreamHost back in college with a few other guys. They were inspired by all the possibilities of the web and wanted to bring that to as many people as possible. He's also a musician and all-around fun guy.

The fun guy has definitely transcended into the DreamHost culture. One of the largest hosting providers in the world, DreamHost is known for its fun atmosphere and laid-back employees.

"I think it's good to expose a little bit of your culture back out [to the public]. I don't think it hurts your product," said Dallas.

DreamHost, who issues a hilariously popular newsletter and doesn't shy away from the creative, shows a lot of personality in most of their work. They consider their customers "fans" and even hosted a "reach out and touch you" tour earlier this year.

"Having personality helps build the brand and have a differential," said Dallas.

It's not all fun and games for DreamHost though. As one of the most popular hosting providers out there, they are focusing on building a better product and expanding their network.

In the fall, DreamHost will be bringing on a new, east coast data center in Virginia to grow their network and better serve their customers. They are also developing a cloud computing service that is currently in alpha and scheduled to hit beta later this year.

At HostingCon 2012, CloudFlare co-founder Michelle Zatlyn sat down with 28 leading experts in the hosting industry. Their conversations were captured live and offer insight into the latest trends and news in hosting.

Russ Reeder from (mt) Media Temple on acquiring Virb and keeping customers happy

Kristin Tarr — Fri, 10 Aug 2012 03:09:00 -0700

Russ Reeder is President and COO of (mt) Media Temple, a top provider of web hosting and cloud services that powers 1.5 million websites for 125,000 customers in 100 countries worldwide. He's responsible for the (mt) brand's overall vision and strategy, and for driving scalable growth for the recently acquired Virb.com.

We caught up with Russ at HostingCon to get the latest scoop on (mt) Media Temple's recent acquisition.

"We just announced today that we acquired Virb," said Russ. "It's a great application for people to go in, create a blog or website, and within half an hour it's up and running."

(mt) Media Temple has always focused on web developers and designers as customers. With the acquisition of Virb, (mt) Media Temple has expanded to help even the least savvy tech folks out there start a blog or website.

One of the most notable characteristics of (mt) Media Temple is their customer support and culture.

"The culture is about the employees," said Russ. "Everyone has this sense of how to take care of our customers."

In the future, (mt) Media Temple is looking to continue their focus of adding value to their small business customers and provide more security and marketing services.

To catch all the latest (mt) Media Temple news, check out Russ' conversation with CloudFlare co-founder Michelle Zatlyn.

At HostingCon 2012, CloudFlare co-founder Michelle Zatlyn sat down with 28 leading experts in the hosting industry. Their conversations were captured live and offer insight into the latest trends and news in hosting.

Ryan Hurst talks site performance and SSL and how GlobalSign is growing geographically

Kristin Tarr — Thu, 09 Aug 2012 19:30:00 -0700

As the Chief Technology Officer for GlobalSign, Ryanis responsible for developing the overall technology vision, product architecture, standards development, compliance and overseeing the engineering and operations organizations.

When asked about how GlobalSign is addressing the issues of SSL and website performance, Ryan had a few things to say.

"There are a number of things that we have ongoing related to performance and SSL. One of which obviously has to do with our use of CloudFlare."

GlobalSign recently moved their revocation repository behind CloudFlare to give better performance and reliability for revocation services.

In addition to their partnership with CloudFlare, GlobalSign has recently sponsored improvements to the NGINX platform to help remove the dependency on customers needing to contact GlobalSign to retrieve revocation information at connect time, improving performances across the board.

The newest developments at GlobalSign aren't just their service improvements, but their physical growth as well.

"We're opening new engineering offices in Seattle and Manila (Philippines), said Ryan. "We're making some big investments in furthering our API support and a number of other platform improvements."

Tune-in to hear more on the improvements and growth of one of the leading Certificate Authorities.

*At HostingCon 2012, CloudFlare co-founder Michelle Zatlyn sat down with 28 leading experts in the hosting industry. Their conversations were captured live and offer insight into the latest trends and news in hosting. *

Attracta: Solving a big problem for every host

Kristin Tarr — Wed, 08 Aug 2012 23:52:00 -0700

Troy McCasland is the co-founder and Vice President of Business Development at Attracta. He has been at the company since its inception, working on the executive management team and building the business.

We sat down with Troy at HostingCon and asked, what exactly does Attracta do? "Attracta makes the worlds most popular search engine optimization platform," said Troy. "We solve a really big problem for every host here at HostingCon."

According to Troy, if a customer purchases a site through a host and then searches for their site in Google, only to find their site isn't there, it can be a big problem for the host. Attracta solves that problem by crawling the site and offering a XML Sitemap of the site to major search engines such as Google, Yahoo!, Bing and Ask.

Attracta currently has more than 2.3 million customers and adds over 100,000 customers a month.

Mike Auger confirms free ecommerce software - “Exciting Times at Pinnacle Cart”

Kristin Tarr — Wed, 08 Aug 2012 16:37:00 -0700

Mike Auger is the President and CEO of Pinnacle Cart, the shopping cart software that has been developed from the ground up. Mike bootstrapped the company to surpass $1 Million in sales and has established partnerships with the largest providers in the industry and the biggest brands in the United States.

Growth continues at a lightning pace for Pinnacle Cart and they've just announced something huge.

"We literally just announced that we are offering ecommerce software for free," said Mike. "We are big believers that the space is going into a commodity play and we partnered for our first offering with Chase and Parallels to offer a free version of Pinnacle Cart to customers."

With a competitive market and ecommerce software becoming a more commodity-based field, Mike said they felt now was the time to go with a free version of their service.

"We are going to be first to market with that play. It's big times for us," said Mike.

By offering a free service, Pinnacle Cart has now empowered any entrepreneur to get started, with the hope that once those businesses succeed and grow they will upgrade to paid services. Kudos to Pinnacle Cart for making the process to becoming a business owner easier and much more affordable.

Tune-in to the interview clip to hear more on Pinnacle Cart's free ecommerce software.

At HostingCon 2012, CloudFlare co-founder Michelle Zatlyn sat down with 28 leading experts in the hosting industry. Their conversations were captured live and offer insight into the latest trends and news in hosting.

Vienna, Austria: CloudFlare's 19th Data Center

Matthew Prince — Wed, 08 Aug 2012 06:11:00 -0700

CloudFlare's global expansion continues. After launching Sydney, Atlanta, Seattle, and Toronto over the last week and a half, this week we plan on turning up several more facilities in Europe. Today we're launching Vienna, Austria — our 19th global data center and number 5 of the 9 new cities we're launching as part of this round of expansion.

Europe Expansion

Eastern Europe is one of CloudFlare's fastest growing regions. Frankfurt currently serves most traffic from the region and, as a result, has become one of our busiest data centers. Vienna is one of the new facilities designed to help take some of the load off Frankfurt and generally improve performance throughout the region.

As part of this expansion, we're also increasing the number of transit providers we're using in the region for in-bound transit. This will both improve overall performance and also increase the resiliency of our network if there are ever issues with one of our current providers.

Getting Lost in Vienna's Bermuda Triangle

I was last in Vienna for RSA Europe a number of years ago. While I did spend time in the city's appropriately named Bermuda Triangle (Bermudadreieck) district — home to many of the city's best pubs, bars and restaurants — I have never seen the city's famous Lipizzaner Stallions. I'm hopeful that now that CloudFlare has a data center there I'll have an excuse to go back. Until then, enjoy the following YouTube video.

Elliot Noss from Tucows announces Ting

Kristin Tarr — Wed, 08 Aug 2012 00:14:00 -0700

As president and CEO of Tucows Inc., Elliot leads the company's business strategy and vision. He is responsible for overseeing operations for the company's domain registration, domain portfolio, email and retail lines of business. Elliot has been a leader in the Internet industry for over a decade. He champions areas of vital interest to the service providers and Internet users including privacy, ICANN reform and registrar matters, and the implications of emerging technologies.

When we sat down with Elliot at HostingCon, we were expecting to hear the newest updates on Tucows' hosting, domain name and email services. What we didn't expect was their announcement of their mobile service, ting.com. Just like Tucows changed how domain registration worked over the last 10 years, Tucows figured the mobile industry needed the same overhaul and launched Ting.

"It's a flat-out mobile service. We will be your phone carrier - the kinder, gentler phone carrier," said Elliot. "We decided to just reinvent the whole thing from the ground up and be the mobile carrier you wish you had."

Serving just the United States for the next two to three years, Ting has been on the market for about six months and customers are already loving it.

"Customers are thrilled. They are saving a lot of money and they are loving the experience," said Elliot.

Currently Ting is only available on Android devices, but Elliot and his team are working to get the iPhone supported as well.

We are excited for Tucows' venture in the mobile market and look forward to the service taking off over the next couple of years. Tune in to the full interview to hear details about Ting.

At HostingCon 2012, CloudFlare co-founder Michelle Zatlyn sat down with 28 leading experts in the hosting industry. Their conversations were captured live and offer insight into the latest trends and news in hosting.

"We never sit still" - Jack Zubarev from Parallels

Kristin Tarr — Tue, 07 Aug 2012 20:16:00 -0700

Jack Zubarev is a founder and President of Parallels. Since the company's founding, he has held multiple leadership roles within the company, including running the Marketing and Alliances teams, COO, Division President, sales engineering, product marketing, and business development. Jack has been instrumental in building the company from its start to today with more than 800 employees.

Today, Jack is excited about how quickly Parallels has grown and the recent release of Plesk 11.

"We are growing very quickly," said Jack. "Over 9,000 service providers and over 10 million small businesses are using our platform through their service providers."

When asked how he sleeps at night, knowing that many people depend on them to keep their sites online, Jack is quite honest, "I don't," he says.

In addition to the release of Plesk 11, Parallels is focusing on their web presence builder product and tools, building it to be as intuitive, seamless and widely used as possible.

We couldn't pass up the opportunity to ask about a future Plesk 12, in which Jack told us "we never sit still. We are definitely working on the next updates."

Watch Jack's full interview with CloudFlare's co-founder Michelle Zatlyn to hear the details of Parallels growth and new products.

At HostingCon 2012, CloudFlare co-founder Michelle Zatlyn sat down with 28 leading experts in the hosting industry. Their conversations were captured live and offer insight into the latest trends and news in hosting.

Ben Metcalfe from WP Engine weighs in on a new breed of hosting, using social media for support, and how he missed out on @ben

Kristin Tarr — Tue, 07 Aug 2012 01:45:00 -0700

Ben Metcalfe is the definition of an Internet professional. With more than 11 years industry experience under his belt, Ben has previously spent time working on the development of MySpace and websites at the BBC.

He is the founder of two companies, Swordfish Corp and WP Engine. Having used the Internet since 1992, Ben is passionate about the Internet's ability to enrich, inform, educate and entertain our lives and connect us with others.

CloudFlare was lucky enough to catch up with Ben to hear the latest on his newest company, WP Engine.

"We are one of a new breed of webhosts, we are a managed WordPress hosted platform," said Ben. "What that means is we only support WordPress, we are not a traditional hosting account. You point a domain at our IP address, and we provide you an instant WordPress site that's running on our platform. We can immediately make sure that (your site) is set up in a way that is most optimum in way of speed, security, performance and scalability."

Having been around the Internet for some time, it's no surprise that Ben is active in social media and encourages his company and staff to have a social media presence.

"We use Twitter a lot at WP Engine," said Ben. "I think it's important that the business itself and my employees get their own following and reputation. It's a really tricky medium for support, but we use it to highlight and add value, encouraging customers through Twitter to use email support."

Ben's biggest mistake in social media? Not claiming @ben when he had the opportunity in 2006.

Listen in to hear the latest on WP Engine and what's next for the company.

At HostingCon 2012, CloudFlare co-founder Michelle Zatlyn sat down with 28 leading experts in the hosting industry. Their conversations were captured live and offer insight into the latest trends and news in hosting.

Chris Sheridan from eNom talks predictions for 2013 and Jersey Shore

Kristin Tarr — Mon, 06 Aug 2012 20:47:00 -0700

Chris Sheridan, Vice President of Business Development at eNom, has been in the domain and hosting business since 1998. Chris has spent the past five and a half years at eNom focusing on retail sales, reseller sales and business development.

We sat down with Chris last month to discuss what's new at eNom and how their business is looking to grow. We also uncovered the connection between HostingCon and the hit TV show, Jersey Shore.

"eNom is primarily known for our large reseller network, 8,000+ resellers. Primarily our business is domain name registration," said Chris. "Over the last couple years years, we have been very focused on our value added services, so we're heavy now into security services, like SSL."

Two of the biggest growth areas for eNom are international expansion and the historic announcement in June of more than a 1,000 new TLD's coming on the Internet next year.

"It sounds corny, but who knows when this will happen again," said Chris, with regards to the TLD expansion. "It'll bring in a whole new wave of resellers. Overall, it's great for everyone in this room."

Tune-in to see Chris' full interview and get the exculsive on his rumored nickname "CWOWW."

At HostingCon 2012, CloudFlare co-founder Michelle Zatlyn sat down with 28 leading experts in the hosting industry. Their conversations were captured live and offer insight into the latest trends and news in hosting.

Mars Attacks!

John Graham-Cumming — Mon, 06 Aug 2012 16:06:00 -0700

Following on from my recent post about when attacks hit CloudFlare, here's a follow up looking at where they come from. Or at least where they say they come from. Looking at attack statistics for the month of July 2012 the largest source of attacks is Mars.

Of course, not literally from Mars itself, but from what network engineers refer to as 'Martian IP addresses': IP addresses that are otherwise valid but shouldn't appear on the public Internet. For example, many corporate networks use the 10.0.0.0/8 IP address range, and many home users will be familiar with addresses in the 192.168.0.0/16 address range. Both of those addresses are valid, but not valid on the public Internet. When machines in a home or corporate network actually communicate onto the public Internet those internal addresses are converted to a public address by a technique called Network Address Translation (or NAT). If you're on a home or corporate network right now you can find your public IP address using the CloudFlare-powered WhatIsMyIP.com.

Full details of the reserved blocks of IP addresses can be found in RFC 5735.

We see vast numbers of attacks with IP addresses such as 192.168.91.84 (apparently from inside a network), 127.0.0.1 (apparently from inside the machine that's being attacked--spooky), 10.99.88.226 (inside a corporate network), and 169.254.85.75 (used on a local, private link).

(Thanks, XKCD)

Because these addresses are invalid on the public Internet when we build our attack statistics they appear as an invalid network. Looking at the top, potentially spoofed, networks that attack CloudFlare shows that invalid accounts for 23% of attacks, followed by 3.45% from (apparently) China Telecom, 2.14% from China Unicom, 1.74% from Comcast, 1.45% from Dreamhost and 1.36% from WEBNX. And then, on and on, for 37,284 different networks around the world. With a total of 41,838 networks, we've apparently been attacked during July by 89% of the networks on (this) planet.

It's also not surprising to see China Telecom high on the list: it's the largest network in the world announcing to the Internet that it has 114,212,832 addresses.

Of course, this traffic didn't necessarily actually come from those networks, and certainly not from Mars, unless NASA's Curiosity dropped off some servers on its way down, because the source IP address is being spoofed or forged.

The attackers can spoof the source IP address (where the packet says it comes from) because they do not need a reply. In most cases the attacker is simply seeking to overwhelm a CloudFlare Internet connection, or server, with traffic. There's no need for a reply at all, just a huge flow of packets. Forging the IP address means that the source cannot be easily determined.

And in some cases the IP address is legitimate but actually the IP address of an innocent victim. In the case of DNS or SNMP reflection attacks where a legitimate DNS or SNMP server is fooled into sending packets to attack CloudFlare, the source IP address is a real server. CloudFlare sees a great number of reflection attacks against our DNS infrastructure.

DNS reflection works like this: an attacker sends a DNS query to a DNS server on the Internet with a forged source address. The source address being forged is set to the address of a DNS server at CloudFlare that the attacker wants to overwhelm. The innocent DNS server that receives the packet sends a reply to the source adderss at CloudFlare. Of course, CloudFlare never sent the original packet, by address spoofing made it look like we did.

Of course, the attacker doesn't do this with one DNS server, they do it with hundreds or thousands so that CloudFlare suddenly receives a huge wave of DNS replies we didn't ask for. And, ironically, we get frantic calls from network managers asking us to stop attacking them. Since they see the source IP address is CloudFlare they, naturally, assume we've sent the traffic.

Worse, reflection attacks also have an amplification effect as well. Since a DNS query packet can be very small (60 bytes, for example) and the reply much larger (e.g. 512 bytes), it's possible for an attacker to amplify the bandwidth available to them by sending thousands of small packets to DNS servers which respond to the CloudFlare server with a large packet. That means that the attacker uses a small amount of outgoing bandwidth to hit CloudFlare at a much greater rate.

The upshot is that when looking at layer 3/4 attacks the source IP is mostly useless: it's likely bogus or the address of a victim not an attacker.

So, Curiosity can trundle across the Martian landscape safe in the knowledge that CloudFlare's network team won't be firing back any response to those Martian packets.

Always Online v.2

Matthew Prince — Sun, 05 Aug 2012 21:33:00 -0700

The video on CloudFlare's home page promises that we will keep your web page online "even if your server goes down." It's a feature we dubbed "Always Online" and, when it works, it's magical. The problem is, Always Online doesn't always work.

This blog post is to announce that we've just released a new version of Always Online which we believe will make the feature significantly better. But, before I get to that, let me tell you a bit about the history of Always Online, how it has worked up until recently, and why it didn't always work. Then I'll turn to what we've done to create Always Online v.2.

An Accidental Feature

Prior to starting CloudFlare, Lee and I ran Project Honey Pot. The Project Honey Pot website is database driven and contains a virtually infinite number of pages. One of the biggest challenges we had wasn't human traffic, which followed a predictable browsing pattern and could therefore reliably be cached, but instead dealing with traffic from automated crawlers.

These crawlers, whether legitimate (e.g., Google's bot) or illegitimate (e.g., spam harvesters), tend to crawl very "deep" into sites. As a result, they hit pages that are unlikely to have been crawled in a while and, in doing so, can impose significant load on a database. I've previously written about the hidden tax web crawlers impose on web performance.

At Project Honey Pot, Lee built a number of sophisticated caching strategies in order to help lessen the load of automated crawlers on the site's database. At CloudFlare, he realized that we could provide the same type of caching in order to cut the burden bots placed on backends. In essence, we automatically cache content for a short amount of time and, if it hasn't changed since the last request from a bot, deliver it without having to burden your web application. It works great.

In the process of building the bot content cache, Lee realized he could implement something else: a system to serve static versions of pages if an origin server fails. Using human traffic to build such a cache is dangerous because you don't want to expose one user's private information to another user (e.g., we can't cache when one user visits their bank's website to view their statement and then show that statement to another user). However, search engine crawlers are the perfect anonymous user to build a site's cache. The logic was: if it's in Google, then it's already effectively cached.

Good, Not Perfect

The approach of using known search engine bot traffic to build CloudFlare's cache was clever, but it had some problems. The first was that CloudFlare runs multiple data centers around the world and the cache in each is different. The solution was to find the data center with the most search engine crawler traffic and, if a copy of the page didn't exist in the local data center's cache, fall back on the "master" data center. In our case, our Ashburn, Virginia data center received the most crawl traffic so we added a lot more disks there and used it to build up the Always Online cache.

That worked great for some sites, but for others we still would not have content in our cache when the server went offline. Seemingly bizarrely, the more static the page the less likely it was to be in our cache. The explanation was the source of the cache data: search engine crawlers. These crawlers are generally setup to visit pages that change regularly more often, and for pages that rarely change only occasionally. If the page returned a 304 "Not Modified" response then the content didn't get recached. We didn't help things by automatically expiring items in our cache after a period of time.

The net result was, far too often, when someone's site would go offline their visitors wouldn't see a cached version of the page but, instead, a CloudFlare error page telling them that the site was offline and no cached version was available. This became one of the top complaints from our users and the visitors to their sites. When our support team dubbed the feature "Always Offline" we knew it was time to make it better.

Version 2

We made a number of improvements in how we cache pages in order to improve Always Online, but the biggest change we made was to begin to actively crawl pages ourselves. CloudFlare now runs a crawler which periodically crawls our customers' pages if they have the Always Online feature enabled. The crawler's useragent is:

Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +http://www.cloudflare.com/always-online )

You can learn more about the crawler's behavior by visiting: www.cloudflare.com/always-online. The frequency that we refresh pages in the Always Online depends on your plan. We crawl free customers once every 9 days, Pro customers once every 3 days, and Business and Enterprise customers daily. We are tinkering with the amount of time we spend crawling each site as well as tuning the crawler to ensure it doesn't visit sites when they're under load or otherwise impose any additional burden.

Given that we can now control exactly what is in our Always Online cache, our next iteration will be to turn that control over to our users and allow you to both "pin" the pages you want to ensure are always available and "exclude" any pages you never want cached. In the meantime, we're using data we have about the most popular portions of each site in order to choose what pages to prioritize in the cache.

Our goal is to make the Site Offline error a thing of the past. We started building the new cache a couple days ago and expect everyone with Always Online to have a more robust cache available within the next few days. While everyone hopes their origin server will never go down, with Always Online v.2 we're happy to provide better peace of mind in case it ever does.

The Wednesday Witching Hour: CloudFlare DoS Statistics

John Graham-Cumming — Fri, 03 Aug 2012 17:06:00 -0700

Data from inside CloudFlare's network shows that over 40% of the time there's a denial of service attack happening and directed at us. And that's just up to network layer 4 (i.e. it doesn't include more sophisticated attacks targeting applications themselves at layer 7).

Still of the Night

Those attacks literally keep our network engineers awake at night: the busiest time is during the night in the USA. This graph shows the number of 'attack minutes' (number of attacks in each sampled minute; often our engineers are dealing with multiple attacks at the same time) by UTC hour. The peak corresponds to morning in Europe and the deep, dark night in the US.

The attacks also keep them busy all week, but, like many of us, attackers seem to be at their best mid-week when they've shaken off those Monday morning blues and aren't winding down for the weekend. This graph shows the number of attack minutes by day of the week.

So, the worst time for DoS attacks is in the middle of the night from Tuesday to Wednesday: the Wednesday Witching Hour. But the real message of those graphs is that DoS attacks simply never let up: they're happening 24/7.

And attackers try everything to bring us and sites on us down. The following graph shows the breakdown of DoS attacks by IP protocol: UDP just nudges past TCP as the majority as reflection attacks using both DNS and SNMP have become very popular. One SNMP reflection attack hit CloudFlare with an aggregate data rate of 21Gbps late last year.

As CloudFlare is a protection and acceleration service for web sites it's not surprising that 92% of the DoS attacks using TCP are on port 80 (HTTP); and on UDP 97% are against port 53 (DNS). But we've also seen DNS attacks on TCP port 53 and UDP attacks on port 870 and 514 (syslog). Looking into TCP, SYN flooding remains the favorite attack method with 84% of the attacks.

Ironically, DNSSEC is currently making some DNS reflection attacks worse because of the large amount of data that DNSSEC can return. Attackers make EDNS0 requests to servers that are able to interpret them; they do that from forged IP addresses resulting in a large amount of data (in the form of valid EDNS0 replies) hitting a target IP range.

Carpet Bombing and Drive-Bys

We've also seen attackers increasing the intensity of attacks by 'carpet bombing'. To knock off a single web site we see attackers attempting a TCP SYN to the web site's IP addresses, SYN flooding against the DNS server handling the web site and DNS reflection and then the same thing across the entire /24 IP range handling the web server and the entire /24 IP range handling the DNS server.

Those massive attacks keep our network engineers up at night keeping CloudFlare web sites online and fast. But the overall trend in attacks has been slightly down over the last 6 months. We believe that attackers are becoming aware of CloudFlare's DoS protection and are switching to other attack methods (such as trying to break into web sites and not just knock them off line) and we've seen attackers try sophisticated technical and social engineering attacks to break into CloudFlare.

(Image credit: Flickr user philcampbell)

The other trend is the use of 'booter' web sites to knock other web sites off for a short period of time. These attacks last less than five minutes and appear to be a show of strength by hackers wishing to demonstrate that they can remove a web site from the Internet. Unlike long running DoS attacks designed to make a political point, or cause a business to lose money, these drive-bys are hackers flexing their DoS muscles.

In a future post I'll look at the attacks we see at layer 7 and how our engineers and firewalls keep them at bay.

Toronto, Canada: CloudFlare's 18th Data Center

Matthew Prince — Fri, 03 Aug 2012 06:43:00 -0700

Last week we turned up three new data centers in Sydney, Atlanta, and Seattle. This week we are launching the next location to continue to expand CloudFlare's global network. Our ops team just put the final touches on our latest North American facility in Toronto, Canada. Traffic has begun flowing through the facility which will build up its cache over the next 24 hours. Going forward, it will take load away from Chicago and Newark and improve CloudFlare's performance in much of Canada.

It's Aboot Time!

CloudFlare has deep Canadian DNA. Michelle, who co-founded CloudFlare with Lee and me, grew up in Saskatchewan, Canada. She attended college at McGill University in Montreal then began her career in Toronto. If you appreciate CloudFlare's easy-to-use, no-nonsense interface, and our relentless focus on building a product that is great for our users, you have Michelle to thank.

Beyond her work at CloudFlare, Michelle works to ensure that the Internet ecosystem is as healthy as possible. She was invited to represent Internet technology startups on the United States Federal Communication Commission's Open Internet Advisory Committee. At CloudFlare we're working to build a better Internet. That extends beyond the technology we deploy in data centers around the world, but also to work Michelle and other members of our team do to create the policies that will ensure a healthy and vibrant web worldwide.

Given CloudFlare's deep connections to Canada it was high time we opened a facility there. What's next? Assuming some cooperation from customs in a few countries in Europe and Asia, we'll be bringing five more facilities online over the coming several days. Stay tuned!

SPDY Now One-Click Simple for Any Website

Matthew Prince — Fri, 03 Aug 2012 00:34:00 -0700

About a month and a half ago, CloudFlare announced limited support for SPDY as part of a private beta. SPDY is the new protocol pioneered by Google to make the web faster. If you're curious, we've written about what makes SPDY speedy in previous blog posts.

Since that announcement, we've been testing SPDY with a couple hundred of CloudFlare's customers, as well as on CloudFlare.com itself, in a private beta. The results have been great and today we're excited to announce that SPDY is now available to any eligible CloudFlare customer from their Performance Settings page.

Who Gets Speedy?

The current implementation of SPDY requires TLS/SSL. As a result, SPDY is only supported for paying CloudFlare customers who have SSL enabled. Even if you don't have your own SSL certificate installed on your server, you can take advantage of SPDY by enabling CloudFlare's Flexible SSL. If you're a Free customer, you can upgrade to one of CloudFlare's paid plans and enable SPDY immediately. If, in the future, the SPDY protocol supports non-HTTPS connections, we plan to extend SPDY support to Free customers as well.

Assuming you have SSL enabled, you can turn SPDY on with a single click for all traffic that passes through CloudFlare. SPDY will automatically be enabled for HTTPS traffic to browsers like Chrome and the latest version of Firefox which supports the protocol.

With widespread SPDY support, we're excited to continue to push the web forward as we continue our mission of building a faster, safer Internet.

CloudFlare (aka. KickassDNS)

Matthew Prince — Wed, 01 Aug 2012 19:17:00 -0700

One of the things we don't talk about much at CloudFlare is how we've built one of the largest, fastest, easiest, and most resilient authoritative DNS networks in the world. A report from SolveDNS was just released that shows CloudFlare as the second fastest authoritative DNS provider, well ahead of companies EasyDNS, UltraDNS, and Verisign.

Company	Average Speed (ms)	Min	Max	Standard Deviation	AnyCast
He.net	6.77	4.49	10.28	2.37	Yes
CloudFlare	7.45	4.44	33.48	3.80	Yes
Dyn	8.37	4.81	21.42	4.01	Yes
DNSMadeEasy	9.46	4.27	17.70	5.10	Yes
VerisignDNS	22.34	5.13	56.43	24.04	Yes
Netriplex	29.81	5.38	82.96	28.13	Yes
UltraDNS	44.96	4.39	81.44	26.47	Yes
Nettica	48.97	29.14	79.95	19.12	No
ZoneEdit	64.65	18.19	142.16	38.62	No
EasyDNS	76.42	3.70	163.86	62.08	Yes

(data from SolveDNS August 1, 2012 report)

CloudFlare runs an authoritative DNS network in order to make provisioning our performance and security services as easy as possible. Because we're relentlessly focused on making the web faster, we built our DNS infrastructure to be as fast as possible. While we give you two name server domains when you sign up for CloudFlare, the reality is that those domains reference clusters of servers in each of our 17 (soon to be 23) data centers worldwide. That's quite different than the average registrar's DNS infrastructure, and part of the secret on how we're so fast.

Any Server Can Answer Any DNS Query

We've written about Anycast before, but it's an important technology we use to make DNS fast and highly available. Where traditionally, one IP corresponds to one server on the Internet, with Anycast multiple servers in distinct locations announce the same IP address and traffic is automatically routed to the one that is closest to the system making the request. If a server (or entire data center) goes offline, traffic immediately and seamlessly fails over to the next-closest location.

Since CloudFlare is provisioned via DNS, we set out from the beginning to make the fastest, most fault-tolerant DNS network in the world. In fact, every server we run in every data center around the world can answer any DNS query for any one of our clients. While this is built for redundancy and stability, the side effect is that it's extremely fast. And, as we continue to build out our global network, our DNS speeds will continue to get even faster.

The DNS providers listed in the SolveDNS chart all specialize in DNS service and most run Anycasted networks (although few are as large as CloudFlare's). Many people continue to use their registrar's DNS, which is puzzling since they're often overloaded and slow. Since every request for a domain starts with a DNS query, a slow DNS provider is a hidden tax on your website's performance. With fast, free solutions like CloudFlare, it's a puzzle why anyone puts up with slow DNS.

KickassDNS

When CloudFlare was first getting started we knew that DNS was going to be critical. Before we deployed a single server we started to focus on every aspect of DNS, even the control panel for customers to interface with it. We'd been frustrated by the state of DNS control panels, whose UI providers woefully underinvest in. We spent several months surveying every DNS provider in the world to see if we could build a better DNS UI. The result, we think, is the easiest to use DNS control panel in the world.

Beyond ease of use, the way that CloudFlare's DNS infrastructure works means updates are extremely fast. It takes less than a second for a change from the DNS control panel to be propagated across our entire network. And, since you can change your backend server's IP address without having to change the IP CloudFlare announces to the world, the result is you can change from one backend server to another without having to wait for DNS propagation. It's pretty slick.

We're continuing to make additional improvements to both our DNS infrastructure and how it is deployed by our customers. One of the requested features from businesses was that they wanted custom DNS servers that used their own domain name, not CloudFlare. As a result, that feature is now included with all Business and Enterprise plans. Going forward, CloudFlare will begin offering its massive DNS infrastructure and custom name servers to hosting provider partners so they can ensure their customers have the fastest, most resilient DNS service without requiring any name server changes.

So while we don't talk about it much, we're spending a ton of time thinking about DNS. As John Graham-Cumming on our team just suggested, "We clearly need to change our name to something like KickassDNS." That's probably not going to happen. But, if you've ever hesitated to sign up for CloudFlare because you were concerned about changing your DNS, chances are we'll be significantly faster and more resilient than whatever you were using before.

CloudFlare’s Newest App Partner: Verelo

Kristin Tarr — Tue, 31 Jul 2012 15:58:00 -0700

Verelo is a website monitoring service that provides sub-minute checks, downtime alerts by SMS, phone or email, and malware detection.

A group of CloudFlare customers have recently been testing Velero during a silent launch, testing our latest app integration. Deployment has been very successful and with over 100 CloudFlare customers using Verelo, we have decided it's time to publicly announce them as our newest app partner.

Verelo is a unique company. Started by two individuals (Mike and Andrew) who previously worked in the hosting space, Verelo provides webmasters constant monitoring of their sites. Because of their background, Mike and Andrew were able to identify a need that hosting providers weren't able to provide for their customers and so they launched Verelo as an answer to the site and server monitoring need.

Features

Velero offers prevention, detection, responses and reporting for malicious activity on any website, making running a site that much easier.

Features include:

Downtime alerts by SMS, Phone or Email
Malware detection
Response time and uptime graphs
Sub-minute monitoring
911 conference call system

Plans

Starting at free, Velero offers a variety of plans to suit your needs:

Get started!

Verelo is now available to all CloudFlare customers. For more information on the app and how to get started, visit the app detail page today: https://www.cloudflare.com/apps/verelo

App: Infolinks Earns You Money With In-Text Ads

John Roberts — Tue, 31 Jul 2012 02:25:00 -0700

At CloudFlare, we're interested in making running a website easier. That's a broad definition, on purpose, going far beyond security and performance.

If your goal is making money from your website, we want to make that easier, too.

So we're pleased to introduce Infolinks to CloudFlare customers. Available now, the Infolinks app will automatically add in-text ads to your website.

How It Works

Infolinks turns keywords into relevant ads, which only trigger when a website visitor hovers over the link, as in the screenshot below. Infolinks operates in real time, extracting content and determining intent. The automatically-created in-text ads overcome "banner blindness" delivering a richer experience for your visitors right on your website. You earn revenue when your visitors click on the ads. Monthly payments are sent to your bank or PayPal account.

To enable Infolinks, you don't need to change the layout of your site. Instead, turn on the app in your CloudFlare dashboard, and upon approval, the Infolinks code will be applied by CloudFlare to your sites, automatically turning keywords into relevant ads.

A Few Things to Note

Infolinks is appropriate for publishers with rich content.

Infolinks is not compatible with every website. Websites with adult or offensive content are not allowed. Every website goes through a manual review process by the Infolinks team.

Turn on the Infolinks app today and start making money from your website with in-text ads.

Seattle: CloudFlare's 17th Data Center

Matthew Prince — Fri, 27 Jul 2012 05:17:00 -0700

Our ops team is on a roll! Hot on the heels of Sydney and Atlanta going live yesterday, we're happy to announce Seattle, Washington as the location of CloudFlare's 17th data center. We just turned on the facility and traffic will ramp up over the next few hours. The facility will help serve traffic from the Pacific Northwest and take load off San Jose, which is one of our busiest data centers currently.

"Water Is Blue, Not Land. Duh!"

This update also marks a dramatic change to the CloudFlare Network Map. For quite some time, the map matched the overall color scheme of CloudFlare's site and represented the land mass with a blue color. You'd be amazed how many customers wrote in to tell us that we'd made a mistake and the water should be blue, not the land. It became a bit of a joke around the office.

It is therefore with some sadness I report that Kevin, our designer, has given in to the masses and switched away from the blue land network map. The new map represents everything as shades of gray. I find it somewhat more ominous, but hopefully it'll end people's confusion.

Oh, and in the process of writing this, I just noticed we crossed 400 billion page views through our network and 40 petabytes of bandwidth saved for our users, which is pretty cool too. Stay tuned, six more data centers coming online soon. Any guesses what city in what country we'll turn on next?

HostingCon: The Launch of our Optimized Hosting Partners program, Railgun Nerf guns, Limos, Breakfast and much more!

Meg He — Thu, 26 Jul 2012 17:55:00 -0700

Team CloudFlare had an incredible time at HostingCon last week - thanks to everyone for dropping by and chatting with us! We loved catching up with many of our existing partners and meeting potential new ones.

It's difficult to put such an exciting (and exhausting) week into words, especially when there was so much happening at the conference. Here's a summary of our HostingCon 2012 by the numbers...

Limos: On Sunday, we chauffeured nearly 200 HostingCon attendees to their hotels in style in our three black, stretch limos (CloudFlare: delivering you, fast and safe). We loved seeing the photos and tweets of the rides from our limo passengers, which made our 14 hours at all four terminals of Boston Logan International Airport fly by. We'll be back with our limos again next year, so be sure to look for our email to sign up.
Breakfast: We hosted three days of breakfast from Monday to Wednesday, preparing HostingCon attendees for their action-packed days on a full stomach. I especially liked our coffee cups: CloudFlare, delivering your coffee, fast and safe.
Optimized Hosting Partners: Fun stuff aside, we also did some pretty serious work at HostingCon. We launched our Optimized Hosting Partners Program, allowing Optimized Hosts access to Railgun, our revolutionary new optimization product, for free. We are extremely pleased that seven of our current Partners are participating in our Optimized Hosting Partners' launch (VEXXHOST, A2 Hosting, Media Temple, Blue Host, DreamHost, HostPapa and Site5)
- we hope that many of our existing and new partners will join this list in the very near future.

Railguns: Of course, we couldn't possibly launch our Optimized Hosting Partners Program without any goodies. Our Optimized Hosting Partners are getting Railgun, so we brought you...real Railguns! The CloudFlare team worked hard over the weekend before HostingCon to box-and-unbox 500 Railgun Nerf guns for your pleasure. The Nerf guns were loaded with over 3,000 limited edition CloudFlare Optimized Nerf darts. We're really excited about Railgun (both the product and the Nerf version) and we hope you are too.

Interviews: Our co-founder, Michelle, conducted live interviews with 20 thought leaders in the hosting industry, including founders and executives. Who will be featured? Stay tuned in the next few weeks to find out as we publish all of these videos online.
Panels: Our other co-founder, Matthew, spoke on two panels during the conference on Railgun's integration into our Optimized Hosting Partners Program and Technology Strategies and Practices for Defending Against DDoS Attacks.

T-shirts: We gave out hundreds of our latest CloudFlare t-shirt! We now have six different designs - do you have them all?

We had an awesome time at HostingCon - here's all of our photos if you want to relive your memories. If you want a Railgun Nerf battle, this is an open invitation to come and visit CloudFlare in our San Francisco office to take on the team! We hope you had a great journey back from Boston. #zoomzoom

Atlanta: CloudFlare's 16th Data Center

Matthew Prince — Thu, 26 Jul 2012 05:46:00 -0700

We're not slowing down! Just a few hours ago we launched Sydney, Australia as CloudFlare's 15th data center. Now our ops team just threw the switch and turned on Atlanta, Georgia as CloudFlare's 16th. I told you to stay tuned! These are the first two of nine new facilities we're turning up over the next few weeks.

I'm not sure what to say about Atlanta. It's the home of Coca-Cola, which our team drinks a lot of. In fact, John Graham-Cumming, one of our engineers who works from London, objects whenever we make him fly Virgin Atlantic to San Francisco because they only serve Pepsi.

The other connection is my and Michelle's former business school classmate, Fred Smith. Fred was very supportive when we were working on the original plan for CloudFlare and now lives in Atlanta where he's working on opening a nightclub. Fred also happens to be one of the stars of Bravo's new reality television show "Taking Atlanta" (formerly known as "Atlanta's Most Eligible"). Tune in this Fall!

Another connection to Atlanta is it is home to one of our App partners: Codeguard. Codeguard makes it easy to make sure your website is always backed up. The founder and CEO, David Moeller, may not be on Taking Atlanta, but he is certainly running one of Atlanta's hottest startups. Interestingly, David was previously on another reality TV show "American Inventor" where he pitched The Claw bicycle mount. What's with Atlanta and reality TV? Weird.

Anyway, here's to Fred, Coca-Cola, Codeguard, Atlanta, and more CloudFlare data centers! More to come, stay tuned...

Sydney, Australia: CloudFlare's 15th Data Center

Matthew Prince — Thu, 26 Jul 2012 00:40:00 -0700

Over the next few weeks, CloudFlare will be significantly expanding our global network. In total, we'll be adding 9 new data centers and doubling the size of our existing facility in London. When we're done we'll have 23 global data centers and nearly 70% more network capacity. I'm excited to announce the first of these 9 new facilities just came online: Sydney, Australia.

We choose the locations of our data centers in large part based on where we can most improve network performance. Australia has been one of the problematic regions for network providers. In CloudFlare's case, traffic from Australia has been served from our Singapore or Los Angeles facilities. In either case, ping times were over 160 milliseconds. With our new Sydney facility, ping times from Australia and New Zealand are now averaging under 40 milliseconds.

Straight to the Pool Room

The challenge of opening a facility in Australia has been the cost. Bandwidth is in the region is notoriously expensive. We talked with bandwidth providers for almost a year without much luck. Finally, Tom, a CloudFlare network engineer who happens to be Australian, suggested we watch the movie The Castle. After that, whenever we'd get a call from a bandwidth provider in the region, Tom would ask them, "How much?" He'd relay the price to me and I'd simply say, "Tell 'em they're dreamin'."

Anyway, the rest played out pretty much just like in The Castle. We eventually wore down the big, bad bandwidth providers. And, without having to kick anyone out of their home, we now have found one of our own down under.

The new routes are propagating now and all Australia and New Zealand traffic should be hitting Sydney within the next 24 hours. If you're in the region and the Internet starts feeling faster, now you'll know why. Stay tuned here for updates as we turn on the rest of the 9 new data centers over the next few weeks.

Introducing: Single File Cache Purge

Matthew Prince — Sat, 14 Jul 2012 07:51:00 -0700

CloudFlare has supported a way to purge your cache for the last year. Unfortunately, it was all or nothing. While that, for practical purposes, didn't cause a significant performance hit for most websites, since the cache would be rebuilt with the most popular files automatically and quickly, it still seemed inefficient. Why purge the whole cache, customers asked, when only a single file was updated?

The answer is technical: for a number of reasons based on how we store cache it's easier to mark all files as expired than to single out a single resource. And so, for the last year, purging everything or waiting for the cache to expire were our users only options.

Single File Purge

We're excited today to announce single file purge. The feature allows you to purge the cache of any URL without affecting any other cached files. To access the feature, from your My Websites page visit CloudFlare Settings and select Single File Purge from the Cache Purge section. There you can enter the URL of the object you want to purge from the cache. Hit the Purge button and, typically within less than a second, all the nodes in CloudFlare's global network will fetch a new copy of the file.

The feature is provided to all CloudFlare users, regardless of your plan type. Moreover, it's available via our API. We'll be providing more information over the next week, but our hope is developers will create systems that will detect when files on a server have been updated and automatically send an update to CloudFlare's network to ensure that the latest copy is fetched.

Thoughts on Abuse

Matthew Prince — Fri, 13 Jul 2012 23:47:00 -0700

One of the behind the scenes topics we think about a lot at CloudFlare is how to handle abuse of our network. I realized that we hadn't exposed our thoughts on this clearly enough. In the next few days, we'll be making some minor updates to our Terms of Service to better align it with how we handle abuse complaints. However, I wanted to take the time to write up a post on how we think about abuse. Make sure you're comfy, this is going to be a bit of a marathon post because it's an important and complicated issue.

CloudFlare sits in front of nearly a half a million websites. Those websites include banks, national governments, Fortune 500 companies, universities, media publications, blogs, ecommerce companies, and just about everything else you can find online. Every day we process more page views through our network than Amazon.com, Wikipedia, Twitter, Zynga, Aol, eBay, PayPal, Apple, and Instagram — combined. That's dumbfounding given that CloudFlare is only a year and a half old from our public launch.

Problem Sites

While the vast majority of sites on CloudFlare are not problematic, just like on the Internet itself there are inevitably some unsavory organizations on our network. Almost exactly a year ago, I blogged about the notorious hacking group LulzSec using CloudFlare's services and our decision not to terminate their service. As I wrote a year ago:

CloudFlare is firm in our belief that our role is not that of Internet censor. There are tens of thousands of websites currently using CloudFlare's network. Some of them contain information I find troubling. Such is the nature of a free and open network and, as an organization that aims to make the whole Internet faster and safer, such inherently will be our ongoing struggle. While we will respect the laws of the jurisdictions in which we operate, we do not believe it is our decision to determine what content may and may not be published. That is a slippery slope down which we will not tread.

Today there are hundreds of thousands of sites using CloudFlare and we remain concerned about the slippery slope. To be clear, this isn't a financial decision for us. LulzSec and other problematic customers tend to sign up for our free service and we don't make a dime off of them. When they upgrade they usually pay with stolen credit cards, which causes us significant headaches. The decision to err on the side of not terminating sites is a philosophical one: we are rebuilding the Internet, and we don't believe that we or anyone else should have the right to tell people what content they can and cannot publish online.

Who We Terminate

There is no more thankless job than running an abuse desk. In the last week, our abuse team has had to deal with "senior Iranian officials" threatening us over the fact that a pro-Israeli website was on our network while, at the same time, dealing with threats from an Israeli group who was extremely upset that a website supporting the Iranian regime was also on our network. We didn't terminate either of those sites.

No matter how repugnant an idea may be to one person or another, we don't believe we are qualified to act as judge. There are, however, at least two clear cases where we believe our network can cause harm and therefore we do take action: spreading malware or powering phishing sites.

Originally, when we would receive reports of phishing or malware we would terminate the customers immediately. The challenge was that this didn't actually solve the problem. Since we're just a proxy, not the host, us terminating the customer doesn't make the harmful content disappear. Terminating the site effectively just kicked the problem further down the road, moving it off our network and onto someone else's.

(photo credit: Erectus Bee)

This was unsatisfying to our abuse team so we reached out to the experts on the issue of malware and phishing at StopBadware. StopBadware is the organization Google trusts to explain about phishing and malware when they detect problems on pages that appear in the company's search index. We worked with StopBadware to design a Google-like block page that we can display on pages where malware or phishing are detected. This solution actually eliminates the knowm malware and phishing from our network and, at the same time, teaches visitors who may have been fooled by the malicious content about its risks.

This sounds easy — and, as a matter of policy, it was easy — but, technically, it was actually extremely tricky to implement. To give you some sense, we average about 150,000 requests per second through our network and we're doubling every 3 months or so. To make the block pages work, we needed to check every one of those requests against regular expressions that match known phishing or malware sites. All without slowing down requests. It took us longer than I would have liked to find a solution that could scale, but now that it is in place we are actively adding data sources to ensure we promptly remediate any malware and phishing sites on our network.

The Rock and the Hard Place

While we believe we have found a good solution for malware and phishing abuse reports, other abuse requests still present a vexing issues. Originally, when CloudFlare received a DMCA complaint for an alleged copyright infringement, our practice was to turn over the IP address of the site's host to the person filing the complaint. This allowed them to then take the issue up with the hosting provider.

CloudFlare has become very, very good at stopping online attacks, including DDoS attacks. As a result, people launching those attacks have begun looking for ways to bypass our protection. Starting about a year ago, we saw a spike in what turned out to be illegitimate DMCA requests. They would look technically correct, include all the required information, but the complaintant wasn't the actual copyright holder but an individual looking to attack the site. As soon as we turned over the origin IP address they would launch an attack, completely bypassing CloudFlare's protection. In other words, attackers were abusing our abuse process — a problem I wrote about when discussing how SOPA could make things even worse.

(photo credit: Rojak Daily)

If there is a way to reliably tell the difference between a legitimate and an illegitimate DMCA abuse complaint, we haven't found it. As a result, we adjusted our abuse process in order to meet the requirements of the law and allow legitimate complaintants to serve notice to infringers, but not expose our customers to attacks.

In many ways, our abuse flow today is also a sort of reverse proxy. When we receive a complaint, after some checks to ensure it's validity to the extent possible, we forward a copy of the complaint to the site owner via email. We also send a copy of the complaint to the site's hosting provider, including the site's origin IP address and instructions on how they can test to ensure that the site is, in fact, hosted on their network. We then respond to the complainant explaining how CloudFlare works, how we've relayed their complaint, and providing the identity of the site's actual host (although not the site's actual IP address).

We are continuing to refine the process over time to maximize two goals: ensuring our customers are protected from attacks, and ensuring that we don't stand in the way of legitimate complaintants. If you have suggestions on how we can improve the process while balancing these interests, we welcome your input.

CloudFlare Heads to HostingCon 2012

Kristin Tarr — Fri, 13 Jul 2012 18:26:00 -0700

CloudFlare is going to be at HostingCon 2012 in Boston from Sunday, July 15th to Wednesday, July 18th. We are excited to see our current partners and meet potential new partners at the show! Our team is looking forward to an extremely busy conference and we have several fun events planned, including complimentary limousine service between Boston Logan International Airport and the Boston Sheraton Hotel, a daily CloudFlare hosted breakfast, an exhibit booth and of course, attending all of the parties.

If you are already one of our certified hosting provider partners, be sure to stop by and introduce yourself. We will have CloudFlare t-shirts and some mystery goodies to celebrate the launch of Railgun. If you are not a partner yet, stop by to learn more about how CloudFlare can save you server and bandwidth resources, provide DDOS protection, IPv6 compatibility and much more.

Our schedule for the week is as follows:

Sunday, July 15th
Limo transfers from Boston Logan International Airport to the Boston Sheraton Hotel.

Registration is now closed for the limos, however, if you send an email to limos@cloudflare.com with your flight number, airline and time of arrival, we will try to accommodate you.

Monday, July 16th
8am-10am: CloudFlare sponsored breakfast in Room #312
11am-12pm: Our co-founder and CEO Matthew Prince speaks on Railgun in the 3rd level meeting rooms
5pm onwards: Come and find the CloudFlare team at the welcome reception!

Tuesday, July 17th
8am-10am: CloudFlare sponsored breakfast in Room #312
11:30am onwards:CloudFlare is in the Exhibit Hall at Booth #809.
1pm-6pm: Conversations with CloudFlare - We will be conducting live video interviews with thought leaders within the hosting industry, including founders and executives.
3:30pm: Our co-founder and CEO Matthew Prince will be speaking on the DDoS panel discussion "Technology Strategies and Practices for Defending Against DDoS Attacks"in the 3rd level meeting rooms

Wednesday, July 18th
8am-10am: CloudFlare sponsored breakfast in Room #312
11:30am onwards: CloudFlare is in the Exhibit Hall at Booth #809.

Connect with us on Twitter during the event to find out where we are and what's coming up next:

#hostingcon, @hostingcon, @CloudFlare

See you next week in Boston!

Stop worrying about Time To First Byte (TTFB)

John Graham-Cumming — Thu, 05 Jul 2012 09:17:00 -0700

Time To First Byte is often used as a measure of how quickly a web server responds to a request and common web testing services report it. The faster it is the better the web server (in theory). But the theory isn't very good.

Wikipedia defines Time To First Byte as "the duration from the virtual user making an HTTP request to the first byte of the page being received by the browser." But what do popular web page testing sites actually report? To find out we created a test server that inserts delays into the HTTP response to find out what's really being measured. The answer was a big surprise and showed that TTFB isn't a helpful measure.

When a web browser requests a page from a web server it sends the request itself and some headers that specify things like the acceptable formats for the response. The server responds with a status line (which is typically HTTP/1.1 200 OK indicating that the page was available) followed by more headers (containing information about the page) and finally the content of the page.

CloudFlare's TTFB test server behaves a little differently. When it receives a request it sends the first letter of HTTP/1.1 200 OK (the H) and then waits for 10 seconds before sending the rest of the headers and page itself. (You can grab the code for the TTFB server here; it's written in Go).

If you ask WebPageTest to download a page from the CloudFlare TTFB server you get the following surprise. WebPageTest reported the Time To First Byte as the time the H was received (and not the time the page itself was actually sent). The 10 second wait makes this obvious.

Exactly the same number is reported by gomez.

The TTFB being reported is not the time of the first data byte of the page, but the first byte of the HTTP response. These are very different things because the response headers can be generated very quickly, but it's the data that will affect the most important metric of all: how fast the user gets to see the page.

At CloudFlare we make extensive use of nginx and while investigating TTFB came across a significant difference in TTFB from nginx when compression is or is not used. Gzip compression of web pages greatly reduces the time it takes a web page to download, but the compression itself has a cost. That cost causes TTFB to be greater even though the complete download is quicker.

To illustrate that we took the largest Wikipedia page (List of Advanced Dungeons and Dragons 2nd Edition Monsters) and served it using nginx with and without gzip compression enabled. The table below shows the TTFB and total download time with compression on and off.

                          |  TTFB   |  Page loaded

--------------------------- | ------- | ------------- No compression (gzip off) | 213us | 43ms Compressed (gzip on) | 1.7ms | 8ms

Notice how with gzip compression on, the page was downloaded 5x faster, but the TTFB was 8x greater. That's because nginx waits until compression has started before sending the HTTP headers; when compression is turned off it sends the headers straight away. So if you look at TTFB it looks as if compression is a bad idea. But it you look at the download time you see the opposite.

From the end user perspective TTFB is almost useless. In this (real) example it's actually negatively correlated with the download time: the worse the TTFB the better the download time. Peering into the nginx source code we realized we could cheat and send the headers quickly so that it looked like our TTFB was fantastic even with compression, but ultimately we decided not to: that too would have negatively impacted the end user experience because we would have wasted a valuable packet right when TCP is going through slow start. It would have made CloudFlare look good in some tests, but actually hurt the end user.

Probably the only time TTFB is useful is as a trend. And it's best measured at the server itself so that network latency is eliminated. By examining a trend it's possible to spot whether there's a problem on the web server (such as it becoming overloaded).

Measuring TTFB remotely means you're also measuring the network latency at the same time which obscures the thing TTFB is actually measuring: how fast the web server is able to respond to a request.

At CloudFlare TTFB is not a significant metric. We're interested in optimizing the experience for end users and that means the real end-user page being visible time. We'll be rolling out tools specifically to monitor end-user experience so that all our publishers get to see and measure what their visitors are experiencing.

Go at CloudFlare

John Graham-Cumming — Tue, 03 Jul 2012 17:21:00 -0700

The other day I blogged here about our new Railgun software that speeds up the back haul between CloudFlare data centers and our clients' servers. At CloudFlare we're using a number of different languages depending on the task: C or C++ for all core services, PHP for the main web site, Lua for customization of nginx and an extensive amount of JavaScript. Railgun is slightly different as it's about 4,000 lines of Go of which about 3,000 are code (not comments).

(Image source: stanleylieber.com; created by Renée French)

We chose to use Go for Railgun because Railgun is inherently highly concurrent. A single instance of the Railgun client should be able to handle large numbers of requests from the CloudFlare data center for content and then multiplex them across an Internet connection to be handled. Go's concurrency makes writing software that must scale up and down very easy.

Railgun makes extensive use of goroutines and channels. Goroutines handle both the multiplexed Internet connections (of which there could be many between a single CloudFlare data center and our clients) and the connections needed to get content from origin web servers and provide the content back to the nginx server that sends it on to the web browser.

Probably the nicest thing about goroutines and channels is that they make it easy to create 'fire and forget until needed' systems. You create a channel, create a goroutine that communicates on that channel and then read from the channel when needed (perhaps using a select statement).

(Aside: for those who studied computer science, Go owes a lot to Hoare's CSP and Dijkstra's Guarded Commands.)

A small example of a goroutine inside Railgun is this unique ID generator. It generates a sequence of IDs that are used to identify streams (a stream contains a single HTTP request) being sent between the CloudFlare data center and a client site.

It works by adding data to a SHA1 hash and each time a read is made on the channel id a new string ID is created by hashing the data. The whole thing is running as an independent goroutine that only does work when needed. (You can play with this code live here).

Another powerful aspect of Go is the way in which it handles object orientation. Go has a notion of an interface which is used to identify a capability of an object and where it can be used. One common interface is io.Writer. To be an io.Writer an object has to implement the Write function which has the signature Write(p []byte) (n int, err error). Any object that implements that can be used wherever an io.Writer is needed.

In Railgun there's a simple object called a Counter that is an io.Writer. It turns an ordinary io.Writer into one that writes, but also counts how much it's written. When its Write is called it keeps track of the number of bytes and calls the underlying io.Writer. It looks like this:

As an example of its use, here's how the unique ID generator above can be altered to count the number of bytes of data that have been written to the SHA1 hash. Since h implements io.Writer it can be passed to counter.New and it can be used to write data to the hash and keep a count. Reading from the count channel would retrieve how many bytes had been written. (See the live version for an example).

Part of the reason Railgun is so small is that Go's library is extensive and easy to work with. Go has libraries for HTTP, raw network connections, URL manipulation, TLS, many different types of serialization systems, cryptographic hashing, compression, and the more mundane string manipulation, date/time, and logging.

Another reason Go has been helpful is that is generates a single executable that can be distributed to our clients. There's no complex dependency chain or layout of shared libraries to worry about.

Making Edge Side Includes (ESI) Automatic and Easy

Matthew Prince — Tue, 03 Jul 2012 09:00:00 -0700

On HTTP, caching is done at the file level. A browser will cache the JPEG, CSS, and Javascript files on a page. However, the HTML of most pages is dynamically generated. As a result, the pages cannot be cached. This is unfortunate because the HTML of even highly dynamic pages rarely changes more than 10%. The 90% of the HTML that is the same from one request to the next is transmitted needlessly.

On the web, compression equals performance. If you can compress a response by 50% you will, roughly, double network performance. Given that 90%+ of HTML doesn't need to be transmitted over the network, if you could only transmit the actually dynamic parts of the content then you'd get a massive performance increase.

Last Gen Solution: Edge Side Includes

Recognizing this opportunity, traditional content delivery network (CDN) vendors created the Edge Side Include (ESI) protocol. The protocol was submitted as an official standard to the World Wide Web Consortium (W3C) but it was never accepted. A handful of other old school CDNs today support ESI, although it's adoption has been slow.

Here's how ESI works: when you create a web page you determine what parts are static and dynamic. You implement the static portions as a file that you upload to your CDN. Within that file you include tags that reference the dynamic portions of the content, with URL of where to fetch the dynamic portions from. The CDN fetches each of these dynamic resources and combines them with the static portion in order to render the HTML of the page before it sent across the network back to the browser.

If that sounds easy to implement then you likely haven't done much web development. To get a sense of the complexity, check out this 106 page ESI developer's guide. While ESI can theoretically deliver significant performance benefits, the pain of actually developing for it is significant. And, once you've developed for it, there's significant process lock-in: good luck ever leaving. We think you shouldn't have to learn a new programming language or change a single line of your code just to make your site fast.

And if you're spending your time bending your HTML so that a CDN can serve it up better then you're not spending it on developing your actual web site.

Next Gen: Faster, Easier, Better

Yesterday, we posted about Railgun and how it lets you cache what was previously uncacheable content. One way of thinking about Railgun is that it is like automatic ESI support without the work. Rather than you having to tag your own content to mark what is static and what is dynamic, Railgun automatically determines the static portions of HTML and caches that at the edge. Dynamic portions of HTML are always fetched from the origin without you needing to change a single line of code.

Moreover, the caching logic is responsive to what is actually happening on the page. If different elements on the page change at different rates then Railgun's cache will deliver them optimally, never wasting a byte that doesn't otherwise need to be transmitted. And, since you don't need to change how you write code in order to support Railgun, there's no process lock-in if you ever decide to turn the service off. While you won't get the benefits of Railgun without CloudFlare, you won't need to completely rewrite your code. In fact, you won't need to change a thing.

CloudFlare: We Fight for the Publishers

We talk to a lot of web publishers and the constant refrain we hear is that the performance and security tools that are available to them are too expensive and too complicated. We've had nearly half a million websites sign up for CloudFlare largely because we focused on these two issues. Railgun takes ESI, another technology that was previously reserved for only those sites with huge budgets and dedicated CDN management teams, and makes it available in a way that is affordable and easy to implement.

Because Railgun requires software to be installed on the origin server, we have limited its availability to our Business and Enterprise customers. However, our plan is to roll it out to CloudFlare's Free- and Pro-level customers if they are hosted on a CloudFlare Optimized Hosting Partner. If you're interested in Railgun, you can upgrade to CloudFlare Business or Enterprise. Alternatively, ping your hosting provider to know they should become a CloudFlare Optimized Host. It's free for hosts and, if they tell us you're the person who convinced them to sign up, we'll send you a T-shirt and make sure you're one of the first of the host's customers to get access to Railgun.

Caching the uncacheable: CloudFlare's Railgun

John Graham-Cumming — Mon, 02 Jul 2012 07:44:00 -0700

CloudFlare recently rolled out a premium service called Railgun that's available to CloudFlare Business and Enterprise customers. Railgun is web optimization software that's designed to speed up the delivery of content that cannot be cached.

One of the major advantages of using CloudFlare is that cacheable content (such as images, JavaScript, CSS and HTML) is both cached by CloudFlare and delivered from our data centers around the world. Because CloudFlare has data centers covering the entire globe, cached content gets delivered quickly to web surfers wherever they are (and overcomes latency problems).

But only about 66% of content is cacheable. The other 34% must be obtained from the real origin web server. Railgun overcomes this problem by using a scheme that is able to cache dynamically generated or personalized web pages dramatically reducing bandwidth used and improving download times.

(Image credit: DaveFayram)

Caching the Uncachable

Railgun works by recognizing that uncacheable web pages do not change very rapidly. For example, I captured the CNN homepage HTML once, and then again after 5 minutes and then again after one hour. The page sizes were 92,516, five minutes still 92,516 and one hour later 93,727.

CNN sets the caching on this page to 60 seconds. After one minute it's necessary to download the entire page again. But looking inside the page itself not much has changed. In fact, the change between versions is on order of 100s of bytes out of almost 100k. Here's a screenshot of one of the small binary differences between the CNN home page at five minute intervals. The yellow bytes have changed, the rest have not:

Experiments at CloudFlare has revealed similar change values across the web. For example, reddit.com changes by about 2.15% over five minutes and 3.16% over an hour. The New York Times home page changes by about 0.6% over five minutes and 3% over an hour. BBC News changes by about 0.4% over five minutes and 2% over an hour.

Although the dynamic web is not cacheable, it's also not changing quickly. That means that from moment to moment there's only a small change between versions of a page. Railgun uses this fact to achieve very high rates of compression. This is very similar to how video compression looks for changes from frame to frame; Railgun looks for changes on a page from download to download.

The Technical Details

To a web client the Railgun system looks like a proxy server, but instead of being a server it's a wide-area link with special properties. One of those properties is that it performs compression on non-cacheable content by synchronizing page versions.

The sender then reconstructs the page from its cache and the difference sent by the other side.

Blowing the Doors Off Standard Gzip

Of course, compression is used on web pages today. The most common technique is to gzip the page itself. CNN actually does this and sends 23,529 bytes of gzipped data which when decompressed become 92,516 bytes of page (so the page is compressed to 25.25% of its original size). And Google has proposed a somewhat complex dictionary based scheme called SDCH which is not widely deployed.

But the Railgun compression technique goes much further. The compression between versions 1 and 2 of the page above (at five minute intervals) results in just 266 bytes of difference data being sent (a compression to 0.29% of the original page size). The one hour difference (versions 2 to 3 above) is 2,885 bytes (a compression to 3% of the original page size). Clearly, Railgun compression outpeforms gzip enormously.

Railgun means that for premium CloudFlare customers the entire web becomes (almost completely) cacheable. Learn more about Railgun or upgrade to a CloudFlare Business or Enterprise account to enable it for you site.

Protecting CloudFlare sites from phishing

Damon Billian — Sun, 01 Jul 2012 21:28:00 -0700

Editor's Note: This post was co-authored with Ray Bejjani, CloudFlare's engineer that was the lead for this project.

As the internet has grown, phishing attacks have continued to be a problem. While better awareness and focus on security has helped reduce their number, the RSA and other services tracking phishing still show that somewhere between 20-30,000 phishing attacks generally occur every month. In the past, criminals used to launch most of these phishing attacks through email but they have now switched to hacking sites to better hide their tracks and dupe more unsuspecting consumers into divulging their personal information unknowingly to fraudsters. This can become a difficult problem for less technical users that wish to participate and contribute to the World Wide Web.

Given our position in the internet ecosystem, we often receive phishing reports via our abuse channel. Previously, this meant a manual process to notify the appropriate parties especially the site owner. We felt we could serve our customers better, and their customers in turn, with a solution unique to CloudFlare. When we have identified a URL that is phishing we notify the owner and provide them summary information and notifications when they log in. We also begin serving a warning page in place of the bad URL. This page can be bypassed by the visitor at their preference.

Why did CloudFlare create this new anti-phishing process?

Our mission is to help make the web faster and safer.
We can block phishing pages as soon as they are reported to us. This provides the following benefits to sites using CloudFlare:
- Stops site visitors from potentially falling victim to identity theft
- Stops site owners from being penalized for unknowingly hosting phishing content. Many search engines will blacklist your site if you're hosting malicious content, which only compounds the issue for site owners that don't know that they have been compromised
- We can quickly notify site owners about the issue on their site quickly so they can clean up the malicious files.

How do I protect my site from future hacks and phishing attempts?

If you're interested in protecting your site, whether you have been hacked or not, you can take the following steps that can secure your site:

Use SSL on your site to encrypt information between your site and your server.
In addition to providing a layer of security to your site already, CloudFlare has partnered with a number of app providers that can help further protect site owners from malicious intrusions and provide additonal site monitoring.
Always update your site's CMS platform, plugins and server software. If you have a notification from your provider that there is a software update available, these updates were probably done to fix known exploits that have shown up since the last release to the plugin or platform. Since doing these updates often only takes a few minutes or so, you can save yourself from a potential world of hurt by doing it "now" instead of "later".

What should I do if CloudFlare has notified me of phishing pages on my site?

CloudFlare will send you an email advising you as to what pages on your site are phishing content. You will also see a message on your 'My Websites' page when a domain has pages blocked for a phishing report, with a link to take you to more information about the report.

Steps you should take if you receive an email from us or see a message on your dashboard:

If you are an experienced web administrator, chances are you already know how to remove the page(s) from your site. Remove the pages in question and then request a review that will be processed by the abuse team.
If you are not an experienced web administrator, we would recommend that you contact your hosting provider for assistance in removing the pages. You should then request the review so we can confirm the phishing pages have been removed.

Where should I report a site on CloudFlare that has a phishing page?

If you see a site on the CloudFlare network that has a phishing page, please report the site to us via our abuse form.

Please be sure to include the following in the report:

The domain in question
The actual page that the phishing link is located on.

We have more exciting things on the way to help protect internet users from phishing and malware. Please stay tuned to CloudFlare updates for more developments.

Thoughts on the AWS outage: making the cloud more resilient to failure

Matthew Prince — Sat, 30 Jun 2012 21:56:00 -0700

A huge storm rolled across the eastern United States last night, topping trees and knocking out power. Amazon Web Services (AWS) had one of their primary data centers in Virginia lose power. While data centers typically have backup generators for when they lose power from the grid, it appears something in the backup systems failed and AWS's EC2 Northern Virginia region went offline. That took down much of Netflix, Pinterest, Instagram, and other services that rely on Amazon's cloud hosting service.

My favorite comment on the incident came from Phil Kaplan (@pud) who tweeted: "The cloud is no match for the clouds." It got me thinking about the different types of "cloud" services, their different sensitivities to failure, and how they can be made more resilient.

Cumulus, Stratus, Cirrus, Nimbus

There are a lot of different products that call themselves "cloud" services. What that means, however, is very different from one service to another. For example, Salesforce.com was among the first to trumpet the benefits of the cloud. In their case, they were comparing themselves against traditional customer relationship management (CRM) systems that required you to run your own database and maintain your own hardware. In Salesforce.com's case, "cloud" means you can run a specialized application (CRM) on someone else's equipment and pay for it as a service.

For their core CRM product, Salesforce.com runs their own hardware in multiple locations around the world. However, Salesforce.com purchased another "cloud" service provider called Heroku. Heroku was originally built as a platform to run applications written for the Ruby programming language. It has expanded over time to provide support for other languages including Java, Node.js, Scala, Clojure and Python. Where Salesforce.com's original cloud service allowed you to run their CRM application as a service, Heroku lets you run any application you want from their managed platform.

Salesforce.com runs on the company's own servers, but Heroku runs atop Amazon's AWS service. In other words, Heroku provides a cloud service that makes it easier to write and deploy your own applications, but they use someone else's infrastructure to deploy it. Before everyone started calling all these "cloud" services, the analysts gave them more specific names that started with a letter and always ended with "aaS." Salesforce.com was Software as a Service (SaaS), Heroku was Platform as a Service (PaaS), and AWS was Infrastructure as a Service (IaaS).

I'd add a further distinction: the three cloud services I've mentioned so far are all what I'd call Data & Application (D&A) cloud service. In one way or another, they let you store data and process without having to think about the underlying hardware. They may all be cloud services, but they are very different from what we're building at CloudFlare (more on that in a bit).

Servers All the Way Down

In Hindu mythology, there story that talks about how the world is supported on the back of a giant turtle. Steven Hawking's book A Brief History of Time included an anecdote about a scientist giving a lecture to the public on the structure of the universe:

At the end of the lecture, a little old lady at the back of the room got up and said: "What you have told us is rubbish. The world is really a flat plate supported on the back of a giant tortoise." The scientist gave a superior smile before replying, "What is the tortoise standing on?" "You're very clever, young man, very clever," said the old lady. "But it's turtles all the way down!"

While it's easy to forget with the abstractions provided by these services, under all these clouds are servers, switches, and routers. If you're using Salesforce.com for CRM and your company adds a large number of new customers, you don't need to think about adding more drives or servers to scale up. Instead, Salesforce.com handles the process of adding capacity across its hardware. If you're developing on top of a cloud service like AWS's EC2, as your application scales you can "spin up" new instances to provide more computational power. These instances are fractions of the capacity on a physical server which may be shared with other EC2 users. Because each EC2 customer only uses whatever is necessary for their application, the utilization rates across the servers is very high.

When Clouds Go Boom

It is inevitable that the hardware that makes up these clouds will, from time to time, fail. Spinning hard drives crash, memory goes bad, CPUs overheat, routers flake out, or someone disconnects the wrong power circuit bringing a whole rack of equipment offline. When those pieces of hardware fail, different cloud services will react in different ways.

Salesforce.com runs their own hardware and their own software. They have created systems that replicate the application itself across multiple hardware systems. If one system fails, a load balancer switches to a different hardware system to process the request. Customers' data stored with Salesforce.com is also replicated by the software. While I don't know the explicit details of Salesforce.com's redundancy strategy, it's a safe bet that they use RAID to replicate data between multiple disks that are part of a storage array and backup to some long term storage in case of a major failure. They also likely replicate data between multiple storage arrays within a particular data center and, maybe, replicate the data between data centers.

Replicating data is relatively easy. Replicating data and keeping it in sync is hard. The problem becomes harder if the locations are geographically separated. The speed of light is very fast, but it still takes a photon of light traveling under perfect conditions nearly 60ms to roundtrip from San Francisco to Amsterdam. It's slower through the actual fiber and copper cables that make up the Internet, and much, much slower when you take into account the real world performance of the Internet. If two people change the same piece of data in two locations during the latency window between updates, very unpredictable bad things can happen.

The Challenge of Being in Sync

For certain systems, replicating data is easier than others. Compare Google and Twitter. If you're running a search on Google you'll hit one of the company's many geographically distributed data centers and get a set of results. Someone else running a different search hitting a different data center may get slightly different results. Google doesn't promise that everyone will see the same search results. As a result, they have a relatively straight forward data replication problem. The data that makes up Google's index will be "eventually consistent" across all their facilities, but that doesn't harm the underlying application.

Twitter, on the other hand, promises that you'll see real time updates from the people you follow. This creates a much more difficult data replication problem and explains why Twitter has a much more centralized infrastructure and continues to experience many more scaling pains. Facebook provides an interesting case study as well. As Facebook has scaled, they have deemphasized real time updates to timeline in order to make it easier to scale their infrastructure.

Twitter, Facebook, Google (with their new emphasis on products that require more data synchronization), and a lot of other smart people are working ways to mitigate the problem of data replication and synchronization but the speed of light is only so fast and at some level you'll always bump into the laws of physics. What is key, however, is that choosing to host in the cloud alone is not sufficient to ensure your application is fault tolerant. The data and application layer remain difficult to scale, and even with a service like AWS creating resiliency still requires programmers to make their application servers redundant and replicate their data to the extent possible and practical.

Front End Layer Scaling

While data synchronization makes geographic scaling of the Data & Application layer difficult, there is a part of the web application stack that is a natural candidate for massively distributed scaling: the Front End layer. All web services have a front end. It is the part of the service that receives the requests and hands it off to the application to begin churning. The front end layer also returns the response from the application and databases back to the user that requested it.

Unlike the Data & Application layer, the Front End layer doesn't need specific knowledge about the application. This means it can be distributed geographically without special application logic or a need for complex data replication strategies. The Front End layer can help tune the response from the Data & Application layer depending on the characteristics of the user. For example, rather than the Data & Application layer changing the presentation of a response based on whether someone is on an iPad or Internet Explorer on a desktop PC, the Front End layer can handle the response.

The Front End layer can also shield the Data & Application layer from potential threats and attacks. In fact, if you use a protocol like Anycast to route requests geographically, you can isolate attacks or any network problems to only impact a small part of the overall system.

Front End In the Cloud FTW

While there remains hesitation in some quarters to turn over the Data & Application layer to the cloud, moving the Front End layer to the cloud is a no-brainer. That, of course, is exactly what we've built at CloudFlare: a scalable front end layer that can run in front of any web application to help it better scale. What's powerful is that any web site or application can provision CloudFlare simply by making a DNS change. Since the Front End layer doesn't need to synchronize data, CloudFlare can begin working to accelerate and protect web traffic immediately and without any changes to the Data & Application layer.

Because we've focused on the Front End layer, CloudFlare's scaling and failure characteristics are very favorable to traditional Data & Application cloud services. Today, for instance, we had a hardware failure in our San Jose data center. Most customers never noticed because 1) it only impacted a limited number of visitors in the region; and 2) we were able to quickly and gracefully fail the data center out and traffic automatically shifted to the next closest data center. The logic to make this graceful failover didn't need to be constructed by our customers' programmers at the application layer because the Front End layer doesn't need the same synchronization as the Data & Application layer.

We've also worked to make sure that our Front End layer continues to serve static content when one of our customers' Data & Application layer goes down. One of the ways we first get word when AWS or another major host is struggling is when our customers write to us letting us know that they'd be entirely offline if not for our Always Online™ feature. We've got some big improvements to Always Online coming out over the next few weeks which will make the feature even better.

Going forward, we will continue to make scaling web application easier by providing services like intelligent load balancing between various Data & Application service providers both to maximize performance and also to ensure availability. Moreover, since every CloudFlare customer gets the benefit of all our data centers, as we continue to build out our network it inherently becomes more resilient to failure. Over the next month, we'll be turning on 9 new data center locations to further expand our network. While I expect the decision of where to host your Data & Application layer will remain vexing, we're working to make using CloudFlare as your Front End layer a no-brainer.

How to make SSL fast

Matthew Prince — Fri, 29 Jun 2012 04:15:00 -0700

HTTP, the protocol of the web, is unencrypted by default. That means it is trivial for someone using the same local network as you to spy on all the data you send to and receive from most websites. If, for example, someone sniffs your session cookie for a website then they can install it themselves and log in as if they were you without knowing your password. One high profile example was when actor Ashton Kutcher's Twitter session cookie was sniffed off the TED conference's wifi network, allowing a hacker to take control of his Twitter account.

The solution to problems like Ashton's is to use an encrypted HTTPS connection which requires a website to support SSL. The problem is that the SSL protocol imposes a heavy burden. I'm fairly technically savvy and still find the process of installing and maintaining SSL keys on a web server Byzantine. SSL sessions also add considerable CPU load to web servers and slow down web performance.

CloudFlare is the web performance and security company. For security, we believe it is incumbent on us to make SSL easier and more widely deployed. Given that, in order to maximize performance, we are spending significant resources in order to improve the performance of SSL.

What Makes SSL Slow?

SSL imposes an additional tax on web performance, but it's important to understand exactly how. The primary source of the SSL performance tax comes from the initial setup of a new connection. Before any web page data can be exchanged, a SSL session must be "negotiated" between the client and the server. A simplified version of this process is as follows.

If you have multiple resources on your web page under different domains, your visitors' browsers will need to negotiate an SSL session for each domain. Because each SSL session requires at least 4 additional exchanges, the problems of flakey connections with high loss rates (e.g., connections to mobile devices) are amplified over HTTPS.

OCSP & CRL: Hidden Vampires

SSL certificates are issued by a certificate authority (CA). The CA attests that the website behind the certificate is who they say they are. Sometimes, after a certificate is issued, it is lost or compromised and needs to be revoked. CAs maintain a list of certificates that have been revoked, known as a Certificate Revocation List (CRL). Browsers, when they access a SSL-protected site, can query to download a CA's CRL and parse the list to see if a particular site is on it. Alternatively, they can issue a request via the Online Certificate Status Protocol (OCSP) to check if a particular site has had its certificate revoked.

The problem is, for most CAs CRL and OCSP performance appears to be an afterthought. The CRL and OCSP servers are also not typically geographically distributed or tuned for performance. Verisign, one of the largest CAs, averages nearly 300ms to respond to a OCSP request. That inherently means an additional third of a second gets added to your page load time for any visitor connecting to your site if you're using Verisign's SSL. To add insult to injury, most CAs don't yet accept IPv6 requests to their CRL or OCSP servers.

The problem of CRL and OCSP slowness is a significant enough impact on SSL performance that browser vendors have begun to discuss removing the verifications entirely. While that may help performance, it would hurt overall web security and make it harder to reliably trust a site's SSL.

More SSL Sluggishness

A number of other factors further contribute to SSL's overall slowness. Often certificates are chained using a number of intermediate certificates, which can increase the amount of data that needs to be exchanged during the initial session negotiation. Most sites haven't taken the time to optimize their crypto cypher. Choosing a crypto cypher that is too weak can subject your site to a number of potential vulnerabilities. On the other hand, choosing a stronger cypher can add load if your site is already CPU bound.

Overall, we regularly hear from customers who want to have SSL protection on their sites but find it either too difficult or to resource intensive. The end result is that many sites that should use SSL don't, sacrificing security for performance. And, for those sites that do support SSL, it is often implemented in a way that causes a significant performance penalty. Neither case is good for the web, so we wanted to see what we could do to help fix the problem.

Speeding Up SSL

CloudFlare is working on a number of initiatives to speed up SSL. First, we have a number of techniques to limit the number of connections that a browser needs to make and therefore reduce the number of session initiation handshakes that need be made. Rocket Loader, for example, combines requests for scripts, even across third party services, into a single connection under a site's own domain. We're also rolling out support for SPDY for all SSL-enabled sites, which allows requests to be multiplexed across a single connection.

Beyond reducing the number of connections to improve SSL performance, we're working with Globalsign, our primary CA, to speed up CRL and OCSP. Today, Globalsign's CRL and OCSP requests are powered through CloudFlare's global network and response times are under 100ms. We are also planning on rolling out OCSP stapling, which allows the query for whether a certificate is invalid to be sent from the server without making an additional request to the CA's infrastructure.

CloudFlare runs all its own infrastructure, which allowed us to spec the hardware for maximum SSL performance. We worked with Intel to integrate a custom tuned version of OpenSSL which takes advantage of special instruction sets on our servers' CPUs to make encryption and decryption as fast as possible. In our tests, it is about 30% faster and uses fewer CPU resources. This allows us to choose stronger cyphers that maximize performance without suffering a performance penalty.

Expanding SSL Adoption

Equally important, we've worked to make SSL as easy as possible, regardless of the capabilities of your own infrastructure. SSL is included automatically for every paid CloudFlare plan and is provisioned automatically with a single click. We allow customers to choose Full SSL, which ensures end-to-end encryption, or Flexible SSL, which encrypts the connection from the browser to CloudFlare's network, the riskiest part of the transaction, but doesn't require you to run SSL on your origin server. This allows you to add SSL to services like Google AppEngine and Tumblr, which don't allow SSL support themselves or only include it as an expensive add on.

While CloudFlare's Pro accounts include SSL certificates issued by our own systems, with the launch of our Business and Enterprise tiers, customers can now upload their own certificates to use on our network. This allows support of extended validation (EV) certificates on our optimized infrastructure.

As new performance technologies like SPDY require SSL support, and as new threats continue to emerge online, ensuring SSL runs as fast and secure as possible becomes increasingly important. CloudFlare will continue to work to make SSL performance faster in order to fulfill our mission of building a web that is both safer and faster.

Why mobile performance is difficult

John Graham-Cumming — Thu, 28 Jun 2012 08:48:00 -0700

Mobile web browsing is very different, at the network level, to browsing on a desktop machine connected to the Internet. Yet both use the very same protocols, and although TCP was designed to perform well on the fixed-line Internet, it doesn't perform as well on mobile networks. This post looks at why and how CloudFlare is helping.

We start with a simple ping. Here's a ping from my laptop machine (which is connected via 802.11g WiFi to a 20Mbps broadband connection) to a machine at Google. Looks like I'm getting a roundtrip time of about 20ms.

Here's the same ping done from my iPhone on the same WiFi network at the same location in the house. The ping time has gone up to about 60ms. So, in this instance, the round trip time had tripled just from going from laptop to phone.

But to see the real cost of mobile it's necessary to switch off WiFi and onto 3G. Here's the ping time on 3G to the same machine. Here's it's both much higher (we're now into 1/10 to 1/5 of a second territory) but it's also variable:

And then I get up and move to the front of the house and try again. The ping time has changed completely (the number of bars didn't) and I'm seeing between 0.5s and 1s of round trip time. That will have a serious effect on web browsing.

And for a final test I return to my original location and grip the iPhone firmly in my hand. The number of bars falls away and the round trip time becomes infinite! Pings simply aren't working any more.

What this illustrates is something that any smartphone user knows instinctively: network performance on phone is very variable and susceptible to location and environment. TCP would actually work just fine on a phone except for one small detail: phones don't stay in one location. Because they move around (while using the Internet) the parameters of the network (such as the latency) between the phone and the web server are changing and TCP wasn't designed to detect the sort of change that's happening.

In past posts I've looked at the effect of high latency on web browsing and TCP's connection and slow start cost. One of the fundamental parts of the TCP specification covers congestion avoidance: the detection and avoidance of congestion on the Internet. At the start of a connection TCP's slow start prevents it from blasting out packets until it's detected the maximum possible speed it can transmit at, and during a connection TCP actively watches for signs of congestion. The smooth running of the Internet as a whole relies on protocols like TCP being able to detect congestion and slow down. If not there'd likely be a congestion collapse.

(Image credit: joiseyshowaa)

TCP spots congestion by watching for lost packets. On the wired Internet lost packets are a sign of congestion: they're a sign that a buffer in a router or server somewhere along the route packets are taking is full and is simply dropping packets. When lost packets are detected by TCP it slows down.

That all falls apart on mobile networks because packets get lost for other reasons: you move around your house while surfing a web page, or you're on the train, or you just block the signal some other way. When that happens it's not congestion, but TCP thinks it is, and reacts by slowing down the connection.

It seems like it might be a simple matter to change the congestion avoidance algorithm in TCP to take into account the challenges of mobile networks, but it's actually an area of active research with many different possible replacements for the existing basic algorithm. It's hard because trying to balance maximizing throughput, preventing congestion on the Internet, dealing with actual congestion, and spotting phony congestion is complex.

(Image credit: mikecogh)

And if that weren't enough mobile networks also introduce another tricky problem: packet reordering. Although TCP is designed to cope with reordering of packets (because they might have followed different routes between source and destination) large reordering can occur in mobile networks when a mobile phone is handed off from one tower to the next.

For example, a stream of packets being transmitted by a moving mobile user (perhaps sending a large email) might be split with some going down one route through one tower and the rest through a different tower and by a different route.

This causes problems for some of the newer congestion avoidance algorithms (such as TCP New Reno) and can cause additional slow downs.

CloudFlare helps solve these problems for our customers in two ways. Firstly, we customize the parameters inside the TCP stacks in our web servers to tune for the best possible performance and secondly we actively monitor and classify the connections from people surfing our customers' sites.

By classifying connections we are able to dynamically determine the best way to behave on a connection. We know whether this is likely to be a high-latency mobile phone browsing session, or a high-bandwidth broadband connection in someone's home or office. Doing that allows us to give the best performance to end users, and ensure that customers' web sites are snappy wherever and however they are accessed.

And we continually look at ways of improving network performance for our customers by tuning TCP, monitoring performance, opening new data centers and introducing features like Rocket Loader, Mirage, Polish, SPDY, and Railgun.

Dome9 + CloudFlare = Combined Security For Your Website and Web Server

Kristin Tarr — Wed, 27 Jun 2012 18:08:00 -0700

This is guest post from Roy Feintuch, Co-Founder & CTO of Dome9

Dome9 is a new kind of security management service that protects your cloud or hosted server firewall, including all the admin services for all your server's applications (e.g., phpMyAdmin). It's a great complement to the web security that CloudFlare provides.

Like CloudFlare, Dome9 is super simple. You simply create a free Dome9 accountand install a lightweight agent on your host machine. After that, Dome9 will secure your server's host firewall, or your EC2 security groups -- Dome9 can work agentless on EC2.

The magic of Dome9 is its ability to lock down all your administrative services so hackers can't brute force attack (or exploit a vulnerability of) SSH, RDP, MySQL and whatever else you've got running. Dome9 can open those services on demand (with just the click of a button) only for a specific user, service, and time period.

Here's an optimal server security policy you might set with Dome9:

Port 80 - Open *only* for CloudFlare's proxy servers
Port 443 - Open *only* for CloudFlare's proxy servers
Port 22 (SSH) - Closed, and opened only on demand
Port 8080 (phpMyAdmin) - Closed, and opened only on demand
All the rest - Closed

With this, your web server only communicates with CloudFlare and whomever you authorize.

How hard is it to set this up? It takes just a few minutes.

Activate your Dome9 account [2 clicks thanks to CloudFlare Apps]
Install the Dome9 Agent on your server (Linux and Windows are supported). [3 minutes]
In Dome9 Central, create a ‘Web Servers' security group with the above policy. [2 minutes]
Attach your server to the new security group. [2 clicks]

Setting a service to ‘On-Demand' is as easy as clicking a button.

Dome9 Magic IPs allow traffic only through the CloudFlare network

You're probably wondering how to allow only CloudFlare IPs? That's where Dome9's Magic IPs come into play.

Magic IPs are special IP address lists that Dome9 creates and maintains. We've created a special Magic IP for CloudFlare customers: {cloudflare}

With the {cloudflare} Magic IP you never have to worry about changes to CloudFlare's proxy IP addresses – Dome9 maintains them automatically.

Hence, your port 80 policy within Dome9 will look like this:

With CloudFlare and Dome9 you can kick your cloud or hosted server's security up a notch with fun, easy to use, next generation security services.

Visit the Dome9 App page today to get started.

App: Trumpet Offers Simple, Consistent Notification to Site Visitors

John Roberts — Tue, 26 Jun 2012 12:03:00 -0700

This is a guest blog post from Martin Reistadbakk, developer of Trumpet.

Say hello to Trumpet.

Trumpet is a simple CloudFlare App that lets you place a short message on top of every page on your site. You turn it on, and when you want to tell your users something, you enter a message that will show up the next time the user loads your site. If the user dismisses the message by clicking on it, Trumpet will not show it again for 24 hours. It is that simple.

Trumpet was conceived to scratch a personal itch. My clients often wanted to notify users of planned maintenance and downtime. This was usually done with a news article that only showed up on the front page. I needed something that would show up on all pages, which led me to making the first version of Trumpet.

My design goal for Trumpet was to make it very reliable, small, lightweight and without any dependencies on JavaScript libraries. A lot of the inspiration and code for the initial version of Trumpet came from humane-js, an excellent library without any dependencies. This was important for me. I didn't want sites to have to include large JavaScript libraries just to be able to show a small message, especially when the overall codebase of Trumpet was so small. I didn't want convenience for me to make the product worse.

Trumpet was originally made as a Google app engine project where the site that wanted to display the message included a JavaScript. This JavaScript then checked with the server if there was a message to be displayed. This had to overcome the usual cross site execution restrictions.

Then late last year I contacted CloudFlare about making Trumpet available on their platform. CloudFlare was very welcoming and helpful. Using their platform also meant that I could eliminate the Google app engine server, as CloudFlare can let users enter the message in the app configuration screen and embed this message with the JavaScript. This makes for a much faster and more reliable experience for the user.

If you use CloudFlare today, please check it out. If you don't use CloudFlare, what are you waiting for?

Turn on Trumpet whenever you need to make sure your visitors receive a notification.

About The Author

Martin Reistadbakk spends his days building and supporting websites built on Plone, a leading open source CMS, in a small town outside Oslo Norway. His company Blåstolen Web Solutions maintains and supports the websites and intranets of some of the leading brands in Scandinavia.

Develop Your Own App

If you'd like to develop your own CloudFlare App, check out the git repo of a sample app and then sign up and get started. If you're looking for ideas, we have several customer suggestions, so just ask.

What makes SPDY speedy?

John Graham-Cumming — Mon, 25 Jun 2012 09:14:00 -0700

(Image credit: Loco_2)

Google has proposed a new protocol for downloading web pages called SPDY and CloudFlare will shortly be making it available in beta form. SPDY is designed to make web browsing faster without replacing HTTP. This blog post explains how it works and why it helps.

Current web browsing makes use of the HTTP protocol running over TCP. The TCP protocol underlies many other uses of the Internet (such as sending and receiving email) because it provides reliable delivery of data. HTTP is independent of TCP and provides a mechanism for a web browser to ask for pages, graphics and other files needed to display a web page.

SPDY sits between HTTP and TCP to speed up the HTTP protocol by changing how it interacts with TCP. First, let's look at getting a web page with HTTP over TCP without SPDY and then see how that changes with SPDY.

A typical web browsing session goes something like this: you type cloudflare.com into your browser and it sends an HTTP request over TCP to the cloudflare.com server asking for the the HTML of the page. The page is delivered and the browser sets about parsing the page to determine what else it needs to download (e.g. the style sheets and images that make up the page).

Here's a screen shot from Firebug running in Mozilla Firefox. You can see the page being downloaded at the top and then the parts of the page (such as JavaScript, CSS and images) being downloaded in parallel.

But what's not clear from that view is how Firefox is actually connecting to the web server and retrieving the parts of the page. For that it's necessary to dig a little deeper. Using a combination of Wireshark and a custom program written in Processing here's a view of the TCP connections and downloading of each part of the CloudFlare home page.

The scale at the top shows elapsed time in seconds. Down the left hand side is the identifier of each TCP connection; each row in the diagram is a single TCP connection. On the right hand side is a number indicating how many separate parts of the page (images, CSS, JavaScript) were downloaded by the connection on that row.

The colors just indicate different parts of the page being downloaded. The size of the bars equates to the total time to get that part of the page.

The first connection begins by downloading the HTML of the page and then straight after that Firefox reuses the connection and opens another 6 connections to retrieve parts of the page. By opening multiple connections Firefox gets to download in parallel, by reusing a connection Firefox saves time starting a connection.

After the first set of connections there are another set used to download parts of the page. Some of these connections are reused multiple times.

It's likely pretty obvious why Firefox uses multiple simultaneous connections (because downloading can happen in parallel), but its reuse of connections is a little less obvious. Connections are reused because of two costs: connection set up time and TCP slow start.

First, it takes time to set up a TCP connection. The browser connects to the server and goes through a handshake to establish the connection. In the example above each connection was taking roughly 50ms to establish. If you did that for all 36 items being downloaded it would add up to 1.8s (longer than the entire download took). So, clearly reusing a connection helps.

(Image credit: Charles & Clint)

But TCP slow start also matters. In a previous post I looked at the effect of bandwidth and latency on downloading. What that post didn't mention is that the theoretical download speed is only reached after a period of slowness at the beginning of a TCP connection.

To avoid causing congestion on the Internet, every TCP connection starts slowly and works up to the maximum speed available. This is called TCP Slow Start. So, each time a new TCP connection is established there's a double penalty: connection set up time and slow start. TCP slow start is also problematic on high latency links because detecting the maximum speed available on the connection requires back-and-forth packet exchanges.

Thus, to make efficient use of TCP the ideal browser would open a small number of connections and reuse them. That way the connection cost would be low, TCP slow start would be minimized and download speed would be maximized.

So, why did Firefox open 20 separate connections and only reuse them for a small number of requests each? Take a look at the line corresponding to connection 47108 in the diagram. Notice how a small download (in blue) had to wait behind a large download (in red).

Since the web browser can't predict how large or small the response to each request will be it is not able to order requests for efficient delivery. So, there's a balancing act to find the right number of connections to minimize page load time as bandwidth, latency, connection time and slow start have to be taken into account.

Using a small number of connections would result in blocking (a long slow download of say a large image could block a small quicker download because there's no 'overtaking' in HTTP); a large number of connections would avoid blocking and pay the price of connection set up and slow start.

Also, opening a large number of connections would place a load on the web server as each connection takes up resources on the server itself.

These problems come about in part because HTTP is synchronous: a request is made for part of a page and the connection it was made on needs to wait for the response. A better protocol would allow multiple requests to be sent and the responses returned in the order they are available. Line 47108 would look very different in that case: all three requests could be sent and the small one would be returned first even though it was the second request.

(Image credit: Capt' Gorgeous)

SPDY fixes all these problems in one go.

SPDY allows a single connection to be used for multiple requests. The requests can be sent in any order and responses come back in the order that they are available. Since a single connection is used the connection set up cost and slow start are minimized. Since requests can be answered in any order there's no blocking.

Of course, SPDY also does other things (such as compression of previously uncompressed parts of HTTP), but its core benefit is that it decouples HTTP from TCP and in doing so allows asynchronous, overlapping HTTP requests on a single connection. All without changing HTTP at all.

And shortly CloudFlare will be rolling out SPDY to our customers. Information about the beta is here.

The bandwidth of a Boeing 747 and its impact on web browsing

John Graham-Cumming — Thu, 21 Jun 2012 08:29:00 -0700

(Image credit: markyharky)

This is the first in a series of posts looking at what makes the web slow and what to do about it.

Suppose you needed to transfer 1TB of data (perhaps your home movie collection) from San Francisco to London. What would be the fastest route? Put the disk on British Airways flight 286 at SFO, or transfer it across the Internet using a 100 Mbps connection?

Surprisingly, the answer is the former not the latter. If you had a perfect 100 Mbps Internet connection and could fill it completely with data the transfer would take 22 hours 13 minutes. British Airways make the flight in under 10 hours.

But even with a 100 Mbps Internet connection you're unlikely to get 100 Mbps of transfer speed between San Francisco and London. The details of the TCP protocol used on the Internet and the speed of light collude to make the effective transfer speed much lower.

To really understand the speed of an Internet connection, be it transferring 1TB of data or downloading a web page, there are two values that you need to know: the bandwidth and the latency.

The bandwidth is how much data can be sent on the connection in a unit of time. In the example above the Internet connection has a bandwidth of 100 Mbps, the Boeing 747 has a bandwidth of 222 Mbps (the 1TB carried divided by the flight time).

The latency is the 'flight time' of data across the connection. For a connection between London and San Francisco across the Internet the latency will be something like 150ms. That figure is governed and limited by the speed of light. For the 747 the latency is the literal flying time of 10 hours.

One thing British Airways ensures while flying the 1TB of data is reliability. The data is very, very unlikely to not arrive in London. The Internet does not provide the same guarantee. As data is transferred across the Internet it gets delayed, lost, corrupted and misordered. So, the Internet's core protocol TCP provides mechanisms to ensure the reliable delivery of data despite the lossy network the data is passing through. It's these mechanisms that slow the transfer of data down and where the speed of light comes into play.

(If airlines experienced aircraft loss at the rate the Internet sees packet loss there'd be 28 crashes per day in the US alone).

(Image credit: El Ronzo)

To ensure reliability, TCP breaks data to be sent up into chunks (which are further broken down into packets) and sends chunks of data and then waits for an acknowledgement that the chunk was successfully received. It's while waiting for the acknowledgement that the speed of light comes into play.

Imagine that a 65kB chunk of data has been sent across a link with a latency of 150ms. The 65kB take 150ms to reach their destination and the receiving machine sends an acknowledgement that takes a further 150ms to arrive. So 0.3s is taken up making sure that 65KB have arrived successfully; that number is called the Round Trip Time.

Those acknowledgement delays significantly hamper connections across long distances (and also from mobile phones).

The amount of data that TCP can send in a single chunk is controlled by the Receive Window of the receiver machine. For web surfers that means that the receiving machine controls how much can be sent without acknowledgement. And the combination of Receive Window and Round Trip Time limit the speed at which downloads can occur no matter what the bandwidth is.

The maximum throughput of TCP is Receive Window divided by Round Trip Time.

For example, on my machine the Receive Window is set at 524,288 bytes meaning that on a slow link from London to San Francisco the fastest download I can get is 524,288 bytes / 0.3s or 14 Mbps. Much less than the 100 Mbps I was hoping to get.

So, my 1TB download would actually take more than 6 days! The speed of light really is a limiting factor in downloading.

How do you fight the speed of light? Since you can't control the Receive Window the only thing you can do is move your web site closer to the people surfing it. And, of course, that's not practical for most web sites since you'd have to have copies of the web site all over the world.

CloudFlare fights the speed of light for you by having data centers around the world. If your site is on CloudFlare then surfers will connect to the data center that's nearest to them.

For example, CloudFlare's own web site is in California, but from London it appears to be only 10ms away because of CloudFlare's London data center. The same distribution of a web site across the world works for any CloudFlare customer.

In my next post I'll look at optimizations you can make to web site content for speedy browsing, and show how CloudFlare helps.

Part two of this series is now available: What Makes SPDY Speedy?

A note about Kerckhoff's Principle

John Graham-Cumming — Tue, 19 Jun 2012 17:56:00 -0700

(Image credit: psicologiaclinica)

The other day I wrote a long post describing in detail how we used to and how we now store customer passwords. Some people were surprised that we were open about this, and others wanted to understand exactly what part of a security system needs to be kept secret.

The simple answer is: a security system is only secure if its details can be safely shared with the world. This is known as Kerckhoff's Principle.

The principle is sometimes stated as "a cryptosystem should be secure even if everything about the system, except the key, is public knowledge" or using Claude Shannon's simpler version: "the enemy knows the system".

The idea is that if any part of a cryptosystem (except the individual secret key) has to be kept secret then the cryptosystem is not secure. That's because if the simple act of disclosing some detail of the system were to make it suddenly insecure then you've got a problem on your hands.

You've somehow got to keep that detail secret and for that you'll need a cryptosystem! Given that the whole point of the cryptosystem was to keep secrets, it's useless if it needs some other system to keep itself secret.

So, the gold standard for any secret keeping system is that all its details should be able to be made public without compromising the security of the system. The security relies on the system itself, not the secrecy of the system. (And as a corollary if anyone tells you've they've got some supersecret encryption system they can't tell you about then it's likely rubbish).

A great example of this is the breaking of the Nazi German Enigma cipher during the Second World War. By stealing machines, receiving information from other secret services, and reading the manuals, the Allies knew everything there was to know about how the Enigma machine worked.

Engima's security relied not on its secrecy, but on its complexity (and on keeping the daily key a secret). Engima was broken by attacking the mathematics behind its encryption and building special machines to exploit mathematical flaws in the encryption.

That's just as true today. The security of HTTPS, SSL and ciphers like AES or RSA rely on the complexity of the algorithm, not on keeping them secret. In fact, they are all published, detailed standards. The only secret is the key that's chosen when you connect to a secure web site (that's done automatically and randomly by your browser and the server) or when you encrypt a document using a program like GPG.

Another example is home security. Imagine if you bought a lock that stated that it must be hidden from view so that no one knew what type of lock you had installed. That wouldn't provide much reassurance that the lock was any good. The security of the lock should depend on its mechanism and you keeping the key safe not on you keeping the lock a secret!

(Image credit: paul.orear)

When storing passwords securely we rely on the complexity of the bcrypt algorithm. Everything about our storage mechanism is assumed to be something that can be made public. So it's safe to say that we choose a random salt, and that we use bcrypt. The salt is not a key, and it does not need to be kept secret.

But even more we assume that in the horrible case that our password database were accessed it will still be secure even though the hashed passwords and salts would be available to a hacker. The security of the system relies on the security of bcrypt and nothing else.

Of course, as a practical matter we don't leave the database lying around for anyone to access. It's kept behind firewalls and securely stored. But when thinking about security it's important to think about the worst case situation, a full disclosure of the secured information, and rely on the algorithm's strength and nothing else.

Keeping passwords safe by staying up to date

John Graham-Cumming — Mon, 18 Jun 2012 00:08:00 -0700

(Image credit: jantik)

Over the last few weeks a number of companies have seen their password databases leaked onto the web and found that despite having made some effort to protect them many of the passwords were easily uncovered. Unfortunately, the disclosure of password databases is an ugly reality of the Internet; entire forums are dedicated to hackers who collaborate to uncover passwords from files and specialized password cracking software is easy to obtain.

To understand password storage it's best to go back to basics and some history.

Plain

The simplest way to store a password is just to store it in a database. When a customer tries to log in and types in the password 'supersecret' that string is compared with the password in the database and the customer is or is not allowed in.

Of course, storing passwords in the clear (or in plain text) is very dangerous. If the database is compromised then the passwords can be read and every account can be broken into. Despite this danger there are many companies that store passwords in plain text. Some attempt to encrypt the password and then decrypt it when you log in. Although that's slightly better than a plain text password in the database, it only adds a small hurdle for a hacker: they just have to take the database and the encryption key and since the key is almost certainly on the same machine as the database it becomes trivial to do.

Despite the poor security offered by encrypted or plain text passwords, many companies still use them. One sure fire way to find out whether a site you are using does this is to ask for a password reset: if the company is able to email you your old password then it was stored insecurely.

Hashed

If you're following along and are new to password security you may be asking yourself: how can you test someone's password when they want to log in if you don't store it in some way? It does seem like an unsovable conundrum until you discover the cryptographic hash function (which I'll just shorten to hash function).

A hash function takes some string (such as a password) and turns it into a long number. In doing so it ensures two things: it's not possible to do the reverse (you can't take the number and run the algorithm backwards to get the string) and the number it generates is unique (i.e. there are no two strings that have the same number).

(Aside: I've simplified things a little in the previous paragraph. "not possible" should really be "infeasible" (i.e. you'd need to have more computers than there are on the planet to find the string) and "unique" should be "vanishingly improbable that two different strings will have the same number").

Hash functions work by taking the string to be hashed and scrambling the bits over and again to produce a number. One popular hash function is SHA-1. The SHA-1 hash of the password 'supersecret' is a761ce3a45d97e41840a788495e85a70d1bb3815 (the numbers are so long that they are typically written like this in hexadecimal instead of decimal. In decimal that number is 955,582,595,971,963,915,918,670,633,711,507,401,334,868,097,045). The SHA-1 hash of 'Supersecret' (note that capital S) is 1b417472fc8e2a0a4d44ed43f874309ca4069099 (as you can see it's totally different).

Hash functions are used for many purposes such as checking that the contents of a file haven't changed. When you download a file from the Internet its hash might also be sent so that your computer can check that no bits in the file have been accidentally flipped in transmission.

Hash functions are also often used in password systems because instead of storing the password, you can simple store the hash. Since the hash can't be easily reversed the stored hash is a secure way of keeping the password. When a visitor comes to the site the hash of the password they entered is calculated and compared with the hash in the database. Since the hashes are unique they'll only be able to log in with the right password.

(Image credit: ToGa Wanderings)

Unfortunately, simply using a hash function like this is dangerous. Over the last few weeks a number of prominent Internet companies have found that their password databases have been cracked even though they 'hashed' their passwords. To see why, try Googling a761ce3a45d97e41840a788495e85a70d1bb3815. You might be surprised to find that the first result tells you that that's the SHA-1 hash of 'supersecret'.

The problem with simple hash functions is that hackers simply get a dictionary and compute all the hashes of all the possible passwords made from the dictionary. These massive databases of precomputed hashes are called rainbow tables. If a password database leaks then the hackers just look up the hashes in the rainbow table. The hashes that aren't found in the rainbow table correspond to those users who created long, complex passwords that weren't precomputed in this way. That's one reason why picking a long, complex password matters: hackers won't have already computed its hash.

Even though the hash function itself couldn't be reversed, it was possible to create a table of precomputed password hashes (especially for poorly chosen passwords).

Salted

The way around rainbow tables is with something called salt. Let's suppose you've picked the password 'supersecret' and company X is going to use SHA-1 to hash the password. Instead of simply hashing the password, company X picks a random salt (a random string of characters) that's unique to you (such as '$f2%38h##f23'). Instead of computing SHA-1(supersecret) they compute SHA-1(supersecret$f2%38h##f23) and get 33438b91ce09e695923 2f698b7939e6ee1d0712a. Try Googling that and you won't get any results.

(Image credit: stlbites)

Since each user has some random salt applied to the hash, rainbow tables are useless. It's not possible to precompute the hashes of all the possible passwords with all the possible salt values.

Until recently a 'salted hash' like this was how CloudFlare stored user passwords.

Unfortunately, password cracking techniques benefit enormously from two things: Moore's Law and the speed of hash functions. Hash functions weren't originally designed for protecting passwords, they were designed to check the integrity of data by detecting changes (notice how just changing from s to S in supersecret dramatically changed the SHA-1 hash above) and for that reason they were designed to be fast, very fast.

As computers have increased in speed with Moore's Law the speed of hash functions has made it possible to do away with rainbow tables and start attacking passwords directly even when salted. When a password database leaks, password cracking software is able to compute millions of passwords per second applying the unique salt to each password and checking the resulting hash value. The software literally tries out combinations of words and letters and computes the hash for each one.

That means that only long, complex passwords are safe with a salted hash.

(Image credit: 4nitsirk)

The solution is to use a hash function that's slow. If the hash function itself is slow then it slows down cracking software. If the speed can be chosen so that over time the hash function can be made slower, then the hash function can be slowed down so that password cracking doesn't get easier.

Future Proof

Happily, hash functions with just that property have been invented specifically to help keep passwords safe. We recently upgraded our entire password database to use bcrypt. bcrypt is just like a normal hash function but it has an additional parameter: as well as being fed the password and some random salt it's fed a cost. The cost tells the hash function how hard to work in computing the hash (and thus determines how long it will take).

Over time the cost can be increased (it's just a number) to keep pace with faster and faster computers and keep passwords safe by making the hash function slower and slower.

Just like all aspects of security, password storage needs to be reviewed from time to time. As we've seen recently many companies don't take the time to upgrade their password security leading to serious problems.

And, of course, users can help out too: password cracking relies partly on the algorithms used to store the passwords and partly on the complexity of the password. Make sure to choose a long, complex password and don't use it on any other site.

Introducing SPDY

Matthew Prince — Fri, 15 Jun 2012 11:43:00 -0700

We want to acknowledge and thank the amazing team of engineers at NGINX. They have been working on a SPDY implementation for quite some time and their work made it possible for us to roll out SPDY across our network. CloudFlare's core is built on top of the NGINX platform and we recommend it highly for anyone looking for a fast, secure, scalable web server. You can read more about their SPDY extension on their mailing list.

In 2009, Google began work on a new network protocol to make web pages faster. Dubbed SPDY (pronounced "speedy"), the protocol is designed to solve many of the bottlenecks that slow HTTP down. Beginning today, we're rolling out a beta of SPDY to CloudFlare customers. Read this post and then, if you're interested in participating in the beta, complete the beta application form.

How SPDY Makes Things Speedy

Standard HTTP needs to make a new TCP request for all the objects on the page, Because there is significant overhead for each new TCP connection, all these connections slow down performance significantly. SPDY's biggest win comes from what is known as connection multiplexing. This means that mutiple objects from a particular site are requested and retrieved from a single request. Less connection overhead means faster page loads.

HTTP also requests objects in a particular order and one slow resource can block the loading of other resources. SPDY allows the browser to query for multiple objects in one request and for the objects to be sent down the wire as they are ready and out of order. Again, this can increase performance by not holding up the delivery of objects that are available quickly because some take longer to request.

SPDY includes some other performance wins as well. It allows HTTP headers to be compressed, which isn't possible with standard HTTP connections. The compression algorithm uses a HTTP-aware dictionary, which means common strings that appear in headers don't need to be sent across the network. Every byte you don't need to send not only reduces bandwidth use but, more importantly, increases web performance.

SPDY Caveats

While there is a lot of excitement around SPDY, there are some caveats. The first is that only certain browsers support the protocol. Google's Chrome and the latest release of Firefox (version 11+) contain SPDY support. While the Internet Engineering Task Force (IETF) is considering SPDY as an official Internet protocol, it has not yet been adopted so it's not clear whether there will be additional support from browsers such as Microsoft's Internet Explorer and Apple's Safari.

SPDY is built on top of TLS, which means it requires a site to have a valid SSL certificate in order to work. This, unfortunately, limits SPDY only to CloudFlare's paid customers who have enabled Flexible or Full SSL support. Microsoft is working on revised IETF proposal that is SPDY-like, but removes the requirement for SSL/TLS. If the TLS requirement is removed in the future, we'll make SPDY (or whatever it comes to be called) available more broadly.

Very few sites currently support SPDY (Google's sites and Twitter being the most notable to support the protocol). As a result, there haven't been a significant number of real world case studies. A recent article by an Akamai researcher pointed out that for much of the web SPDY's performance wins will be limited. We've confirmed similar findings in our tests. The primary reason for this caveat is that most sites are a collection of objects from multiple sources. Since SPDY's multiplexing only works on a per-domain basis, if a site has objects pulled from multiple sources then even if all those sources support SPDY connections still won't be able to be multiplexed between them.

Finally, SPDY is complicated to setup for a lot of sites. Support on most web servers is nascent and unproven. Because of this, sites have been hesitant to setup SPDY support themselves.

How CloudFlare Makes SPDY Even Speedier

The good news is CloudFlare is making SPDY extremely easy and features like Rocket Loader remove some of the biggest SPDY caveats, making the protocol even speedier. For SPDY support, CloudFlare acts as a gateway. Similar to how CloudFlare's Automatic IPv6 Gateway works, an origin server doesn't need to support SPDY. Instead, visitors with browsers that support SPDY connect to CloudFlare over the protocol. We handle the multiplexing and begin sending down objects we already have in our cache. The request to the origin server for non-cached objects is sent over standard HTTP/S. As a result, CloudFlare customers can implement SPDY support with a single click.

CloudFlare's Rocket Loader also helps with some of the multiplexing limitations. Rocket Loader was built to provide multiplex-like support over a standard HTTP connection. By gathering all scripts, regardless of where they're hosted, into a single HTTP request, Rocket Loader limits the number of HTTP connections that are needed. This also means that even third party scripts that appear on your page are requested under your site's domain. As a result, if you enable SPDY and Rocket Loader together then you will be able to get the benefits of multiplexing even for many of the object that make up your site even if they are hosted outside of your domain.

Beta Rollout

CloudFlare is rolling out the SPDY beta to select customers over the coming weeks. To be eligible for the beta, you need to have SSL enabled which requires one of CloudFlare's paid plans. As mentioned above, this is a limitation of the protocol and if that limitation is removed in the future then we'll plan on rolling out SPDY support more broadly. If you're interested in trying it, complete the beta application form and we'll send you an invitation as space is available.

At CloudFlare, we're committed to making the web speedier in every way we can. We're excited to offer the first easy way for websites that want to support SPDY to be able to do so easily and in a way that will get the most out of the protocol.

Mobile Web Performance: Optimizing TCP Congestion Control Algorithms

Matthew Prince — Wed, 13 Jun 2012 00:38:00 -0700

Transmission Control Protocol (TCP) is one of the primary protocols of the Internet. When you request a web page, the server responds with the data that makes up that page. The data is subdivided into discrete packets of data as they are sent across the Internet from the server back to your browser.

The size of these packets and the number of packets in flight can be dynamic and adjusted based on what is known as the Congestion Control Algorithm. While the late United States Senator Ted Stevens got a lot of flack for saying it, the Internet really is akin to a series of tubes. Sometimes these tubes fill up (i.e., a network port fills to its maximum capacity). When that happens, the network is said to be congested and some packets can be lost. It's a lot like a virtual traffic jam.

Traditional Congestion Control

To minimize the impact of congestion, the TCP Congestion Control Algorithm on servers adjust the size of the packets and also the number of packets that are "in-flight" across the network (known as the Congestion Window) depending on whether loss is detected. Historically, loss was most common because some network link would be over saturated. The default algorithms in most OS kernels rapidly shrinks the size of the congestion window at the first sign of loss and then slowly ramp it back up as the loss on the network diminishes.

Mobile Changes Everything

The problem is that congestion on the web today doesn't always occur in the same way that it did when TCP algorithms were first conceived. Mobile, specifically, is radically changing congestion behavior. We've all experienced it. Walking from one side of a room to the other can cause a change in mobile data performance. While in the past congestion indicated an over-saturated network link, today congestion could be the result of something extremely temporary like someone turning on a microwave.

The problem is that the default response to congestion, which assumes a longer-term, latent problem, ends up often being the wrong response in a world where congestion can be random and sporadic. In other words, you need a modern Congestion Control Algorithm that is tuned to take into account the behavior of both traditional wire-line network transactions as well as new wireless connections.

Better Congestion Control From Real Network Data

CloudFlare sits in a unique position to monitor and understand the behavior of the world's networks. We receive traffic every day from virtually all the world's Internet service providers. For the last year, we have been mapping how networks handle TCP. We know, for example, that an Indian wireless provider experiences higher levels of packet loss at noon in Mumbai, when the sun is directly overhead and data usage is at its peak, than at midnight.

We've begun to use this data to tune our own TCP Congestion Control Algorithm to be responsive based on the characteristics of the networks we see. Today we rolled out an update that already appears to have improved performance on mobile networks that experience high loss rates significantly. Going forward, we'll be adjusting our tuning on a per-network basis to optimize performance for our customers' sites based on the actual characteristics of each network.

Core to CloudFlare's value proposition is that our network gets smarter as it grows larger. That is easy to understand with regard to security (i.e., if one site is attacked, information about that attack is used to protect other sites on the network). It turns out that same benefit also applies to performance. By understanding the characteristics of the network for hundreds of thousands of sites, we're able to continually tune our network to provide the best possible performance.

CloudFlare's mission is to build a better Internet, and that involves tuning all the way down to the underlying protocols that serve as its foundation. Stay tuned. A lot more along these lines is yet to come.

Introducing: CloudFlare Business and Enterprise

Kristin Tarr — Thu, 07 Jun 2012 02:38:00 -0700

Over the last 18 months, the CloudFlare team has been busy building massive scale across hundreds of thousands of websites. As our network continues to grow, it becomes smarter, giving us insight into website security and performance that no other company has.

Thanks to the knowledge gained from this massive scale, and to all of our customers who have joined the CloudFlare network, we are now able to provide a performance and security solution for businesses and enterprises that is unlike anything else on the market. We are very excited to announce our newest tiers of service: CloudFlare Business and CloudFlare Enterprise.

What the New Plans Offer

CloudFlare Business and CloudFlare Enterprise add higher levels of customer support, greater customization, and full service level agreements to CloudFlare's existing products. The business-oriented plans also include new technologies including CloudFlare's Advanced DDoS Protection and CloudFlare's Railgun™ Web Optimization.

What Some Business and Enterprise Customers are Saying

We have been quietly testing CloudFlare Business and CloudFlare Enterprise with a number of customers with sites ranging from ecommerce to high-traffic to Fortune 500 to national governments. Their feedback speaks for itself:

"CloudFlare offers the first effective execution of a holistic approach to both performance and security for web applications, in a technology landscape otherwise rife with solutions specializing in either one," said Chris Holland, Director of Technology for LuxuryLink. "Having helped us achieve improved performance while saving us 40 percent in monthly bandwidth, I feel that CloudFlare's philosophy, technological leadership and innovative platforms are best aligned with our commitment to providing a world-class online experience to our discerning audience."

"StumbleUpon chose CloudFlare Enterprise to boost the performance of the product in every channel including the website and mobile applications." said Berk D. Demir, Software Architect at StumbleUpon. "CloudFlare's combination of performance and value made our decision easy and their global network matches StumbleUpon's expanding user base. We are pleased to be one of the first CloudFlare Enterprise customers and excited for the ongoing innovations CloudFlare provides."

"StockTwits chose CloudFlare for security during an attack on StockTwits.com. We were back online within an hour," said Chris Corriveau, StockTwits CTO. "Since joining CloudFlare, we've seen performance benefits, too, and added CloudFlare to several more sites."

"Clicky Web Analytics chose CloudFlare Enterprise after an exhaustive search," said Sean Hammons, co-founder at Roxr Software, creators of Clicky Web Analytics. "We could not find any other CDN, security or performance company quite like CloudFlare. Their solution is easy to implement and CloudFlare's global network is exactly what we needed to serve our hundreds of thousands of global customers."

Experts Agree

We've been putting our new performance and security features through the paces and some of the leading experts in the field agree that they're game changers.

"I test a lot of denial of service attack tools, including the ones used by Anonymous, but it has become a dull exercise because CloudFlare stops them all," explained Sam Bowne, Professor of Cybersecurity at the City College of San Francisco and a leading researcher in denial of service attack mitigation. "CloudFlare has completely changed the game, making DoS and DDoS into trivial, solved problems."

Continuing to Invest in Free and Pro Tiers

Rest assured if you are a current CloudFlare user who loves our existing service, things are only going to get better. This week, we've also added new image optimization features to our Pro service (Mirage and Polish). As CloudAve tweeted this morning, these two web performance features alone make upgrading to CloudFlare Pro a "no brainer."

We will continue to invest in our existing Free and Pro plans. We are 100% committed to always provide a free service with at least the feature set that it has today. Over time, some of the features that are part of paid plans will migrate down into the free service. All our customers, at any level, will benefit from our ongoing network expansion (and we'll have big things to announce on that front shortly) as we continue to make the Internet a faster, safer, better place for everyone.

If you're interested in our new tiers of service, visit our CloudFlare Business & Enterprise page or signup on our new plans page which contains full details comparing CloudFlare different service options.

Introducing Mirage: Automatic Responsive Web Design via Intelligent Image Loading

Matthew Prince — Wed, 06 Jun 2012 04:08:00 -0700

Yesterday, we announced Polish, which helps to automatically optimize the images on your site to decrease their size and thereby increase performance. Today we're releasing something more ambitious: a system to automatically manage the loading of images in order to maximize your site's performance which we call Mirage.

Impact of Images

Images are more than 50% of the data that makes up a typical website. A tool like CloudFlare's Polish substantially reduces the size of images. What would be even better than reducing image sizes would be not loading image data that isn't needed for the page. That's what Mirage does automatically.

To understand this, imagine a typical blog. A blog is usually a long page with a series of stories, the most recent of which are on top. Your browser window is a certain size, which is known as the viewport. When you load a page, you can only see the images within that window. However, most browsers happily download all the images on the page before the page is ready. This not only slows down page performance, but if you never scroll down the page then it also wastes bandwidth downloading images that will never be seen.

Responsive Design: Lazy Loading and Auto Resizing

CloudFlare Mirage is aware of the images on your page, the size of a visitor's viewport, and the type of device and network connection their device has. It then automatically optimizes image loading to only deliver the images that are necessary. Mirage prioritizes the loading of the images that are in the viewport. It then loads the other images as they are needed or as there are spare network resources available. You can see Mirage in action, loading images as a visitor scrolls down the page, in this video.

In addition to lazy loading, Mirage can also automatically optimize images to the size and resolution that is best for the page and device. One of the biggest wastes on the web is that people will upload a large image and then resize it down to a thumbnail or smaller size within HTML. Mirage detects cases like this and instructs the CloudFlare CDN to do the image resizing at the server. This means that if have a 1000px x 1000px image that you're only displaying at 100px x 100px then we deliver it to your visitors at the appropriate size and without wasting unnecessary bandwidth.

Mobile, Mobile, Mobile, Mobile, Mobile

Mirage significantly improves mobile performance in multiple ways. It automatically detects if a visitor is connecting over a mobile operator's network (where bandwidth is limited) or over a wifi conncetion (where bandwidth is less of a concern) and adjusts its image loading algorithm appropriately. So, for example, it will only load images as a visitor scrolls them into the viewport if you're on a 3G connection, where it will lazy load the all images in the background, prioritized based on where they appear on the page, if you're on wifi.

Mirage's dynamic sizing also takes into account the size of the viewport on a mobile device. A background image that is 2,000 pixels wide is wasted on an iPhone visitor's screen that is only 960-by-640-pixels. Mirage can automatically downsize the image so that you're never delivering more pixels than the device in question can display. Less data delivered to a mobile device not only improves a site's performance, but also helps your visitors with limited data plans not saturate their caps.

Automatic Responsive Web Design

Responsive web design is the principle that your site's layout should respond to the environment. Usually this requires significant code changes to your underlying site, which many existing sites have difficulty retrofitting into their current stack. CloudFlare's Mirage automatically enables some of the biggest wins from responsive web design without you having to change a single line of HTML.

You can pick the particular Mirage features you want to enable and they will be applied across your site. Mirage works via Javascript, but has a safe fallback for visitors with Javascript disabled. Since Mirage adds additional resource load to our network in terms of additional storage and CPU usage, we're limiting the feature to paid customers. You can turn on Mirage from the Performance tab of your CloudFlare Settings. Together, Polish and Mirage make a one-two punch that ensures your site is handling images in the best possible way for each visitor to your site.

Introducing Polish: Automatic Image Optimization

Matthew Prince — Tue, 05 Jun 2012 01:08:00 -0700

Today, the average web page has more than 85 objects (images, Javascript, CSS, etc.) that make up more than 750 KB of data. All that data needs to be downloaded when a page loads. On the average web page, more than 50% of the data is made up of images. And images are only getting larger as screen resolutions on devices improve.

A big part of making the web fast is reducing the amount of that data that makes up each page. If you can cut a page's size in half it's effectively as good as making a web connection twice as fast. This is especially important for mobile devices that have limited bandwidth. At CloudFlare, we've just rolled out two new features that help improve image performance: Polish & Mirage. This post is about Polish, which is the simpler of the two features. Tomorrow we'll send out another update about Mirage.

Introducing Polish: One-Click Simple Image Optimization

Polish automatically optimizes the images on your site. When an image is fetched from your origin our systems automatically optimize it in our cache. Subsequent requests for the same image will get the smaller, faster, optimized version of the image. We've been quietly testing Polish for the last few months and the results are impressive. On average, the data transfer saving, and therefore performance gains, from Polish far exceed more common techniques like code minification.

How Much Do You Want To Save?

You can enable two modes: Lossless or Lossy. The Lossless mode removes all the unnecessary bloat from an image file, such as the image header and meta data, without removing any image data. This means images will appear exactly the same as they would have before. In our tests, we're seeing an average file size reduction from the Lossless mode of 21% across actual CloudFlare customer images.

Lossy mode also removes the unnecessary bloat from an image file, but also applies a compression algorithm to suitable images. We picked the compression algorithm we're using to minimize any perceptible visual difference first and then attempt to get the most data savings. On average, using Lossy mode we've seen an average file reduction of 48% across actual CloudFlare customer images.

I'm one of those people who worries about things like losing image quality, so the engineering team gave me a challenge where they'd show me two images: one that went through the Polish Lossy algorithm and one that did not. While I'm sure that there are some people in the world who can spot the difference, for the vast majority of images I have to confess that I couldn't. For most sites, Lossy mode will be a big win but you can experiment between the two and see what works best for your site.

Image Optimization for the Win

Reducing image sizes has a particularly large impact on wireless devices that have limited bandwidth. If a large number of your visitors are accessing your site on a mobile device, Polish will be a big performance win.

Because the compressing of images adds CPU overhead, Polish is only available for paid accounts. If you already have a paid account, you can find it in your CloudFlare Settings under the Performance tab. (For those beta testers who helped us build the feature over the last few months, you'll continue to enjoy Polish even if you don't have a paid account — thanks for all your help!)

Watch this space tomorrow for another new feature called Mirage which helps with image optimization in a revolutionary new way. Stay tuned!

The Four Critical Security Flaws that Resulted in Last Friday's Hack

Matthew Prince — Tue, 05 Jun 2012 00:02:00 -0700

A core value CloudFlare is that security information should be shared between organizations to make the entire Internet safer. That is how CloudFlare's systems work: if one site is attacked, data about that attack is immediately shared with the rest of the network so other sites can be safe. We believe that same core value should apply when we are the victim of the attack. That is why we immediately posted an incident report and have continue to update it as we learn more.

Writing that report wasn't fun, but I believe it is important to share the details of the event so others who may be affected can learn from the events that transpired last Friday. This is not the usual way for the security industry, but we believe it's the way the security industry should be. To that end, here's what we know about the hack.

The Four Key Security Flaws

There were four key security flaws that allowed the hack to happen:

AT&T was tricked into redirecting my voicemail to a fraudulent voicemail box;
Google's account recovery process was tricked by the fraudulent voicemail box and left an account recovery PIN code that allowed my personal Gmail account to be reset;
A flaw in Google's Enterprise Apps account recovery process allowed the hacker to bypass two-factor authentication on my CloudFlare.com address; and
CloudFlare BCCing transactional emails to some administrative accounts allowed the hacker to reset the password of a customer once the hacker had gained access to the administrative email account.

Patching the Holes

We are following up with AT&T to understand more about how the voicemail was redirected. That remains unsettling, but it is not surprising that a phone company's voicemail security procedures are lax. It is also unsettling that Gmail's account recovery process appears to still be vulnerable to the voicemail hack. That is troubling since it means if a hacker knows your phone number then your Gmail account may, at best, only be as secure as your voicemail PIN.

You can mitigate these risk if you are a user by enabling two-factor authentication, ideally relying on Google's Authenticator App rather than anything that passes through the phone company's network. While Google is advising otherwise, I have removed my phone number from all my Google accounts.

Google has publicly stated that the flaw in the Google Enterprise App account recovery process has been patched and you can no longer use it get around two-factor authentication. Again, since any security system is only as strong as its weakest link, we would recommend using an out-of-band authentication that doesn't rely on the phone company's network (e.g., Google Authenticator App, not SMS or voice verification).

Finally, CloudFlare has stopped BCCing password reset and other transactional messages to administrative accounts, closing that attack vector if an administrator's email account is compromised in the future. If you're doing that at your company, and a troubling number of companies do use email as a poor man's logs, you should stop. This incident is why.

Timeline

The event, from start to finish, lasted less than 2 hours. The hackers were in my personal Gmail account for about 1 hour 35 minutes. They were in CloudFlare's email accounts for about 28 minutes, although likely interrupted several times as our ops team reset passwords and sessions. To better understand the hack, we put together the visual timeline (full size image here) below which is our best understanding of the events as they transpired. As we learn more, we'll update the information here and on the official incident report.

Taming BEAST: Faster, Safer SSL now on CloudFlare

Matthew Prince — Sat, 02 Jun 2012 01:55:00 -0700

For some time, the vast majority of the web has been vulnerable to the so-called BEAST SSL attack. The attack was first demonstrated in 2011, and more than 90% of the Internet including large sites like Google.com remain vulnerable to their SSL sessions being decrypted. But, as of today, anyone with CloudFlare account with SSL need no longer fear the BEAST.

The solution to BEAST is implementing TLS 1.1 and 1.2 as well as prioritizing the RC4 cypher suites. We just rolled out support for these across our network. Along with this, we've looked over the SSL cyphers we support. We've added some additional stronger cyphers, removed some of the weakest cyphers, and prioritized them to optimize both security and SSL performance.

Faster & More Secure

This is one of those great updates where we can report that security has improved and performance is also faster. In fact, this update makes our overall SSL performance about 30% faster. Sites behind CloudFlare now receive a 90 (A) score from SSLLabs. That's higher than the scores achieved by sites like Google.com and Facebook.com, and the highest score we think we can get without sacrificing performance. If you have SSL enabled, you can test yours yourself.

All paid CloudFlare plans include SSL by default. If you don't yet have SSL, upgrade to a paid plan today to make sure all the visitors to your site can surf on a secure, encrypted connection. It's now not only the easiest SSL on the web, but also among the fastest and the safest.

Post Mortem: Today's Attack; Apparent Google Apps/Gmail Vulnerability; and How to Protect Yourself

Matthew Prince — Sat, 02 Jun 2012 00:22:00 -0700

This morning a hacker was able to access a customer's account on CloudFlare and change that customer's DNS records. The attack was the result a compromise of Google's account security procedures that allowed the hacker to eventually access to my CloudFlare.com email addresses, which runs on Google Apps. While we are still working with Google to investigate the details, we wanted to highlight it here to make people aware that they too may be vulnerable to similar attacks and provide a full accounting of what happened.

Hack a Long Time Coming

This attack appears to have begun in mid-May. It appears an account request was sent to Gmail for my personal email address. Google's procedure asks for a number of questions to attempt to verify account ownership. We're not clear on how the process works, but it appears that weeks after the process was initiated, the hacker somehow convinced Google's account recovery systems to add a fraudulent recovery email address to my personal Gmail account. The password used on my personal Gmail account was 20+ characters long, highly random, and not used by me on any other services so it's unlikely it was dictionary attacked or guessed.

Once the recovery email address was added, the hacker could then reinitiate the password recovery process and get reset instructions sent to the fraudulent email address. Those instructions were then used to reset my personal email this morning.

Google Apps and Privilege Escalation

Like thousands of other companies, CloudFlare uses Google Apps for email. When we first established CloudFlare.com's email address, I listed my personal email address as a recovery email for my account. The hacker was able to use Google's password recovery and have the password reset sent to my personal email for my CloudFlare.com address. Surprisingly, all CloudFlare.com accounts use two-factor authentication. We are still working with Google to understand how the hacker was able to reset the password without providing a valid two-factor authentication token.

Once the attacker had access to my CloudFlare.com email account, the hacker was able to access our Google Apps administrative panel. The hacker appears to have targeted a particular customer, and initiated a password reset request for the customer's CloudFlare.com account. We sent a copy of these requests to an administrative email account for debugging purposes and, ironically, to watch for invalid password reset requests. The hacker was able to access this account in Google Apps and verify the password reset. At that point, the attacker was able to log into the customer's CloudFlare account and change DNS settings to temporarily redirect the site.

Working With Google to Resolve

We were aware of the incident immediately. We have senior contacts at Google who we worked with in order to regain control of the Google Apps accounts (both my personal Gmail account and my CloudFlare.com account). We were able to revert the change to the customer's account. We manually reviewed all other password reset requests and DNS changes. There were no other CloudFlare.com accounts that were accessed or altered.

To ensure that no other accounts can be compromised, we have invalidated all the password reset logs. We have also removed copies of password reset requests from being set to any administrative email accounts in case our Google Apps account is compromised in the future. From our investigations, it appears that at no time was our database accessed or any additional client data exposed. It appears this was, in effect, a very elaborate and sophisticated attack targeting one particular customer's login information.

Protecting Yourself

My personal email address has been removed from any association with CloudFlare. I've also added two-factor authentication to my personal Gmail account -- something that this incident highlights the importance of. I would recommend if you are using Gmail or Google apps, you take the following steps as soon as possible:

Add two-factor authentication to your account by following the steps here;
Ensure your password on your email account is extremely strong and not used on any other services; and
Change any password recovery email to an account that you do not use for anything else and cannot easily be guessed by a determined hacker.

The final puzzle we don't yet know the answer to is how the hacker was able to bypass Google's two-factor authentication on CloudFlare.com email address. That is troubling. That should have prevented this attack, even if the attacker had the password, so it remains concerning to us that it did not. We are working with Google to understand how two-factor authentication was disabled. As we learn more, we'll update this post.

Update (Saturday, June 2, 2012, 7:40 GMT):Just received notice from Google that they tracked down the issue core issue that allowed a compromise of the two-factor authentication system. Google reports that they discovered a "subtle flaw affecting not 2-step verification itself, but the account recovery flow for some accounts. We've now blocked that attack vector to prevent further abuse." That's great news. I want to reiterate that the Google Security team has, at all times throughout this incident, been responsive and attentive to the issue. In my opinion, they are the model of security on the Internet and we continue to trust them to power email for CloudFlare.com.

Update (Saturday, June 2, 2012, 19:37 GMT): We have found no evidence of unauthorized access to CloudFlare's core systems or other customers accounts. We continue to work with Google to understand the nature of how the Google App's platform was breached. In a review of the contents of the email accounts that were compromised, we discovered some customers' API keys were present. In order to ensure they could not be used as an attack vector, we reset all customer API keys and disabled the process that would previously email them in certain cases to CloudFlare administrator accounts. If you're using an app like the CloudFlare WordPress plugin, you'll need to reenter your new API key.

We've received questions some questions from customers about credit card numbers. CloudFlare's payment systems are designed to never see any credit card numbers. Credit card data is sent directly to a secure payment processor without ever passing through CloudFlare's servers. This is designed to protect sensitive account information even in the case of a full breach by a fully privileged administrator.

Update (Monday, June 4, 2012, 1:40 GMT): Working with Google we believe we have discovered the vulnerability that allowed the hacker to access my personal Gmail account, which was what began the chain of events. It appears to have involved a breach of AT&T's systems that compromised the out-of-band verification. The upshot is that if an attacker knows your phone number and your phone number is listed as a possible recovery method for your Google account then, at best, your Google account may only be as secure as your voicemail PIN. In this case, we believe AT&T was compromised, potentially through social engineering of their support staff, allowing the hacker to bypass even the security of the PIN. We have removed all phone numbers as authorized Google account recovery methods. We are following up with AT&T to get more details.

Update (Thursday, June 7, 2012, 1:30 GMT): We've gotten a clearer picture of how AT&T's systems were circumvented. The morning of June 1, AT&T's customer support receive a call from the hacker impersonating me. AT&T's logs show that the hacker was not able to answer the account's official security question, but the customer support agent verified the account with the last four digits of my social security number. What's strange about that is the account is a corporate account and should not contain my SSN. If the hacker had CloudFlare's EIN, under AT&T's policies that should not have been sufficient to verify the account. As the senior customer support manager said as he was reading through the logs, "This is odd. This is very odd." The hacker asked the voicemail box to be redirected to the phone number (347) 291-1346. That is a landline or VoIP line controlled by Bandwidth.com. That, subsequently, allowed the hacker to fool Google's voice authentication system into leaving the account recovery PIN on my voicemail.

AT&T's Fraud and Criminal Investigations Research Team is continuing their investigation. In the meantime, we have added additional security to our AT&T account. While it is not a well advertised option, it is possible for businesses and individuals to add a 4 - 20 digit passcode which restricts all changes to the account unless the passcode is known. If you are interested in this option, you will need to complete a form provided by AT&T. First level customer service agents cannot add the passcode, and many agents appear to not be aware of the option. If you call to add it, ask to talk to a supervisor and follow up to confirm that the passcode has been properly added.

Finally, I'm happy to report that, as of last night, Google appears to have removed the voice verification option from the Gmail account recovery process. Given how easy it appears to redirect voicemail, voice calls should not be considered a secure out-of-band channel. While AT&T has assured me that there is no way to redirect SMS messages, we have removed any verification channel that relies on a mobile provider network. Instead, our two-factor authentication uses the Google Authenticator app which is free and can be installed on most smart phones. This ensures that there is a completely out-of-band authentication system that does not pass through a potentially insecure carrier's network.

Update (Tuesday, June 26, 2012, 21:40 GMT): It appears that the FBI has arrested most, if not all, of the individuals involved with the attack.

Update: More Page View Counting Refinement

Matthew Prince — Mon, 28 May 2012 19:12:00 -0700

We've written about the challenge of classifying what is a "page view" a couple times before (see Understanding Analytics: When Is a Page View Not a Page View? and A Quick Update on Page Views). CloudFlare sees all web traffic so we're able to more accurately report on numbers like Unique IPs and Hits than other analytics systems like Google Analytics that use beacon-based tracking.

Beacon-Based vs. Hits Tracking

Google Analytics will only see the visitors that trigger their Javascript beacon, so they can't report on crawlers and bots that don't fire the request. Google Analytics also only sees the views of the pages where the beacon is fired, which means if you're trying to get an operations number like the number of requests per second your server is handling you have to estimate it rather than knowing it precisely.

CloudFlare has the opposite challenge. We see every request from every visitor to your site. As a result, our hits number is not an estimation but an exact count. Similarly, our Uniques number is a precise, deduped count of the number of unique IP addresses that visited your site. With CloudFlare, you don't need to trigger Javascript to be counted, so we end up counting a lot of traffic beacon-based analytics systems miss.

That's not to say Google Analytics and other beacon-based tracking is bad. There's a place for both. If you're trying to see how many ads you serve, which is Google's primary interest, then it's good to use a tracking method that is the same as how ad tags are triggered. Therefore, beacon-based tracking makes sense. If you're trying to understand the total load on your server and other operations issues, which is CloudFlare's primary interest, then it make sense to count total requests.

What Hits Are Page Views?

Our challenge is that we then have to look at all the "hits" we see and classify which one of those actually counts as a page view. We're constantly making refinements to the algorithm that make it more accurate. We just pushed out a change to this algorithm late last week. It fixes some cases where objects were reporting their content type as text/html when they were actually images or non-HTML that shouldn't be counted as page views. It also fixes instances where we were counting some 300 redirects as page views, which effectively caused double counting of page views in some cases.

The net impact on overall page view stats for most sites is very small (less than 1%). However, for some sites they'll see a more significant drop (as much as 20%). The change will impact analytics data going forward beginning last Thursday. Unfortunately, we don't store the raw historical data so it's not possible for us to update past analytics reports.

The cases where there is a larger drop align with sites that previously reported a high deviation between our page view numbers and those numbers reported from Google Analytics. Now the two page view stats should be closer in line with one another, although CloudFlare still should report a higher number because we're picking up page views from crawlers and other visitors that don't trigger Javascript. As you'd expect, there's no change to the uniques or hits numbers since those are much more straight forward for us to count and report. We'll continue to refine all our analytics systems to report data about your as accurately as possible.

Global IPv6 Challenge: No More Excuses, Enable the Future

Matthew Prince — Fri, 25 May 2012 00:36:00 -0700

Wednesday, June 6, 2012 is World IPv6 Day. At CloudFlare we're issuing a challenge to every site on the Internet: it's time to support IPv6 and enable the future. CloudFlare has made supporting IPv6 free and easy. Once you're signed up for CloudFlare, it only takes a single click to use our Automatic IPv6 Gateway and join the modern web. In other words: the web is out of excuses, it's time to join the future.

The Challenge

We'll be migrating all CloudFlare users who haven't enabled IPv6 yet to the SAFE IPv6 mode over the next twelve days. If you're a CloudFlare user, watch for an email with details. SAFE mode mimics the same IPv6 strategy used by big sites like Google and Facebook, allowing IPv6 traffic to come to a special subdomain on your site (e.g., ipv6.example.com). We've successfully tested this since we launched our IPv6 Gateway 9 months ago and it works great. Going forward, it will be the default option for all CloudFlare users.

Beyond the default mode, we're encouraging all our users to try FULL IPv6 mode before the June 6, 2012 deadline. FULL IPv6 mode enables a "dual stack," meaning that IPv4 and IPv6 traffic can access your site without having to go to a special domain. This blog, as well as CloudFlare's website, are currently running on FULL IPv6. While last year's World IPv6 Day demonstrated there were still some networks that had misconfigured IPv6 networks, our experience is that those problems have been largely addressed. You can enable FULL IPv6 Mode for all your websites by visiting our Global IPv6 Challenge page:

www.cloudflare.com/ipv6-challenge

Testing IPv6 Networks

In addition to enabling IPv6, we want to gather data on ISPs that are not properly supporting the protocol. Next week we'll be launching the IPv6 Network Testing app. Any website on CloudFlare's network can install the app with a single click. The app runs silently on a sample set of your pages without interfering with other aspects of the page or slowing down your page loads.

The IPv6 Network Testing app tests 1) whether your visitors have IPv6 support; and 2) whether they can support "dual stack" websites that run FULL IPv6 mode. For sites that enable the app, we will not only give you visibility for your own visitors, but we will also publish aggregate data on the overall state of the web and reach out to ISPs we discover with broken networks.

CloudFlare, IPv6 & the Future

CloudFlare's mission is to help build a better web. As the Internet runs low on IPv4 address space, the web will suffer. IPv6 presents a classic chicken and egg problem. We're doing our part in helping address that problem by making it easy for anyone with a website to be available on IPv6 for free. We hope you'll help us spread the word about our Global IPv6 Challenge to your fellow webmasters, network operations teams and favorite websites.

We're proud of the fact that CloudFlare is already one of the top 5 providers of IPv6 connectivity for the largest sites on the Internet. But, as this infographic from Hackertarget.com illustrates, we still have a long way to go. Our challenge is simply: it's time for the web to stop making excuses; join us in enabling IPv6 support today. Learn more:

www.cloudflare.com/ipv6-challenge

CloudFlare App: Experimently - First Heat Map App Now Available

Kristin Tarr — Thu, 24 May 2012 04:25:00 -0700

This is the first app of its kind in the CloudFlare app marketplace and we couldn't be happier to welcome Experimently as our newest app partner. Experimently provides free heat maps (up to 5,000 visitors per month) and A/B testing for any website.

With heat maps, you will visually see where your visitors are clicking so you can see if that image should be made into a link to your sign up form, order page or any other site page. Experimently also lets you create online tests on any website without having to know HTML or graphic design.

Even Easier With CloudFlare

One of the most exciting features of this offer is the sign up process. Customers can sign up for Experimently via CloudFlare in an instant with Single Sign-On (SSO) provisioning and instant home page heat map functionality, even before you dive into the rich features available. Simply toggle Experimently on and see where your site visitors are clicking the most on your home page.

Rich Functionality

Experimently lets you:

Visually edit your site with point-&-click and drag-&-drop ease.
Change, replace, or remove any element on your site including text copy, images, buttons, colors, style sheets or anything else.
Experimently automatically builds everything for you, then displays all the variations until there is a winner.
No need for coding, data entry, juggling spreadsheets, or guessing how long the experiment should last to ensure the results are accurate.
Track how many visitors click on specific buttons, links, submit forms or reach a thank you page by creating conversion goals.
No need for programmers - Experimently was built for marketers.

Programs for Sites of All Size

Experimently's programs start at free, with additional pricing plans available to meet the needs of sites of all sizes, ranging from $99/month for 30,000 unique visitors to $999/month for 500,000 unique visitors.

Visit the Experimently app page now!

Moog Music: Staying online when Google doodles you

Matthew Prince — Wed, 23 May 2012 22:46:00 -0700

Today Google celebrated the birthday of Bob Moog, the creator of the Moog keyboard, with an interactive doodle. If you haven't played with it, it's pretty cool. Check out, for example, this version of the Legend of Zelda theme music composed with today's doodle.

You're Going to See Some Traffic

The top result when you search for Moog on Google is MoogMusic.com. The site celebrates Bob Moog's legacy and sells versions of his keyboards and other devices (including a pretty sweet iPhone app). Google contacted the site owners to give them notice that there was going to be a doodle four days ago. The only catch, they had to keep the information strictly confidential until the doodle went live in Australia (the first region to see new doodles every day). Oh, and, by the way, Google noted, they should prepare for a crushing load of traffic to their website.

That presented a bit of a conundrum. Moog Music turned to their web developers, a technically savvy shop called PurpleCat.net. The site was designed to handle half a million hits a day, so the team at PurpleCat spent the next three days making several improvements to the site to increase its capacity. They migrated the site to new hardware, secured a dedicated gigabit port, migrated the mysql database to its own server, upgraded the backend of the site replacing Apache with NGINX and added layers of caching and optimizations. Needless to say, it was a busy three days for the PurpleCat team while they did all the upgrades to prepare for the traffic surge. The site, however, was on a single virtual private server (VPS) with limited resources and it didn't make sense to upgrade the whole infrastructure for a one-day spike in traffic.

Turning to CloudFlare to Help

As soon as the doodle went live in Australia and the embargo was lifted, PurpleCat called us at CloudFlare and explained what was about to happen. We were happy to help having had a lot of experience dealing with floods of traffic from events like these. The MoogMusic.com site went live behind CloudFlare and we saw the traffic begin to rise.

Things were ok at first, but as the sun rose in Asia, errors started to be returned from the stressed MoogMusic.com VPS. CloudFlare's Always Online technology kicked in and served cached versions of the page, but the dynamic portions were at times inaccessible and the pages loaded slowly as our servers waited for a response or timeout from the backend. It wasn't ideal.

Page Rules and Advanced Caching to the Rescue

We knew we could help even more, so our support team worked with the folks at PurpleCat to setup Page Rules for the MoogMusic.com site. While some of the site is dynamic, large portions are relatively static. That meant a Page Rule could define the static portions -- like the front page and a history of Moog's legacy -- and instruct CloudFlare to Cache Everything, even the HTML. That means we will serve all requests to the pages that match the rule from our infrastructure without any requests getting sent to the VPS.

The results were dramatic. Approximately 92% of requests that would have previously hit the VPS were instead fielded directly by CloudFlare. We also reduced bandwidth needed to serve the requests to MoogMusic.com by more than 90%. With all the static requests offloaded to CloudFlare, the VPS had plenty of capacity to return the dynamic, uncachable HTML of the site.

Focus On Great Content, We'll Keep It Online

MoogMusic.com is currently averaging more than 100 requests per second. (What's pretty incredible is that's less than 1/1,000th of CloudFlare's overall load.) The VPS is doing more traffic every 15 minutes than the VPS it's running on is designed to handle over 24 hours. Peter from PurpleCat sent us a nice post thanking us for our help. He also shared the following with John on our team:

your ceo will like this.we built a separate server to just serve this page up:

http://190-2.purplecat.net/

But thanks to cloudflare, we never had to put it up.

He's right, I do love that. And so does everyone else at CloudFlare. It's exactly why our whole team comes to work every day excited to solve what seem like impossible challenges... like scaling a single VPS to deal with the traffic from the Google doodle.

DNSChanger Update: Nearly 4% of Infections Already Detected

Matthew Prince — Mon, 21 May 2012 00:33:00 -0700

Just a quick update on the initiative between CloudFlare, OpenDNS, and the DCWG to clean up the DNSChanger malware. In the last week, just over 11,000 websites enabled the Visitor DNSChanger Detector App through CloudFlare. Since then, those sites have collectively served more than 56 million page views. Just over 12,000 visitors to those websites have seen the warning about the DNSChanger virus and clicked on the link to learn more and clean up their infection. In just the first week, that's nearly 4% of the total number of estimated infected computersthat the CloudFlare community has already helped notify and get cleaned up.

While hundreds of thousands of computers are still infected and risk losing access to the Internet on July 9, 2012, we're proud of the strong start to this effort by the CloudFlare community along with OpenDNS and the DCWG.

If you haven't yet enabled the Visitor DNSChanger Detector App for your sites on CloudFlare, you can do so by following this link. Thanks for helping us #savetheweb.

Never Deal With DNS Propagation Again

John Roberts — Sat, 19 May 2012 04:13:00 -0700

At CloudFlare, we think a lot about the Domain Name Service -- better known as DNS, the Internet's address book. CloudFlare uses DNS to bring performance and security to our customers. But, we don't spend a lot of time talking about our DNS service, behind the scenes.

The "CloudFlare benefit"

One recent customer comment reminded us of an unsung advantage:

"The thing is, we had this new design running on a QA server for the past week, doing final testing and updating. Yesterday we changed the IP to point the domain to the new QA server, away from the "old" prod server.

BUT unlike every other time when we change a site's IP - the change was instantaneous‚ since the only IP the world has for our sites is CloudFlare, so when we change servers, IPs, etc.

So with CloudFlare, instant server move, web host move, IP change -- never deal with DNS propagation again. That is awesome."

No More Waiting

Because the public IP addresses don't change, using CloudFlare removes the wait for TTL (Time To Live) record expiration and DNS propagation when you make a change to one of your site's DNS records. This waiting period is often an excruciating part of development and deployment changes, because it's completely out of your control. Yes, you can lower your TTLs ahead of time, but it's still a seemingly endless wait for the world's recursive DNS servers to come get the new record.

If DNS isn't something you spend a lot of time thinking about, that's fine. Just know that once you're on CloudFlare, your changes are live faster, and you can get back to everything else on your plate. Making running a website easier...that's part of CloudFlare's goal, even in the small things.

P.S. - We recently made major improvements to our DNS management interface. You can also manage DNS via API.

CloudFlare Meetup - Monetizing Your Sites: Your Questions. Answered.

Kristin Tarr — Thu, 10 May 2012 19:59:00 -0700

As a publisher, boosting revenue is often top of mind. Join us for a meet up in San Francisco on Thursday, May 31.

Site monetization experts from VigLink, SayMedia and CloudFlare will be on hand to discuss the best practices for monetizing your site. They will share tips on how to boost revenue through advertising and affiliate programs, and share common pitfalls to avoid. It will be a lively and informative evening.

The event opens at 7:00pm at our office in SOMA at 3rd and Townsend. Sign up here

Monetizing Your Site: Your Questions. Answered
Thursday, May 31, 2012
Doors open at 6:30pm, Panel starts at 7:00pm
CloudFlare Office - 665 3rd Street, Suite 207 (SOMA)

Do you have a meetup suggestion? Would you like to host a meetup of your own at the CloudFlare office? Let us know by commenting below or sending an email to meetups@cloudflare.com.

We hope to see you on May 31!

CloudFlare & OpenDNS Work Together to Help the Web

Matthew Prince — Thu, 03 May 2012 15:00:00 -0700

Several years ago, some suspected cyber criminals on the Internet wrote a family of malware dubbed DNSChanger. About a year ago, law enforcement tracked down the suspected cyber criminals behind this malware, arrested them, and took over the servers they were using to redirect customers to rogue sites.

As a result of a court order, the Internet Systems Consortium (ISC) under the direction of the FBI, has continued to run the DNS servers used by the malware for the last year. However, the court order will soon expire and those servers are scheduled to be shut down on July 9, 2012. When that happens, hundreds of thousands of Internet users whose systems are still infected and/or affected could lose access to the web, email, and anything else that depends on DNS. This is the story of how two Internet infrastructure startups — CloudFlare and OpenDNS — are playing a small part to help solve the problem.

A Bit of DNS Background

Up front, in order to understand this story, you need to understand there are two types of DNS servers: recursive and authoritative. Everyone who uses the Internet needs a recursive DNS server. Your ISP usually provides these types of services or you can use a provider like OpenDNS, Google, DNSAdvantage, other public resolvers, or even run a server yourself to handle your recursive DNS queries.

On the other hand, every domain needs at least one authoritative DNS server. Authoritative servers are where a particular domain's records are hosted and published. Many domain registrars provide authoritative DNS servers, or you can use a service like CloudFlare and we provide authoritative DNS. When an Internet user types a Universal Resource Identifier (URI) aka Universal Resource Locator (URL) into their browser, clicks on a link, or sends an email, their computer queries their recursive DNS provider. If the recursive DNS provider has the answer cached then it responds. If it doesn't have the answer cached, or if the answer it has is stale, then the recursive DNS server queries the authoritative DNS server.

As mentioned above, OpenDNS provides recursive DNS. Their customers are web surfers and they provide a terrific service that helps speed up Internet browsing and protect people on the web from malware. CloudFlare provides authoritative DNS. Our customers are websites and we make those sites faster and protect sites from attacks directed at them. While we're often asked if OpenDNS and CloudFlare are competitive, in reality both services are complementary just using different parts of DNS (recursive and authoritative) to achieve a similar mission: a faster, safer, better Internet.

How Suspected Cyber Criminals Use DNS to Do Bad Things

The DNSChanger malware family was designed to change the recursive DNS server that Internet users' computers queries. Instead of directing DNS queries at the recursive server you or your ISP configured, the malware modified computer settings to route queries to recursive DNS servers controlled by the suspected cyber criminals.

The job of DNS is to translate a domain name such as dcwg.org, which humans prefer, into an IP address, like 108.162.205.64, which servers and routers can use. If you are a cyber criminal and you can gain control over someone's recursive DNS then you can direct traffic to certain sites to a fake version of the site. Once DNSChanger had web surfers querying rogue recursive DNS servers, all requests for legitimate websites could be directed to a fake website. For example, even if you typed your bank's domain name into your browser, if the suspected cyber criminals control recursive DNS then they can send you to a malicious site and steal your information.

Over the years DNSChanger operated unchecked, more than a million computers and home routers had their DNS configurations modified. Thankfully, law enforcement was able to track down the suspected cyber criminals behind the malware, arrest them, and seize control of the rogue recursive DNS servers. Unfortunately, hundreds of thousands of computers are still using the formerly rogue recursive DNS servers. On July 9, 2012 the court order directing ISC to operate the servers expires and those servers are scheduled to be shut down. On that date, all systems which still have their DNS settings modified by DNSChanger will effectively be cut off from the Internet.

Getting the Word Out

The DNSChanger Working Group (DCWG), a loosely affiliated organization comprised of some of the world's largest and most competent ISPs, search engines vendors, software vendors, security companies, and others, has been working to get the word out about the problem and reduce the impact of the shutdown of the DNSChanger recursive servers. The DCWG launched a website (dcwg.org) to provide information about the malware, let people test whether they are infected, and provide recommendations on how to fix their systems. CloudFlare first became involved when the folks at dcwg.org reached out to us because their site was under heavy load after attention from major media outlets. CloudFlare helped keep the dcwg.org website online under the load caused by media attention over the last 10 days. We offloaded more than 95% of the traffic to the site, ensuring the site ran fast and stable even when it was being featured on the front page of cnn.com.

Unfortunately, one of the challenges in trying to address situations like DNSChanger is that you only know to go to the dcwg.org website if you already know about it. What you needed was something akin to an emergency broadcast system that would inform people who were infected that they had a problem as they surfed the web. In the process of working with the DCWG, we realized we might be able to help.

Some of our engineers created an app named Visitor DNSChanger Detector App. Any website on CloudFlare can enable the app with a single click from our apps marketplace. The app installs a small bit of Javascript on the page that tests visitors to see if they're infected. If the tests do not detect anything, nothing happens. If the tests indicate that the DNSChanger recursive servers are being used, then a banner is displayed across the top of the page and visitors are directed to instructions on how to clean up the infection (more on that in a second).

More than 470 million people pass through CloudFlare's network on a monthly basis. Our data suggest that more than half of the people infected with DNSChanger would visit at least one site on CloudFlare per month. The power of the Visitor DNSChanger Detector App is that as CloudFlare publishers enable it then there is an increasing likelihood that people who are infected will get information about their infection before they are no longer able to use the Internet on July 9, 2012.

While we've made it extremely easy for publishers on CloudFlare's network to help get the word out, we didn't want to restrict participation to only those sites using our service. We therefore decided to release the code for the checks publicly and as open source so anyone who can install a few lines of Javascript on their web pages will be able to install it on their own sites to inform their potentially infected users. You can access the code from the following GitHub Repo. We're hopeful that sites both large and small will take the time to install the code in order to help inform their visitors who may be infected.

What Should People Notified of This Infection Do?

While CloudFlare is able to assist with informing web surfers they have an infection, we aren't particularly well situated to actually fix the problem. After all, it isn't our customers that are directly impacted, but rather the customers of our customers. Many of the folks infected can get help from their ISPs, but for some this might not be an option. CloudFlare reached out to David Ulevitch, the CEO of OpenDNS and he saw this as a great opportunity to further OpenDNS's mission of helping build a better Internet. We added OpenDNS as a resource for publishers to display to their customers when the Javascript detects the use of the DNSChanger recursive servers.

The Power of the DNS

This incident illustrates to me the importance and power of the DNS system that underpins the Internet. The suspected cyber criminals were able to modify DNS settings to steal advertising revenue and perform other illegal activities. CloudFlare uses authoritative DNS in order to provision powerful tools to make sites faster and even help create a sort of emergency warning system for the Internet. OpenDNS provides high performance recursive DNS caching services for their customers. Combined, we hope to help the DCWG get the word out so the hundreds of thousands of Internet users still impacted by the DNSChanger malware will be able to take steps to ensure they'll be able to use the Internet on July 10, 2012 and beyond.

Today's Outage Post Mortem

Matthew Prince — Wed, 02 May 2012 22:41:00 -0700

CloudFlare had an outage across much of our network today. The outage began at 20:19 (GMT). It affected approximately 75% of traffic to CloudFlare's network. The length of time for the outage varied depending on region, but the maximum period of downtime was approximately 15 minutes. I wanted to quickly get information out about what happened, why it happened, and what we're doing to ensure it never happens again.

Routes, Routers and Routing

To understand the problem, you need to understand a bit about how Internet routing works. The Internet is a massively interconnected network. Networks send packets to each other across routes. These routes are set for each network by routers. A route defines the path for packets to take to get to a particular IP address. One network will announce that it is responsible for a particular set of IP addresses. That fact is then shared to upstream routers so if they see a packet bound for a particular IP they can send it in the correct direction.

Routers exchange routes between each other using something called Border Gateway Protocol (BGP). When two networks interconnect, they generally trust each other's routes. If a routing change is announced by one router, the immediately connected upstream routers will pickup the routing change. They will subsequently pass the change on to other routers that are further upstream.

Bad Route to Hong Kong

Today we had a scheduled maintenance for our Hong Kong data center while its systems were being upgraded. The data center was taken offline by shutting down all the in-bound Anycast routes. This, as we intended, caused all traffic that would have gone to that data center to hit the next closest facility (either Singapore or Tokyo).

While the systems were being upgraded, our network team worked to optimize some of the routing in Hong Kong. At some point, the out-bound routes were entered into the in-bound interface. The out-bound routes describe our entire net range so the net effect was the router in Hong Kong was announcing that it was the correct place to send all traffic bound for CloudFlare's IP space.

Our upstream provider trusts our routes so, via BGP, they were quickly relayed throughout their network and to their upstreams. The result was traffic from around the world was directed to the Hong Kong data center, which was offline. We realized the issue and announced the corrected routes. It took approximately 15 minutes from the beginning of the problem to when the routes were corrected network wide. About 25% of CloudFlare's in-bound traffic comes from direct peers. This traffic was not affected by the routing because the direct peers trusted our routes more than the ones they were receiving from other upstreams.

Future Prevention

We are implementing systems to run all routing changes through a verification layer that double check before any routes are announced. We are also talking with all our upstream providers to enable additional checks on their networks that do not automatically propogate major routing changes without confirmation.

This is only the second significant outage in CloudFlare's history (here's our post mortem from the other). Any period of downtime is completely unacceptable to us. On behalf of our whole team, I apologize for the problem. We have learned from this experience and are already implementing the safeguards to ensure it will not happen again.

CloudFlare—WebOps for everyone

John Graham-Cumming — Tue, 01 May 2012 21:41:00 -0700

No matter whether you run a personal blog or the IT operation of a corporate enterprise, you have discovered that in addition to running a web site, and updating its content or application, a web site comes with difficult operational challenges. CloudFlare handles these WebOps challenges with a simple, five minute change to a web site's settings.

From Flickr user flightlog

CloudFlare's WebOps-as-a-service covers five main areas: Security, Metrics, Acceleration, Reach and Transformation (SMART). With CloudFlare, every website owner (from the smallest to the largest) gains access to the tools that have been reserved for the largest web sites in the world.

Security

Web sites are constantly attacked by hackers from around the world. Some look to turn a web site into a command-and-control server for a network of malicious bots, or to use a site to host a fake bank for phishing, steal sensitive private information (such as credit card information), or host malware. Attackers can range from organized criminal groups that use the threat of attacks to extort money to simple vandals.

What is worse is that attacks are a never ending battle. Everything from the operating system of the server hosting a web site to each individual fragment of code (such as blogging software) is a potential attack point, and must be constantly updated and fortified to evade the latest threats. This creates an enormous management burden for a web site owner and requires expertise that few possess outside of the world's largest web sites.

Because of CloudFlare's global reach and traffic we are able to automatically block attacks as they occur, and instantaneously roll out patches that prevent newly discovered vulnerabilities from ever exposing your servers to attack.

Additionally, CloudFlare's global network means that Distributed Denial of Service attacks (DDoS) can be controlled and absorbed before they ever reach your server. Web sites around the world rely on CloudFlare for DDoS protection precisely because it is so difficult: effective DDoS protection requires an intimate knowledge of the operation of the Internet and a 24/7 operations team on hand to deal with attacks as they emerge.

Many web site owners also find that SSL encryption is necessary to ensure that connections to their web site are encrypted, and their visitors protected. But SSL is complex to set up and requires specialist knowledge. CloudFlare's SSL service enables web sites to be SSL-enabled with a single click.

Finally, for publishers, CloudFlare's ScrapeShield app automatically protects valuable content against automated scraping tools and tracks content that is stolen.

Metrics

Every web site owner knows that metrics are critical to track who visits your site, which pages or content are popular and how visitors come to find your site to begin with. CloudFlare makes it trivial to enable any number of metrics services with single click deployment of the service on to every one of your pages, and provides its own metrics service with highly accurate data as an additional complement.

For example, adding Google Analytics or Clicky to a CloudFlare site is a simple one click operation. No need to change the code yourself; CloudFlare automatically inserts the JavaScript necessary and within seconds metrics start to be collected.

CloudFlare also collects and makes available its own highly accurate metrics. Because CloudFlare sees every page view and hit for each web site it is able to provide 100% accurate metrics based on the actual visits made by visitors to your web site.

CloudFlare does not rely on JavaScript inserted into a page to track metrics unlike other common metrics tools. Rather, CloudFlare sees and reports on every page request even from visitors that deliberately block JavaScript or tracking tools. CloudFlare is also able to report on malicious traffic (in the form of hackers and bots) as it sees, blocks and records those visits as well.

Acceleration

Study after study has shown that web site speed is directly linked to revenue and visitor satisfaction. Even a tiny millisecond delay in the load time of a web page causes people to leave a site (and, perhaps, never come back) which means lost engagement and revenue. Page speed is also taken into account by large search engines when deciding on how highly to rank a page, and therefore can play a significant part in your search engine optimization (SEO) efforts.

CloudFlare's content acceleration and caching services mean that web sites using CloudFlare see automatic acceleration of their web site just by signing up. CloudFlare automatically caches content so that it can be delivered quickly to a site's visitors around the world, and optimizes content that can't be cached so that it is delivered as fast as possible. On average, our customers' web sites load 2X as fast after signing up for the service.

CloudFlare also has a collection of available acceleration tools that can be enabled with one click. These tools perform content optimization, such as minimization of image sizes, minification of JavaScript, loading JavaScript asynchronously and preloading parts of a page, further improving page load times.

Not only does CloudFlare cache and optimize your content, it also reduces the bandwidth used by your web server. This means that your web site saves money on bandwidth while at the same time improving performance for your visitors.

Reach

CloudFlare operates data centers around the world and uses an Anycast network that directs a web site's visitors to the server nearest to them. After signing up for CloudFlare, your web site will have instant global reach---within five minutes of signing up with CloudFlare a web site is distributed around the world and visitors from every corner of the globe see an instant performance upgrade.

CloudFlare's global network also means that Internet outages around the world do not affect your web site. Our team constantly monitors data center and Internet performance to ensure that the best route is taken for every individual visitor to your site ensuring that visitors, wherever they are, experience fast and always available web sites.

Additional CloudFlare services keep a web site online even when the actual web server is down and perform geolocation so that web servers can instantly understand where visitors come from.

CloudFlare can also automatically enable IPv6 on any web site so that visitors from the newest reaches of the Internet can visit web sites that continue to use the older IPv4 protocol—all with the click of a single button in the CloudFlare management UI.

Transformation

As traffic from a web site protected and accelerated by CloudFlare passes through CloudFlare's servers it is transformed. Some transformations target security (such as filtering out bad requests) and others performance (such as optimizing JavaScript).

But CloudFlare's available transformations go a step further: CloudFlare is able to modify your web site's pages automatically. For example, turning on a service like Google Analytics can be achieved in a single click in CloudFlare's management interface. Once enabled CloudFlare will automatically insert the appropriate JavaScript in every page. There is no need for you or your staff to make any changes to the web site itself.

CloudFlare can also protect sensitive content from potentially malicious users. For example, email addresses can be automatically detected and obfuscated so that humans can read them but machines can't (helping to cut down on spam). A "Server Side Exclude" feature allows a web site owner to mark content so that it is hidden from suspicious visitors (such as potential bots scraping content).

Another transformation provides automatic hotlink protection for images so that valuable bandwidth isn't taken by third-party web sites that embed images directly from a CloudFlare protected site.

CloudFlare can automatically minify HTML, JavaScript and CSS to make it smaller and load faster, and CloudFlare's ScrapeShield app can insert tracking beacons on your web sites to detect and track content theft.

As new devices proliferate (such as tablets or smartphones) CloudFlare's transformation features reformat a web page for optimal viewing on those devices, automatically. What's best is that every single transformation requires nothing more than signing up for CloudFlare's SMART webops service.

Summary

CloudFlare is a 24/7 WebOps team for any web site, no matter the size. It provides security, metrics, acceleration, reach and transformation with minimal change. And as CloudFlare enhances its services, all web sites using CloudFlare receive the benefit, automatically. CloudFlare's customers range from individual bloggers to Fortune 100 corporations, and even national governments.

App: GoSquared, Real-time Web Analytics So You Can Act Now. Not Tomorrow.

Kristin Tarr — Tue, 01 May 2012 04:48:00 -0700

We are excited to welcome GoSquared as the next CloudFlare app. The GoSquared team is based in London, but we were pleased to meet them on a recent visit to California. Both of us are excited to help make their real-time web analytics service incredibly easy to turn on for CloudFlare-powered sites.

GoSquared analytics allows you to react and respond to activity on your website in real-time, as it happens. Watch their video to learn more:

GoSquared provides all your essential metrics in one beautiful dashboard that is scalable to any device and looks great on a large screen, keeping the whole team driven by data. GoSquared also has a powerful API available for anyone to integrate real-time data into their site.

GoSquared features

With GoSquared you can respond, engage, and act now. Not tomorrow.

GoSquared's real-time dashboard allows you to view what's happening on your website in a single glance.
See what's popular on your site right now.
See where your visitors are coming from and whether they're engaging with your content.
Monitor the impact and effectiveness of campaigns in real-time.
Receive detailed alerts when traffic levels on your site are out of the ordinary.
React to visitor activity on your site before the opportunity has passed.
iPad app so you're always in the loop

Easy instant on

Via CloudFlare Apps, you can turn on GoSquared instantly, with the right size plan for your site. No code to deploy. Interested in learning more? Visit the GoSquared App page today!

App: Panopta Provides Advanced Server Monitoring and Outage Management Services

Kristin Tarr — Thu, 26 Apr 2012 00:58:00 -0700

CloudFlare customers have websites of all sizes, so we're pleased to introduced a new advanced server monitoring and outage management service from Panopta as a CloudFlare App. Appealing to both enterprises and SMBs, Panopta will be the first to tell you if your infrastructure is down and provide you with tools to fix it.

What makes Panopta different? They offer three simple and unique areas that separate them from the rest of the market.

Deep and Wide Monitoring
No False Alerts
Intelligent Alerting

Deep and Wide Monitoring

Panopta gives you in-depth checks every 60 seconds from their global monitoring network, the Panopta Monitoring Agent and the Panopta Monitoring Appliance.

No False Alerts

Panopta guarantees no false alerts. No more chasing problems that aren't there.

Intelligent Alerting

Alerts are escalated to the right people at the right time, so you can rest assured that problems will be fixed as soon as possible.

Getting Started

Panopta offers four different plans, including:

$15/month for Solo
$50/month for Basic
$100/month for Intermediate
$250/month for Advanced

See all the details and sign up for Panopta's advanced server monitoring service now, via CloudFlare Apps.

Immediately, the first monitor will be set up for the home page of your site, with opportunity for detailed customization and additional monitors within the Panopta control panel. Try it now!

30% More Traffic in Less Than a Blink of an Eye

Matthew Prince — Thu, 19 Apr 2012 00:43:00 -0700

CloudFlare's traffic grows in a number of different ways. The most obvious is that we sign up more websites. We also grow as the natural traffic to sites using CloudFlare increases as they get more popular themselves. Another way we grow is less obvious but extremely cool: as CloudFlare makes the web faster, visitors end up surfing more pages.

Save 25ms, Get 30% More Page Views

We had a really good example of this in just the last couple days. Our networking team was able to work some routing magic to more efficiently get traffic to our Singapore data center. This, on average, saved about 25 milliseconds -- 0.025 seconds, a tiny sliver of time, about 1/12th the time it takes you to a blink of your eyes -- for requests from countries in the region including Thailand, Malaysia, and Indonesia.

The graph above shows traffic over the last 9 days from TOT, one of the largest ISPs in Thailand. You can see from the graph that traffic rises and falls depending on the time of day, which is normal, but if you look at the peaks you'll see they step up dramatically in the last two days.

Digging into the details, there was approximately a 30% increase across bandwidth, hits, and page views in the region after the improved routing. The increase holds even if you control for the day of the week, remove new sites that signed up over the period, and compare other regions that also benefited from the routing so as to control for other potential explanations like weather, news events, or anything else would have had more people surfing the web in Thailand in the last few days. In other words, just eliminating 25ms in latency resulted in a 30% increase in traffic. That's really cool.

Faster Means More

The very nature of the way that TCP, the protocol of the Internet, works means that any performance benefit tends to be amplified. Google, Amazon and the other Internet giants have known for a long time that faster means higher engagement and more Internet use. At CloudFlare, we have network engineers that have helped build the networks for some of those Internet giants now at work tuning connections to save milliseconds for the rest of the web. We'll continue to add data centers and improve routing toward our mission of making a faster, safer web for everyone.

CloudFlare Tips: Recommended steps after activating through a partner

Damon Billian — Mon, 16 Apr 2012 21:04:00 -0700

CloudFlare has partnered with a number of CloudFlare Certified Partners to make it simple for website owners that want a faster and safer website. Since signing up for CloudFlare through a hosting partner is different than signing up for CloudFlare directly, we wanted to provide some quick tips to help you get the most out of your CloudFlare experience.

Things you should know about right away

You do not need to change your name servers when activating through a hosting partner. You would still manage your DNS entries at your hosting provider or registrar.
CloudFlare can only be enabled for CNAME records when activating through a hosting partner. To enable CloudFlare on your root domain (yourdomain.com), which is an A record, you need to have your hosting partner set a 301 redirect from your root domain to www. Not only will the redirect help accelerate and protect the root domain, this will also make the statistics in your CloudFlare account accurate. Note: If you have a naked domain, 'yourdomain.com', and you don't want your visitors to go to 'www.yourdomain.com', then you need to signup directly with CloudFlare.
What you should do if you see any of the following error messages after enabling CloudFlare:

"Host Not Configured to Serve Web Traffic" error message will appear on the first request to your site after activating through a partner, then will go away after a few minutes. If it lasts for more than 10 minutes, then contact your hosting provider and our support teams will work together to resolve.

"CloudFlare-nginx 502 Bad Gateway": This is an issue on the CloudFlare network. We deal with these quickly (less than 10 minutes). We publish all announcements regarding our network status on @CloudFlareSys.

"Website is Unavailable": Either your server is offline and we don't have a copy of your site in cache or something on the origin server is blocking CloudFlare's IPs.

If your server is online, then work with your hosting provider to find out what could be blocking CloudFlare's IPs on your server. The most common culprit is a security solution like a firewall like CSF or IP tables. As soon as the block is removed, the error page will disappear.

Key CloudFlare features
SSL
If you have SSL on the domain(s), you will need to upgrade to a Pro account. The cost for a Pro account is $20.00 per month for the first website and $5.00 for each additional site. In addition to the SSL support, you will also receive additional security and performance benefits.

Note: You will find the option to upgrade to Pro in your CloudFlare account.

Development Mode
If you are making changes to the static content on your website, temporarily bypass CloudFlare's cache so any changes appear immediately. You can find Development Mode either right in your hosting provider's control panel or by logging in to your CloudFlare account under CloudFlare Settings.

PageRules

PageRules gives you more powerful performance and configuration options, including:

Advanced Caching Configurations

Excluding URLS from CloudFlare's default caching and security options

Setting URL forwards and redirects

Recommended (Free!) Optional CloudFlare Features
CloudFlare has developed web content optimization features called Rocket Loader and Auto Minify. Both Rocket Loader and Auto Minify are designed to load your site's resources even faster than the default CloudFlare configuration.

Rocket Loader: Rocket Loader will speed up the delivery of your pages by automatically asynchronously loading your JavaScript resources. Rocket Loader works well for websites that have a lot of ads, widgets or plugins.

Auto Minify: Removes all unnecessary characters from HTML, CSS, and JavaScript to reduce file size.

Note: Both of these features are still in beta. If you encounter any issues, such as a broken plugin or JavaScript not working properly, then please turn the feature off and report any bugs to our team.

To turn on Rocket Loader and Auto Minify, you need to log in to your CloudFlare account and go to CloudFlare Settings.

IPv6 Gateway

Make your website IPv6 compatible, by turning on the CloudFlare IPv6 gateway.

Where you can find out more about CloudFlare
The CloudFlare Support Center has answers to a number of questions. Searching our knowledge base is the fastest way to get a quick response to the majority of questions. Don't see the answer to your question? Please contact CloudFlare.

Updates and Giveways
We frequently post about product updates, early beta access to new features, system issues, and giveaways, so we recommend that you follow us on Facebook, Twitter or Google+:

Facebook
Twitter
Google+

Thank you for joining CloudFlare in partnership with your hosting provider.

App: SiteLock Helps Protect Your Online Reputation, Keeps Your Business From Being Blacklisted

Kristin Tarr — Fri, 13 Apr 2012 17:30:00 -0700

SiteLock is a website security monitoring service that protects your online reputation and provides additional security to your website.

There have never been more threats to your website than now. Hackers use malware, SQL Injection, Cross-site scripting and more sophisticated techniques to steal your customer data or redirect your traffic, ruining your site's reputation.

SiteLock will alert you if your site is vulnerable to these issues, as well as if your site gets blacklisted for any reason by search engines or spam monitoring tools. SiteLock combines two types of scanning to provide an additional layer of security beyond the existing protection of CloudFlare to ensure your investment is protected and your reputation is safe.

Proactive scanning: Searches your site and network for common weak spots hackers exploit to inject malicious code into your site

Blacklist monitoring: Monitors search engine and spam blacklists to make sure your customers are seeing your site and receiving your messages

SiteLock's security offers all of these features:

Daily 360-degree scanning for
- SQL Injections
- Cross-Site Scripting (XSS)
- Applications
- Viruses
- Malware blacklisting
- Spam blacklisting
On-Demand Expert Services to help you fix any security issue on your site
Alerts & Email Notifications
Dashboard Reports

In addition, SiteLock provides a Trust Seal for sites that are secure. The SiteLock Trust Seal provides customer confidence and has been proven to substantially increase your sales and conversions, with 70% of web visitors looking for a verifiable 3rd-party certification before providing personal data.

SiteLock is now available via the CloudFlare Apps.

Introducing: I'm Under Attack Mode

Matthew Prince — Thu, 12 Apr 2012 01:12:00 -0700

CloudFlare provides a broad level of protection from a wide range of attacks. We do this while minimizing false positives or annoyances to legitimate customers. CloudFlare didn't begin as a DDoS mitigation service, but we've rapidly found that we are good at protecting sites from these attacks. Today we're offering a new security mode to make our DDoS protection even better.

A Brief History of DDoS

In the OSI model, traditional DDoS attacks targeted the Layer 4. The so called "transport" layer of the network stack specifies the protocol (e.g., TCP or UDP). These attacks flood an interface with garbage traffic in order to overwhelm it's resources in one way or another. Usually, the attack fills up the capacity of a network switch or overwhelms a server's network card or CPU's ability to handle the traffic.

CloudFlare has largely mitigated these attacks by building out significant capacity across our network. We have fat pipes and lots of machines to absorb floods of traffic. We also make broad use of the Anycast protocol which has the effect of scattering the load of a distributed attack across multiple data centers, reducing the exposure of potential single point of failure. The result is that no packets from a traditional Layer 4 attack will ever reach a site behind CloudFlare.

HTTP-Based Attacks

A new breed of attacks targets Layer 7, the "application" layer. These attacks focus on specific characteristics of web applications that present bottlenecks. For example, the so-called Slow Read attack sends packets very slowly across multiple connections. Since Apache opens a new thread for each connection, and since connections are maintained as long as there is some traffic being sent, you can overwhelm a web server by exhaust its thread pool relatively easily.

CloudFlare has protections in place against many of these attacks, and in real world experiences we generally reduce the HTTP attack traffic by about 90%. For most attacks and most of our customers, this has been enough to keep them online. However, the 10% of traffic that gets through our traditional protections can still be overwhelming to either customers with limited resources or in the face of very large attacks. We wanted to help in these cases too, so today we're announcing something new.

I'm Under Attack Mode

Introducing "I'm Under Attack Mode." The name is pretty self-explanatory: it's a new security level you can set for your site when you're under attack. The effect is that we will add an additional set of protections to stop potentially malicious HTTP traffic from being passed to your server. While we perform a number of additional checks, the only thing noticeable to legitimate visitors to your site is that when they first arrive they'll see an interstitial page for about 5 seconds while checks are complete. Think of it as a challenge where the tests are automatic and visitors never need to fill in a CAPTCHA.

After verified as legitimate by the automated tests, visitors are able to browse your site unencumbered and won't see typically the test page again. Javascript and cookies are required for the tests and recording the fact that the tests were correctly passed. We've also designed the new checks to not block search engine crawlers, your existing whitelists, and other pre-vetted traffic. As a result, enabling I'm Under Attack Mode will not negatively impact your SEO or known legitimate visitors. What's also cool is that data on attack traffic that doesn't pass the automatic checks is fed back into CloudFlare's system to further enhance our traditional protections.

While CloudFlare did not start as a DDoS mitigation service we have realized this is an area where we can provide a lot of benefit in an easy and affordable way. I'm Under Attack Mode is the first of several new features we'll be releasing over the coming month to offer a full gauntlet of DDoS protection. Stay tuned.

ScrapeShield: The scaled up, deep intelligence anti-scraping service

John Graham-Cumming — Wed, 11 Apr 2012 15:21:00 -0700

Months before I joined CloudFlare as a programmer I signed up for the company's service to protect my blog from hackers, spammers and scrapers. I saw an instant reduction in the amount of spam, an enormous decrease in hacking attempts, and a halt to the bots that scrape site content.

All that was long before CloudFlare launched its specific anti-content scraping app called ScrapeShield that builds on CloudFlare's existing services to provide a complete package of anti-scraping and tracking tools. ScrapeShield exists, and is powerful, because of CloudFlare's deep roots watching and profiling the behavior of bad web visitors that goes far beyond the short history of the company.

Part of the original inspiration for CloudFlare was an anti-spamming project launched in 2004 by some of CloudFlare's founders called Project Honeypot. That project created an enormous secret, dark web that trapped and profiled bots of all kinds. Although it was most commonly used to stop spammers, the same information can be turned on scrapers to stop site content being stolen.

With the launch of ScrapeShield, CloudFlare has put together a package of new technologies that track scraping if it happens and alert the web site owner, and builds on the Project Honeypot foundation of 8 years of deep web intelligence. It's new, but it has a long heritage.

It's an enormous advantage to have been profiling bots and scrapers for years because it means that CloudFlare's new ScrapeShield service comes straight to the web without a long beta or learning period.

It's ready and fit for purpose from day one.

And ScrapeShield layers active anti-scraping features on top of its web intelligence. These beacons are able to detect scraping, if it happens, and alert the site owner to the scraped content. Every day now I take a look in my ScrapeShield report to see if my site's content has been stolen: the good news is that none has. That's not a surprise given how robust CloudFlare's anti-bot technology is.

But the ScrapeShield report is able to tell me more things about my site's content that weren't visible before. I get to see how often it's being read when my site's content is taken from the RSS feed and viewed off the web on iPad apps, when it's translated into another language and when it's reformatted for some random feed reader. I've learnt that I have a large following in Russia with people using Yandex Translate to read my blog.

Although CloudFlare itself has only been around for a short time, it has grown enormously and now does almost 35 billion page views a month. If it were a single site it would be one of the largest in the world. That means that CloudFlare's bot and scraper intelligence is growing and improving constantly. As bots enter the CloudFlare network of sites they are detected, blocked and dissected.

The combination of 8 years of deep web intelligence, smart technology for detecting scraping and enormous scale means that CloudFlare's anti-scraping solution ScrapeShield is powerful, continuously improving and ready for prime time.

No other anti-scraping service has the long history or enormous scale that CloudFlare brings to ScrapeShield.

Of course, ScrapeShield is a free app from CloudFlare and works with other CloudFlare services such as SSL protection, the ability to hide your domain's real IP address completely, our acceleration technologies such as compression, minification, caching and global distribution, and our core security that keeps hackers at bay.

CloudFlare's New UI: Managing DNS records

Damon Billian — Wed, 11 Apr 2012 00:33:00 -0700

As mentioned in a previous post about CloudFlare's New UI, we have released a number of changes to the site to make things easier to find and locate. Since making changes to existing DNS records and adding new ones are vitally important to site owners, we thought it would be helpful to craft a blog post showing the changes to the DNS editor.

How to get to DNS Settings

Easily turning CloudFlare on and off for a DNS record

We have further clarified what an orange cloud and gray cloud mean in your CloudFlare DNS zone file. While users have always been able to turn CloudFlare on or off on a record, many of them were a little confused on what this actually did and didn't know how to do it. If you want to exclude a certain subdomain from CloudFlare's proxy and cache, simply click on the orange cloud to move it to a gray record.

How to edit an existing DNS record

If you need to edit an existing DNS record, such as if you need to change your server IP, you can simply edit the records to point go the new server IP address. The right side of the page shows a gear icon that opens up the editing options.

Clicking on the edit record option let's you modify the fields, then you would save the record when you're done making the change.

How to add a new DNS record

If you need to add a new DNS record, you can do this by scrolling to the bottom of the page for DNS settings. You would then choose the record type and add the appropriate values for that record type.

Subdomains not working on CloudFlare

If a subdomain on your site isn't working, the most common issue is that you didn't add the subdomain to your zone file when you configured the domain. If you are missing a subdomain called "blog", which translates to blog.yourdomain.com, then you simply need to add the missing record as an A record or CNAME.

A record has to point to a server IP address:

CNAME records can be entered as an alias:

Mail not working

If your mail isn't working for your domain, then the issue is you don't have your mail or MX records configured properly in your zone file (or they are missing). You need to add the records in your zone file and make sure that they are presented in the correct format.

Since configuring mail and MX records is a frequent contact for CloudFlare support, we thought that highlighing Google Apps email record entries would be a good one to show visually.

The first thing you would want to make sure is added for Google Apps mail is a CNAME record that points to ghs.google.com.

You would then want to make sure that the records for Google's mail servers are added in your zone field in the format below (note: you will have more than the three entries shown).

Note: Please make sure all mail records have a gray cloud next to them. If you add the record and it creates an orange cloud, please click on the cloud to move it to gray.

Have a suggestion about improving the DNS editor? Please feel free to contact CloudFlare with any suggestions. You can also send us a quick Tweet or message on Facebook.

Non-Latin/UTF8 International Domains (IDNs) Now Fully Supported

Matthew Prince — Tue, 10 Apr 2012 01:03:00 -0700

One of the strangest questions I get when talking about CloudFlare is: "How are you ever going to expand your customer base beyond Silicon Valley?" The reality is that while wandering San Francisco in a CloudFlare shirt gets me an occasional high-five, I've run into almost as many users abroad as I have at home.

The United States remains CloudFlare's largest source of traffic, but China is a rapidly expanding second, Brazil third, Turkey fourth, and the Great Britain fifth. We run 14 data centers (Amsterdam is our busiest), in 8 countries, and on 3 continents. We have a Costa Rican subsidiary, in preparation for our expansion into Latin America, and are setting up a Seychelles subsidiary, in preparation for our expansion into Africa. In other words, we are already a very international company.

The Web's Great Embarrassment

The web wasn't originally setup to support non-Latin alphabets. If your language used characters not represented in ASCII, up until surprisingly recently you were out of luck registering a domain. People began talking about this problem in the 1990s, but it wasn't until 2000 that .com and .net began supporting International Domain Names (IDNs). While these top level domains (TLDs) supported IDNs, browsers were slow to roll out IDNs with support only becoming wide-spread in the last 6 years. If you wanted a top level domain with a non-Latin character, it wasn't until 2010 that ICANN approved the first set.

Today, most DNS interfaces still don't support IDNs. Holders of IDNs need to convert their domains to what is known as Punycode in order to add them to most DNS. Punycodes are ASCII representations of domain names (e.g., xn--camtasia-5x3qu96nkem.com to represent camtasia教程网.com). They're a useful but ugly hack to make the Internet work on a system that never envisioned the global diversity and ubiquity it has obtained.

IDN Support, Now Standard

CloudFlare has supported Punycodes for our DNS from the beginning, but, as I said, that's an ugly hack. I'm happy to announce that, as of today, we now support IDNs directly in our interface. If you've previously entered your domain using a Punycode, you should now see your domain displayed correctly in its native characters. And if you enter a domain in non-Latin characters, we handle all the backend conversion to make it work gracefully with the global DNS infrastructure.

We've also added more support throughout our UI for non-Latin characters. In the past, we were overly restrictive on requiring Latin characters in many of the forms on our site. We've upgraded our UI site wide to add support for the whole UTF8 character set.

CloudFlare is already a global company, and I'm proud that we're now more fully supporting the world's languages and character sets. In other words, if you're looking for a DNS provider for an International domain name, you're welcome here.

Welcome DreamHost!

Matthew Prince — Thu, 05 Apr 2012 15:01:00 -0700

We hung out with the DreamHost team for the first time at HostingCon in August 2011. They threw a great party, had awesome t-shirts, and exuded the kind of excitement and passion for hosting that CloudFlare has for making websites faster and more secure. We immediately knew we wanted to partner with them.

Fast forward nine months to today and we are happy to announce that DreamHost is now an official Certified Hosting Provider. Beginning today they're offering CloudFlare to all their customers with a one-click-simple integration. Prior to this partnership, we had thousands of DreamHost customers who signed up for CloudFlare directly through our site. Now, every DreamHost customer has simple, easy access to CloudFlare with a click of a button and without having to mess with their DNS. Other bells and whistles like mod_cloudflare are now included in all the default DreamHost configurations, so even existing CloudFlare users on the DreamHost network will benefit.

CloudFlare Plus

DreamHost is the latest CloudFlare Certified Hosting Partner, a program that makes CloudFlare one-click simple for any hosts to provide to their customers. We're trying a new experiment and allowing DreamHost to offer a special plan they've dubbed CloudFlare Plus. We worked with them to create this custom CloudFlare plan with the features they thought would be the most interesting for their customers and a price point lower than our current Pro product. For those folks for whom CloudFlare Pro is a bit more than they need, DreamHost now offers another option with some of our most popular paid features.

Just Sayin': CloudFlare ≈ Voltron

I do have to say that DreamHost will also always hold a special place in my heart for what must be the most fun press release I've ever seen, a full copy of which is below. If you only read one part, read the quote near the end that I bolded which is certifiably awesome.

FOR IMMEDIATE RELEASE

DreamHost Partners With CloudFlare

CloudFlare to Provide Free Site Performance Optimization and Security Services for all DreamHost customers

LOS ANGELES, California—April 5th, 2012—DreamHost, a global full-service web hosting company, has today announced a partnership with leading Internet web performance and security company, CloudFlare. DreamHost customers now have immediate access to CloudFlare's robust infrastructure at little or no cost as a standard feature of their hosting plans.

Shared web hosting customers, for many years, have rarely been able to take advantage of the benefits provided by Content Delivery Networks as a result of either the cost or complexity involved. CloudFlare has removed both barriers to entry by distilling the entire setup and configuration process down to a single checkbox and removing the cost entirely. CloudFlare brings the performance and security tools previously available only to Internet giants to anyone with a website.

Hundreds of nodes around the globe power CloudFlare's network ensuring that websites load quickly and consistently, regardless of where in the world users happen to be. CloudFlare's Anycast technology works with static and dynamic sites, routing users to the node on their network for the fastest performance — all without breaking a sweat!

CloudFlare's "Always Online" technology ensures that sites taking advantage of the CloudFlare platform will remain online, continuing to serve cached content, even if the hosting servers on which they are housed become temporarily unreachable. If the hosting industry had a holy grail, it might look a little something like CloudFlare.

In addition to CloudFlare's free offering, DreamHost has also worked with the CloudFlare team to create "CloudFlare Plus," a bundling of CloudFlare's most popular features available exclusively to DreamHost customers. CloudFlare Plus is an optional paid upgrade, weighing in at $9.95 per month, and adds automatic image optimization and support for Secure Socket Layer (SSL) connections. Provisioning of either option has been integrated within the DreamHost customer control panel.

"When we first met with the CloudFlare team at HostingCon 2011 we had no idea if what they were telling us was true," said Kathy Brahm, DreamHost's Vice President of Customer Experience and Partnerships. "You know how typical sales people can be — schmoozy, smiley, 'Let-me-buy-you-dinnery', handsy … all the while making outrageous claims about their product. CloudFlare's team has been the complete opposite of those — and a true pleasure to deal with. We've spent the past few weeks putting CloudFlare through its paces and running some tests of our own. Some of us nearly fainted when the first speed tests came back. One guy cried. Me? I buried my feelings deep inside so I don't have to deal with them. It's just what I do."

"From days of long ago, from uncharted regions of the web, comes a legend; the legend of CloudFlare, Defender of the Interwebs: a mighty robot, loved by good, feared by evil," explains Matthew Prince, co-founder and CEO of CloudFlare, doing his best Peter Cullen impersonation. "As CloudFlare's legend grew, peace settled across the network. From Los Angeles, an Interweb Alliance was led by DreamHost. Together with other good hosts of the network, DreamHost helped maintain peace throughout the Interwebs, until a new horrible menace threatened. A closer relationship with CloudFlare was needed. This is the story of the super force of web explorers, specially trained by DreamHost, to more tightly integrate CloudFlare, Defender of the Interwebs!"

CloudFlare's free offering and the DreamHost-exclusive "CloudFlare Plus" are now available to all DreamHost customers.

Check out CloudFlare's New UI

Damon Billian — Sat, 31 Mar 2012 20:48:00 -0700

CloudFlare has been beta testing a new account interface for several months with many of our followers on Facebook and Twitter. This week, we rolled out the new interface to all CloudFlare users. Since a picture is worth a thousand words, I'm going to cover many of the key changes in a visual walkthrough.

Note: The CloudFlare 'DNS Settings' page has also undergone some changes. Since the changes to that page are worthy of a separate post, I'll do an additional post about DNS settings in the near future.

Add multiple sites at once

Site owners can now add multiple domains at once. To add multiple websites, list them in the My websites input field separated by commas.

Easier Navigation to CloudFlare Reporting and Threat Control

We've made it easier to find the most popular pages, including:

'Dashboards' has been replaced with 'Analytics' and 'Threat Control' to save users from having to do an additional click to get to the CloudFlare Reporting and Threat Control Dashboards.
Development Mode, one of our most widely used features, can now be turned on directly from they My Websites pag, by choosing the gear icon
'Deactivate' has been renamed 'Pause CloudFlare'. We hope the new terminology helps customers understand how to temporarily pause CloudFlare in the event of an issue.

Improved Pages for Settings

The new 'Settings' page makes it easier to find the setting you're looking for by classifying them with different tabs.

Settings Overview

The 'Settings Overview' stores the most frequently used CloudFlare settings so that site owners can access these quickly. You can also quickly make basic changes to your Security and Performance settings on this page. The key settings that you will find here are:

Development Mode and Cache Purge
A SSL drop down menu for turning on SSL options for paid accounts
CloudFlare's IPv6 Gateway

Security Settings

CloudFlare offers a number of security features to protect websites. All of the key CloudFlare security settings have been grouped into a single page, including:

Basic Security Level
Customize Challenge Page
Web Application Firewall

Performance Settings

CloudFlare's CDN-like and Web Content Optimization features are grouped on this page. If you want to adjust your CloudFlare caching level, or if you want to turn Auto Minify or Rocket Loader on or off, then this is where you would want to go.

We hope you like the changes to the UI. As always, please contact CloudFlare with any questions or feedback about the new UI.

App: Observe Earth Hour on Your Website

John Roberts — Fri, 30 Mar 2012 19:55:00 -0700

"It would be great if..."

This app started with a casual request on Tuesday, March 27th, in a public CloudFlare support thread.

I think it would be great if CloudFlare had an app to aid in Earth Hour this upcoming Saturday, March 31 @ 8:30 PM. It's not too late to join in on the effort!

Time was too short for CloudFlare to help, based on other projects. But — just in case — I mentioned the alpha version of our developer platform.

Katie G., a developer in the UK, warned that she was busy with lots of work and other projects, but ended with:

I'll still try to do it though in time for Earth Hour.

Underpromise, Overdeliver

Last night — about 48 hours later — Earth Hour was made available to all CloudFlare customers as a free CloudFlare app!

Turn on the app, and for one hour on March 31 (today, for some; tomorrow for many) your site will go dark and present a message to the site visitor explaining Earth Hour. The app offers site owners the chance to "Save the world, 60 minutes every 365 days at a time."

We applaud Katie for her quick work, and her questions during development have helped us improve the platform already.

What Can You Do On CloudFlare?

More info coming soon about this developer platform. But the impatient can follow links in this thread. If you'd like to develop an app, let us know. Developers can build globally-deployable apps for CloudFlare-powered sites, with no operation costs, to join the existing CloudFlare apps. That's just the beginning.

Don't Forget

It's easy to observe Earth Hour, no matter your time zone.

Introducing ScrapeShield: Discover, Defend & Deter Content Scraping

Matthew Prince — Thu, 29 Mar 2012 18:30:00 -0700

If you're a publisher, whether an individual blogger or major media outlet, you've undoubtedly experienced content scraping. Searching the web for an article you've published or other original content you've created and you find it copied and republished on some other random website. Often the site will be full of ads. And, sometimes, it will even rank higher in search results than your original work.

While you may envision an army of individuals copying and pasting your content on their sites, the truth is content scraping is typically an automated process with bots that grab original content and then republish it without human intervention onto link farm sites. CloudFlare has blocked many of these bots automatically in the past, but we decided it was time to do something to more actively stop them.

Introducing ScrapeShield

ScrapeShield is an app created by the CloudFlare team. It incorporates several existing CloudFlare features like email obfuscation and hotlink protection that serve to protect from content scraping and adds a number of new features as well. Because we believe every publisher of original content should be able to understand and control how their work is used, we're providing ScrapeShield free for every CloudFlare user.

Detect, Defend & Deter

ScrapeShield has different elements to help you detect when your content is scraped, defend your site against content scrapers, and even deter content scrapers from targeting you in the first place. If you enable ScrapeShield, CloudFlare will automatically insert invisible tracking beacons in your content. When automated bots scrape your content, they pull the beacons along with them. CloudFlare detects these beacons when they ping from sites that aren't your own. You can access your ScrapeShield control panel to see where your content is being republished. Not only is this useful in showing scraping, but you can also see users who are reading your content through proxy services like Flipboard or Pulse.

The data from the content beacons is fed back into CloudFlare's protection system. As CloudFlare identifies content scraping bots, we automatically prevent them from accessing your site. Just like Project Honey Pot, the original inspiration for CloudFlare, used traps to detect when spammers were harvesting email addresses, CloudFlare now uses data from ScrapeShield to identify content scrapers and keep them off publishers' sites.

Maze

We didn't want to just stop scrapers from attacking sites on CloudFlare, we also wanted to tie up their resources so they couldn't harm the rest of the web. To do this, we created Maze. Maze routes known content scrapers who are visiting ScrapeShield-protected sites into a virtual labyrinth of gibirish and gobbledygook. We dynamically throttle the bandwidth and speed so instead of the pages loading as fast as possible, the connection is held open to the scrapers and their resources are tied up.

We use excess resources on the CloudFlare network to generate Maze, and it doesn't consume any of our publishers' resources or add any additional load to their sites. What's beautiful about the system is that the only way that content scrapers can be sure they're avoiding Maze is to avoid CloudFlare's IP addresses entirely. For any content scrapers who may be reading this, here's a helpful list of all of our IPs so you can make sure to stay away.

No Pinning

Finally, with the rise of sites like Pinterest, innocent content scraping may become even more prolific. While many sites welcome their images being pinned, we wanted to make it easy to opt out. ScrapeShield includes an option to add the no-pinning meta tag to your site to prevent your images from being pinned to the site. As other similar services include a mechanism to opt out, expect that we'll include an easy way for you to do so right from the ScrapeShield interface.

The health of the web depends on publishers creating original content getting credit for their creations. CloudFlare is committed to building a better web and we're extremely excited about ScrapeShield as a new tool to help publishers do exactly that.

CloudFlare is Now Part of the Hong Kong Internet Exchange (HKIX)

Tom Paseka — Wed, 28 Mar 2012 19:14:00 -0700

CloudFlare is now connected to HKIX (Hong Kong Internet Exchange). HKIX is the largest Internet Exchange in Asia transferring around 200Gbit to its 160 ISP, Carrier and Content networks. HKIX is fundamental to the local Hong Kong ISP Market, with every provider in Hong Kong connected, as well as excellent regional coverage with networks from Thailand to Japan to Australia.

So, what does a connection to HKIX mean for CloudFlare users? Faster performance for your Hong Kong web surfers. Being a part of the HKIX allows us to deliver traffic to all Hong Kong web surfers within Hong Kong. This will improve web browsing performance by keeping the traffic local and the latency low.

Connecting to HKIX is one of the steps CloudFlare is working on to bring the Internet closer to you. Watch for more news to come over 2012!

App: Zoompf Performance Report Helps You Find And Fix Performance Issues

Kristin Tarr — Wed, 21 Mar 2012 14:53:00 -0700

Zoompf is a cloud-based tool for on-demand web performance testing and optimization. We're pleased to be able to offer their robust Zoompf Performance Report as a CloudFlare App for a one-time fee of just $5.

How It Works

Zoompf Performance Report is super simple. Once you purchase the report, Zoompf crawls and analyzes your website for over 100 performance issues. Within minutes, Zoompf generates a rich PDF report and emails you a download link. There is nothing to install or configure.

Value of the Zoompf Performance Report

Zoompf provides a superior performance analysis over free tools in 3 key ways: breadth, depth, and detail.

First, Zoompf scans large portions of your site, not just a single page. This ensures that all your pages are optimized.

Next, Zoompf tests for over 100 different performance issues from their comprehensive database of over 400. Free tools test for only a dozen or so issues.

Finally, while a free tool might provide a sentences or two description about a performance issue, Zoompf provides a wealth of information including a summary, easy-to-fix ratings, and detailed remediation information -- all with accompanying diagrams and code snippets.

Testing more of your website, for more issues, and providing more information about how to be faster makes Zoompf the best front-end performance analysis tool available today.

Results You Can Use

Zoompf Performance Report helps you find and fix performance issues to drop seconds off page load times, improve user satisfaction, drive additional revenue, and reduce operational costs. Get your report today!

The CloudFlare Story

Kristin Tarr — Tue, 20 Mar 2012 21:56:00 -0700

We are frequently asked by customers, media, friends and peers the story of how CloudFlare started. Next week, CloudFlare co-founders Matthew Prince, Michelle Zatlyn, and Lee Holloway will tell this story first hand at the Harvard Club of San Francisco's Founders Series.

The talk will be led by Professor Noam Wasserman, the award-winning Entrepreneurship Professor at HBS and author of the upcoming book The Founder's Dilemmas: Anticipating and Avoiding the Pitfalls That Can Sink a Startup. He will interview Matthew, Michelle and Lee on their experiences and lessons learned on starting CloudFlare.

Anyone is welcome to attend, so if you're in the Bay Area and are interested in hearing how CloudFlare came to be, we welcome you to join us! The talk will be taking place on Tuesday, March 27 at 6:30PM in SOMA (corner of Mission and Fremont). More info can be found here: http://www.harvardclubsf.org/article.html?aid=400

We hope to see you there!

Launching at SXSW? Use CloudFlare to Stay Online!

Matthew Prince — Fri, 09 Mar 2012 23:06:00 -0800

A bunch of us from CloudFlare are packing up to head to Austin over the next 24 hours for SXSW, the interactive, music, and film festival. We've been supporters of SXSW for a while and helped the festival's panel picker stay online under the crushing load of people registering ideas to present over the coming weeks. Everyone who has visited the SXSW site to choose what to do at the festival has passed through CloudFlare's network and the site has been faster and safer as a result.

You Can't Build Buzz If Your Site's Offline

It's only fitting, then, that several of the startups launching at SXSW are using CloudFlare to ensure they stay online. One of the fastest ways to kill buzz is to have your site be unavailable. CloudFlare helps ensure that, even if you're a new startup without a ton of resources, you can have an infrastructure equivalent to an Internet giant.

Start It Up

A few of the startups launching at SXSW who are using CloudFlare have contacted us to give us a sneak peak at what they're working on. Some of them are pretty amazing and it will be interesting if the next Twitter or Foursquare emerges this year. I won't steal any of their thunder, but we're excited to help them with their infrastructure so they can stay online so they can focus on building buzz.

If you're launching at SXSW and you haven't done so yet, there's still time to signup for CloudFlare. It takes 5 minutes and even our free plan will do wonders to make sure your service stays online. Finally, make sure you look for us at the festival. We love startups and are always excited to hear about what the tech community is working on.

CloudFlare Tips: Recommended Steps for New Users

Damon Billian — Fri, 09 Mar 2012 19:12:00 -0800

Editor's Note: Since the post has the word "steps" in it, I thought I would include a picture of some steps from a visit to Mother Temple of Besakih in Bali, Indonesia. I had the good fortune of visiting Bali in 2011, and the most amazing thing about the temple is that it is built on the slopes of an active volcano.

CloudFlare's core value is that we speed up and protect any website. For new and current users, we wanted to pull together a handy guide that helps you get the most out of CloudFlare.

Recommended Steps

1. Best way to Resolve an Issue - 'Pause'
When you sign up for CloudFlare, the service should just work. However, if you notice a problem after activating CloudFlare, do not change nameservers away from CloudFlare. Instead, temporarily deactivate CloudFlare by choosing the 'pause' or 'deactivate' option on your CloudFlare My Websites Page:

Settings->Pause/Deactivate CloudFlare

Changing name servers away from CloudFlare makes it difficult to troubleshoot. If you 'pause' the service, the issue will likely be immediately resolved. Then, contact our technical support team so we can work with you to resolve the root cause.

Also see: Troubleshooting CloudFlare Problems

2. Preserving IP information --> Install mod_cloudflare (or equivalent)

Since CloudFlare acts as a reverse proxy for websites, CloudFlare's IPs are going to show in your server logs. There is an easy fix to restore original visitor IP for any web server.

If you have issues with things like GeoIP or .htaccess blocks not working properly on your site, installing mod_cloudflare will resolve the problem immediately.

You should also whitelist all of CloudFlare's IP addresses with your hosting provider and on your server.

Note: This is not required if you have activated CloudFlare through a CloudFlare Certified Partner.

3. Create PageRules

PageRules gives you more performance and configuration options, including:

Cache HTML (our default configuration only caches these static content files)
Exclude certain URLs from CloudFlare features and caching
302 redirects

We recommend that you use PageRules to exclude the admin section of your website from CloudFlare's services. This will ensure you don't see any error messages or have issues updating plugins. The PageRule you will need to create has the following structure for WordPress:

*mydomain.com/wp-admin

4. Familiarize yourself with the CloudFlare settings

CloudFlare offers a number of optional security and performance options that can be turned on or off on your "Settings" page:

My Websites->Settings->CloudFlare settings

Some of the key features that we'd like to highlight:

IPv6 Gateway

Make your website IPv6 compatible without having to purchase expensive hardware. CloudFlare's IPv6 Gateway is available to all customers free of charge.

Web Content Optimization Features

Rocket Loader and Auto Minify are free beta features that will help speed up your website. What they do:

Rocket Loader:If you have ads, widgets or plugins on your website, Rocket Loader will speed up the delivery of your pages by automatically asynchronously loading your JavaScript resources.

Auto Minify: Removes unnecessary characters from HTML, CSS, and JavaScript to save on file size.

Note: These features are still in beta. We encourage you to try them but if you see an issue, then just turn them off. Please report the bugs to CloudFlare so our magical engineers can work on fixes.

SSL

SSL support can be added to any website on CloudFlare on a paid account.

Paid plans offer additional security and performance features than free accounts. You can see the differences in our comparison chart.

5. Check out the CloudFlare App Store

CloudFlare has partnered with a number of popular web applications to make adding these apps easy. You can get access to any of these services without touching your code or worrying about if they will interfere with another service on your site. The most popular services include:

Google Analytics : Install Google Analytics on all of your pages
VigLink: Make money from the content on your website or blog
CodeGuard: Back up your website content
Highlight: Add contextual search with one click to your site (developed by CloudFlare)

6. Search our Knowledge Base for FAQs

While we love answering questions from our customers as quickly as possible, many questions can be answered by doing a quick search through CloudFlare's help content.

7. Follow us on Facebook and Twitter

We use Facebook and Twitter as our primary communication channels for announcing new product releases, known system issues and to give users early access to beta features. If you want to know as much as you can about CloudFlare, or if you only have a really quick question to ask, Facebook and Twitter are the best places to go.

We also recently launched a Google+ page.

If you ever have a question, please contact us. We read every email that we get and love to hear from our users.

Come Meet Team CloudFlare at SXSW

Matthew Prince — Fri, 09 Mar 2012 02:27:00 -0800

CloudFlare has been a big supporter of SXSW. This summer, when the Interactive, Film, and Music festival opened its Panel Picker for submissions, CloudFlare helped keep it online under the flood of ensuing traffic. A bunch of folks from team CloudFlare are headed to Austin. If you'll be there too, it'd be great to meet up. Shoot us a tweet (@cloudflare) or stop by one of the sessions our team will be participating in. Here are the four panels and talks our team is participating on.

Javascript Performance MythBusters™(SXSW Details)

JavaScript is everywhere from mobile phones and tablets to e-readers and TVs. With such a wide range of supported environments developers are often looking for an easy way to compare the performance between snippets, browsers, and devices. jsPerf.com, a site for community driven JavaScript benchmarks, was created to help devs do just that.

Chris Joel (@robodynamo) from CloudFlare is joined by Mathias Bynens and John-David Dalton from jsPerf.com, and Lindsey Simon from Google/Browserscope in this panel discussion on some of the best dev-created benchmarks and most interesting practices debunked by real-world tests.

Scaling to Infinity: Dealing with Rocket Growth (SXSW Details)

It's every engineers dream to work on a company that becomes one of the top 10 largest sites online. This group of panelists have done just that. From Twitter to Facebook to Yahoo! - these engineers, tech operations, and cloud computing specialists have seen it all and will discuss what it's like to scale the largest infrastructures imaginable. This session will discuss what it takes for a team of engineers to scale to an Internet Giant, how it's done, and what the best practices are for making it happen.

Lee Holloway (@icqheretic), co-founder of CloudFlare, is joined by a powerhouse panel including Andrew Terng from Tumblr, Girish Patangay from Facebook, and Jeremy LaTrasse of Message Bus/Twitter. The panel is moderated by Jeremy Edberg, the former CEO of Reddit now responsible for cloud reliability at Netflix.

Anything You Can Do, I Can Do Backwards in Heels (SXSW Details)

Ginger Rogers may have said it, but today's female entrepreneurs are proving it. Right now, women are starting and leading new and innovative companies at an unprecedented rate. From e-commerce to healthcare to Internet infrastructure, women are breaking new ground across all industries these days. But, why now? What are today's female entrepreneurs doing differently to build sustainable businesses and get the attention and credit they deserve? What unique struggles do they still contend with and what advice can they share with tomorrow's generation of female leaders? These questions and more will be addressed in this entertaining panel.

Michelle Zatlyn (@zatlyn), co-founder of CloudFlare, is on a panel moderated by All Things Digital executive editor Kara Swisher and joined by Piya Sorcar, founder of TeachAIDS, and Victoria Ransom, founder of Wildfire Interactive.

Surviving Lulz: Behind the Scenes of LulzSec (SXSW Details)

On Thursday, June 2, 2011, LulzSecurity.com registered for CloudFlare. One hour after they registered, they published 3.5 million usernames and passwords allegedly stolen from Sony Pictures' website. For the next three weeks, LulzSec claimed to hack organizations ranging from the CIA, to the US Senate, to the Arizona Immigration Police. In the meantime, law enforcement, cyber vigilantes, and rival hackers worked to unmask LulzSec and launch major attacks of their own to knock LulzSecurity.com offline. CloudFlare watched it all from the heart of the crossfire.

I (@eastdakota) asked and received permission from LulzSec to tell exactly what it's like to be one of the most notorious hacking groups of all time and how to keep your site online when the whole world is trying to shut you down.

CloudFlare Now Supporting More Ports

Matthew Prince — Thu, 01 Mar 2012 05:15:00 -0800

CloudFlare protects and accelerates web traffic. As a result, we initially only proxied traffic for the two main web ports: 80 (HTTP) and 443 (HTTPS). One of the top customer service questions we receive is: "Why did my control panel stop working after I signed up?" The answer is that most control panels run on a non-standard web port that we don't proxy. As a result, if you try and connect to cPanel-like control panels through CloudFlare then your traffic will get blocked. Not a great first experience.

Access Control

The solution has always been to access the control panel via the IP address or a subdomain setup to route around CloudFlare's proxy. That works great, but it still requires an explanation and therefore increases the CloudFlare learning curve. We're always looking for ways to make CloudFlare easier. A few weeks ago we began supporting other standard ports used by web control panels. In addition to 80 and 443, the list of supported ports now includes:

2052
2053
2082
2083
2086
2087
2095
2096
8080
8443
8880

This covers most the web major control panels. While we will now proxy traffic through these ports, we won't cache static content or perform any performance or app transformations on requests/responses that flow through them. If you don't use these, we'll also soon provide a method to easily shut down these ports at the CloudFlare level.

FTP, SSH, and Non-Web Protocols

Reading this you may wonder why we can't open ports like 20, 21, 22 and 23 to support protocols like FTP, SSH, Telnet, etc. Unfortunately, while this is an often-requested feature, the protocols don't support it. We know where to send traffic after it connects to CloudFlare's network based on a HOST header in web requests. Non-web protocols like the above don't include a HOST header. As a result, for these protocols we see the traffic connecting to our network and have no way to route it to the origin.

This means that you'll continue to need to SSH and FTP into your server using an IP address or a subdomain you mark as being CloudFlare disabled on your DNS manager (we setup "direct" by default, but you can change it for better security). While this may seem like an inconvenience, there is an upside. By not directly exposing your origin server to traffic over these ports, we add an additional layer of security.

We also monitor all the connections from SSH and other protocol scanners that regularly try to "dictionary attack" logins. We feed this data back into our system in order to better protect from attacks. In other words, while there may be a bit of a learning curve to using SSH or FTP after signing up for CloudFlare, having those protocols blocked by our network means the CloudFare system is always learning.

How to Enable SSL on Tumblr, WordPress, Blogger, AppEngine, Posterous & More...

Matthew Prince — Mon, 27 Feb 2012 01:57:00 -0800

The number of services today that allow you to quickly get online is stunning. Tumblr, WordPress, Blogger, AppEngine, Posterous, Ning, TypePad, Smugmug, and more allow anyone to publish content online. Unfortunately, most don't support SSL. This means that, if you access one of these platforms from a shared network connection, like a coffee shop or airport wifi, someone can use a tool like Firesheep to sniff your session cookie information and gain access to your account even without your password.

While the risk of this for most people isn't very high, our customer support team has been getting more and more inquiries about whether CloudFlare can add SSL encryption to these third party platforms. Justin on our team wanted to find out how easy it was so he used his own Tumblr blog (justinpaine.com) to find out.

Setup a Custom Domain

The first step for any of these platforms is to setup a custom domain. Tumblr, for example, defaults sites to using a subdomain of tumblr.com. While we're working with platforms to allow you to add CloudFlare support when you're on a subdomain account, until the platforms explicitly allow it you'll need to setup your own domain. You can find instructions on how to do so for each of the platforms mentioned already through the following links: Tumblr, WordPress, Blogger, AppEngine, Posterous, Ning, TypePad, and Smugmug.

Adding CloudFlare

Once you have a custom domain for your site, you then need to add the site to CloudFlare. If you don't already have a CloudFlare account, you can create one. After you create a new account, or login to your existing account, you can walk through the quick process of adding your site. The process takes four steps and about five minutes to complete.

To get SSL on your site, you'll need to select one of the paid CloudFlare plans. All paid plans come with SSL support automatically. You don't need to buy or configure a SSL certificate seperately, we take care of that process for you automatically. As soon as you've finished configuring CloudFlare, we automatically add the SSL certificate and activate it for your account.

Flexible SSL

CloudFlare automatically detects that your platform provider doesn't support SSL and defaults you to the Flexible SSL setup. This means that connections from a browser to CloudFlare will be encrypted via HTTPS, but connections from CloudFlare to the platform will pass over unencrypted HTTP.

While it is ideal to have an end-to-end HTTPS connection, securing the connection from the browser to CloudFlare mitigates 99% of the real risk. A way to think about it is if you're worried about the government monitoring your web traffic, Flexible SSL won't offer a complete solution. On the other hand, if you're worried about someone next to you in the coffee shop sniffing your cookie or password information, CloudFlare's Flexible SSL will protect you.

If your platform provider ever begins to support SSL themselves, you can switch CloudFlare to Full SSL mode at any point from the CloudFlare Settings page and have end-to-end encryption. If you think about it, Flexible SSL is a lot like CloudFlare's IPv6/IPv4 Gateway. In that case, we were translating between IPv6 and IPv4 networks seamlessly. In this case, we're translating between the HTTPS and HTTP protocols, also seamlessly.

Surf Securely

That's all there is to it. Once the DNS propogates, you'll be able to connect to your site securely just by entering HTTPS rather than HTTP. You can also use PageRules to force all connections to HTTPS if you'd like to default to an encrypted connection. Check out Justin's Tumblr:

https://justinpaine.com

It's one of the only encrypted Tumblr sites and it didn't require him changing anything in his Tumblr settings, just signing up for CloudFlare. Pretty cool!

One last note: we had previously supported another method for SSL on Google's AppEngine. That method, which relied on domain masking, proved brittle and unreliable so it was depricated for now. This new method using Flexible SSL is 100% reliable and has the advantage of supporting a number of platforms beyond Google's AppEngine. We'll be bringing back domain masking enabled via PageRules in the future since it does have some beneficial uses.

Post Mortem: The Ugly, the Bad & the Good

Matthew Prince — Fri, 24 Feb 2012 16:08:00 -0800

Last night was not our finest hour. Around 07:30 GMT, we were finishing up a push of a new DNS infrastructure. The core of what this new update was built to do is make DNS updates even faster. Before it took about a minute for a change to your DNS settings to propagate to all our infrastructure, with the new DNS update it is almost instant. That is important to understand in order to understand what went wrong.

Making an update to the DNS requires changing underlying code deep in our system and taking servers offline while we do so. We scheduled the update for the quietest time on our network, which is around 07:00 GMT (around 11:00pm in San Francisco). The code had been running smoothly in our test environment and one data center for the last week so we were feeling pretty good. And, in fact, the push of the DNS update went smoothly and was ahead of schedule.

The Ugly

When the update was complete in 10 of our 14 data centers we got word of a minor issue that was affecting some data getting pushed from the master DNS database. In the process of diagnosing the minor issue, the master DNS database was deleted. The new DNS system did its job and rapidly propagated across the 10 datacenters where the update was live. The result was that if recursive DNS looked up a domain and hit one of those 10 datacenters, around 07:30 GMT they would receive an invalid result. That meant those sites went offline and it was entirely our fault.

The Bad

The DNS database is regularly backed up, but it took us about 5 minutes to recognize the issue, retrieve the backup, and push it to production. Our new DNS infrastructure pushed the update out to most of the datacenters immediately, but because it was such a large update it took a few minutes to rebuild. In most places, new DNS requests were correctly answered with less than a 10 minute window of bad results.

Unfortunately, DNS is a series of interconnected caches, many of which are not in our control. If you accessed a page during the issue, your ISP's recursive DNS likely cached the result. Since most DNS providers don't make it easy to flush their cache (compared with a recursive provider like OpenDNS, which does) it extended the outage for people who were already seeing an issue. Generally, within 30 minutes, recursive DNS had flushed and by 8:00 GMT sites were back online.

Two datacenters did not take all the corrected DNS file updates correctly. We are still investigating why, but our speculation is that because the update affected a large number of records the systems choked on the initial attempt at the updates. Requests that hit those data centers returned bad results for some sites until about 8:10 GMT. Some visitors in Europe and Asia would have seen a longer period of downtime on some sites as a result. Our system has multiple layers of redundancy, including at the datacenter level, so we removed the two data centers from rotation as soon as we recognized the issue and affected visitors once again saw correct DNS results.

Two last problems exacerbated things. First, as is normal operations for us, we were dealing with two mid-sized DDoS attacks directed at some of our customers at the time. Nothing abnormal about that, but having two fewer data centers in rotation made us less effective at stopping them and caused a small handful of 500 errors. The impact of those, however, was minimal (less than 0.001% of traffic for around a 12 minute period). Second, there were some DNS entries in our system for TLDs like co.nz that shouldn't have been there. While it wasn't a validated DNS zone record, the way that the DNS update was pushed caused a handful of records that fell under these TLDs to also see an extended outage. When we got reports of this we identified the issue and removed the problematic entries.

The Good

There's not a ton of good in this incident itself. While the system status is green now, we will memorialize the incident on our system status page. I, along with the rest of the team, apologize for the problem and anyone who experienced it. We've built a system that is resilient to most attacks, but a mistake on our part can still cause a significant issue. This is the second significant period of downtime we've had network wide. The first was more than a year ago and also occurred due to an error we made ourselves. Any period of downtime is unacceptable to us and, again, we sincerely apologize.

Going forward, we've already added several layers of safeguards to prevent this, or a similar incident, from occurring. CloudFlare's technical systems are designed to learn over time, that same ethos is in our team itself. While this incident was ugly, I was proud to see almost the entire engineering, ops, and support teams online into the wee hours helping customers sort out issues and building the safeguards to prevent an issue like this in the future.

What I was planning on writing a blog post about this morning is our new DNS infrastructure, so I will end with a bit more detail on that. As described above, one of the main benefits is that DNS updates are even faster than before. In the past, DNS files were replicated every minute or so. Now changes are pushed instantly to our entire network. While that wasn't a great thing last night, in general we believe it is a big benefit to our publishers and makes us the fastest updating global authoritative DNS in the world.

The update to the DNS systems also includes hardening against some of the new breed of DNS-directed DDoS attacks we've begun to see. Going forward, this will help us provide even better protection against larger and larger attacks. Our goal is to stay ahead of the bad guys and ensure that everyone on CloudFlare has state-of-the-art protection against attacks.

I apologize again for those of you who experienced downtime as a result of our mistake. We will learn from it and continue to build redundancy and resiliency into CloudFlare in order to earn your trust.

Social Media and Community Management Meetup

Kristin Tarr — Fri, 24 Feb 2012 00:55:00 -0800

We're excited to announce that we are hosting a Social Media and Community Management meetup at the CloudFlare HQ in San Francisco on Thursday, March 1.

Industry leaders who manage the communities of LinkedIn,TaskRabbit, SoundCloudand CloudFlare will be on hand to discuss the best practices of community management and how to get the most out of social media. It will be a lively and informative evening. The event opens at 6:30pm at our office in SOMA at 3rd and Townsend. Want to be a part of it? Sign up here.

Do you have a meetup suggestion? Would you like us to host a meetup of your own here at CloudFlare? Let us know by commenting below or sending an email to meetups@cloudflare.com

We hope to see you on March 1!

Introducing Page Rules: Advanced Caching (Including Configurable HTML Caching)

Matthew Prince — Wed, 22 Feb 2012 18:24:00 -0800

On Monday, CloudFlare officially announced Page Rules. The new feature allows you to customize behavior on a page-by-page basis. The previous two blog posts have outlined how you can turn off CloudFlare's features based on URL patterns, or accomplish advanced URL forwarding.

In addition to the ability to these abilities, Page Rules also enables a powerful new way that you can enhance CloudFlare's caching. This post is dedicated to that.

Default CloudFlare Caching

CloudFlare operates 14 data centers around the world. When a visitor comes to a CloudFlare-powered website, they are directed to the data center closest to them. CloudFlare analyzes the traffic that passes back through each data center to find the parts of a website that are static. We then cache these objects at the edge for a short period of time.

There are two primary benefits of caching. The first is that is moves static objects closer to the visitor requesting them, which makes their delivery faster. The second is that it decreases the load on the origin web server. Caching plays a big part in how we are able to, on average, reduce server loads, bandwidth costs, and page load times by more than half.

The challenge of caching is making sure you don't cache dynamic content. We are, by default, conservative with what we cache. We refresh the cache, by default, at least every 2 hours and we don't display cached HTML to normal visitors unless the origin server is unreachable.

While this is a safe general rule, one of the most requested features has been the ability for us to cache HTML. A lot of sites are largely static, and the owners of those sites would prefer we serve the contents unless it is marked as dynamic. While we have advanced support for cache headers, we've found that they are often misconfigured or difficult for many site owners on hosted platforms to change. With Page Rules, we realized we were able to provide much more advanced caching for those users who wanted it.

Custom Caching with Page Rules

From the Page Rules interface, which you access from the Settings menu next to each domain on your My Websites dashboard, you can setup custom caching. There are lots of different configurations but, since it is one of the most requested options, for this first example, I'll walk through how to specify certain pages as static so their HTML will be cached by CloudFlare.

Like all Page Rules, the first step is creating a pattern and then applying a rule to that pattern. You'll need to find or create a way to differentiate static versus dynamic content by the URL. Some possibilities could be creating a directory for static content, appending a unique file extension to static pages, or adding a query parameter to mark content as static. Here are three examples of patterns you could create for each of those options:

*example.com/static/* [/static/ subdirectory for static HTML pages]
*example.com/*.shtml [.shtml file extension to signify HTML that is static]
*example.com/*?*static=true* [adding static=true query parameter]

These are just three possible examples. There are virtually infinite ways to create a pattern and the best way to do this will depend on your particular website's setup. You'll want to design the pattern to only describe pages you know are static. For example, you'll want to make sure you exclude pages like the administrative page. If necessary, you can create multiple rules to get the exact caching setup you want.

Once you have created a pattern, you can select the Cache Everything option from the Custom Cache menu. Click the Add Rule button and, going forward, anything that matches the rule will get cached by CloudFlare.

Limitations and Variations on Page Rules Caching

We will attempt to cache objects that match the rule, but the caching is limited by the resources available and the number of objects in the cache. Even with the Cache Everything option set, CloudFlare will still periodically check back to refresh the cache. If, at any time, you want to clear the cache then you can do so from the CloudFlare Settings page by selecting the Purge Cache button. Just like with the traditional caching, this will purge Page Rules-based caching immediately and fetch a new copy of content from your server.

In addition to the Cache Everything setting, Page Rules can also be used to override the default cache setting used throughout the rest of your site. For example, you can specify that certain URLs either ignore or respect the query parameters. Respecting the query parameter can be handy if you'd like to be able to invalidate the traditional cache on an object-by-object basis by updating the query string. Alternatively, ignoring the query string can be useful for Javascript but where you want to pass in variables into the script via GET parameters.

Overall, Page Rules makes CloudFlare's caching much more adaptable to accommodate multiple caching strategies under the same domain. We'll continue to add more flexibility to the powerful framework created by Page Rules. If there are particular options you'd like us to support, please don't hesitate to let us know.

Introducing Page Rules: URL Forwarding

Matthew Prince — Tue, 21 Feb 2012 18:47:00 -0800

In the last blog post, I introduced Page Rules and showed how you could use it to control CloudFlare's features like Apps, Performance, and Security settings on a page-by-page basis. Here I'm going to explain how you can use the same Page Rules interface to enable URL forwarding.

URL forwarding was a surprise request from a number of early CloudFlare users. Some hosting providers and registrars charge just for this feature, which seemed silly. We'd generally supplied people looking to do URL forwarding with instructions on how to do it via HTACCESS. When we created the infrastructure to support Page Rules, we realized we could now support URL forwarding in an easy but powerful way.

The Basic Example

Imagine you have a Google+ profile and you want to make it easy for anyone coming to get to simply by going to a URL like:

www.example.com/+

To setup forwarding, go to the Page Rules administration page which can be found under the Settings menu next to each domain on your CloudFlare My Websites page. There create a pattern to match the URL you want to forward:

*example.com/+

This pattern will match:

http://example.com/+
http://www.example.com/+
https://www.example.com/+
https://blog.example.com/+
https://www.blog.example.com/+
Etc...

It will not match:

http://www.example.com/blog/+ [extra directory before the +]
http://www.example.com+ [no trailing slash]

Once I've created the pattern that matches what I want, click the Forwarding toggle. That exposes a field where I can enter the address I want requests forwarded to. My Google+ profile is at the following URL:

https://plus.google.com/117631136894743822101

If I enter that in the forwarding box and click the Add Rule button within a few seconds any requests that match the pattern I entered will automatically be forwarded with a 302 redirect to the new URL. It's slick.

The Advanced Example

Basic forwarding is good for something like redirecting traffic to Google+, but what if you want to do something like force all traffic to your root domain to use the www subdomain. If you use a basic redirect, then you lose anything else in the URL. For example, you could setup the pattern:

example.com*

And have it forward to:

www.example.com

But then if someone entered:

example.com/some-particular-page.html

Then they'd be redirected to:

www.example.com

Not where you'd want them to go:

www.example.com/some-particular-page.html

The solution is to use variables. Each wildcard corresponds to a variable when can be referenced in the forwarding address. The variables are represented by a $ followed by a number. To refer to the first wildcard you'd use $1, to refer to the second wildcard you'd use $2, and so on. To fix the forwarding from the root to www in the above example, you could use the same pattern:

example.com*

You'd then setup the following URL for traffic to forward to:

www.example.com$1

In this case, if someone went to:

example.com/some-particular-page.html

They'd be redirected to:

www.example.com/some-particular-page.html

Or, if you wanted a more powerful Google+ forwarder than described in the basic example above, you could setup the following pattern:

*example.com/+*

And have it forward to your profile like:

https://plus.google.com/117631136894743822101$2

Note the $2 at the end of the URL, which represents the second wildcard (*) in the pattern above. Then all of the following links would work properly:

http://example.com/+
http://example.com/+/posts
http://example.com/+/about
http://example.com/+/photos
http://example.com/+/videos

Troubleshooting

If you can't get forwarding to work properly, make sure the subdomain you're forwarding from is enabled (orange cloud) from the CloudFlare DNS manager. Also check that multiple rules don't interact with one another in an unexpected way. The rules will take precedence based on when they were created, so if they are not behaving in the way you expect you may need to delete the rules and recreate them in a different order.

Forwarding using Page Rules enables a number of possibilities that used to require you creating complicated redirect rules in HTACCESS. Give it a shot and let us know if you find powerful new uses in the comments below.

Introducing Page Rules: Fine Grained Control over CloudFlare's Features

Matthew Prince — Mon, 20 Feb 2012 23:57:00 -0800

CloudFlare is provisioned by DNS. That means that, from the beginning, you've been able to turn CloudFlare on or off on a subdomain level. From the CloudFlare DNS Manager, you can toggle the little clouds next to your subdomains orange (on) or gray (off) in order to control whether traffic for that subdomain will pass through CloudFlare's proxy.

Often, however, users have wanted finer-grained control. For example, you may want to have a CloudFlare app like UserVoice added to your public-facing pages, but not appear on your private administrative areas of your website. Before today, that wasn't an option.

Introducing Page Rules

Page Rules is a powerful new set of tools that allows you to control how CloudFlare works on your site on a page-by-page basis. The feature provides many of the most popular controls of HTACCESS with a user-friendly interface. Page Rules are now available for all users from the Settings menu next to each domain on the My Websites page. Over the next few days we'll be posting some tutorials on the CloudFlare Blog on various ways you can use Page Rules, but I wanted to start with the example above: how to turn a CloudFlare app like UserVoice off on the administrative portion of your website.

Using Page Rules to Control CloudFlare Features and Apps

The first step to using Page Rules is to define a pattern that defines when the rule is triggered. These patterns can be simple, such as a single URL, or complicated including multiple wildcards. Imagine you have a content management system with a single administrative URL:

https://www.example.com/admin.php

If that is the only URL where you want CloudFlare Apps to be turned off, you can enter it in exactly that form as a new pattern. Then, below the pattern, toggle the "Apps" setting to "Off." It's as simple as that.

Wildcard and Advanced Pattern Matching

The pattern above will only match the following URL:

https://www.example.com/admin.php

It will not match any of the following URLs:

http://www.example.com/admin.php [http ≠ https]
https://example.com/admin.php [missing www subdomain]
https://www.example.com/admin [admin ≠ admin.php]

You can make rules more flexible by including wildcards with the * character. For example, if you wanted the pattern to match all four of the above URLs, you could use a pattern like:

*example.com/admin*

A wildcard can represent zero or more characters and can be used anywhere in the pattern. So, for example, the following pattern:

https://www.example.com/*b*/*

Would match:

https://www.example.com/blog/
https://www.example.com/blog/index.php
https://www.example.com/b/admin/folder/index.php
https://www.example.com/myblog/

But would not match:

https://www.example.com/blog [missing the trailing slash]
https://www.example.com/sam/index.php [sam doesn't contain a "b"]

Practical Example

Many WordPress users want to disable CloudFlare Apps and performance features like Rocket Loader on their WordPress Admin panel while leaving them on for their public facing pages. To do this, for most default WordPress setups, you can now create a PageRule by defining the following rule:

*example.com/wp-admin*

Then toggle Apps and Performance to "off" then click the Add Rule button. As soon as the rule is in place, your WordPress admin pages will not have CloudFlare Apps like UserVoice included and will not be altered by features like Rocket Loader or AutoMinify.

Non-Trivial

Making this work at our scale is non-trivial. To make sure it is fast, when you create a rule, it is compiled into machine code and pushed out to the edge of our network within a couple seconds. Every rule needs then needs to be checked with every request. Given that, under normal load, we're now processing well over 50,000 requests per second, we needed to put some limits on the number of rules per user. Free accounts include three (3) Page Rules per domain. If you need more, you can upgrade to a Pro account which includes twenty (20) Page Rules per domain.

Over the next few days, we'll be posting other powerful things you can do with Page Rules including how you can use it for advanced URL forwarding as well as ways that it can enable powerful new caching.

CloudFlare Is a Community

Matthew Prince — Fri, 17 Feb 2012 22:46:00 -0800

Today, CloudFlare adds more than 250 new customers every six hours or so, but getting our first 250 took several months and a lot of faith from our earliest adopters. When we started working on CloudFlare, an employee at a major CDN company warned us that we had no idea all the crazy things people did with their websites. He wasn't kidding. For the first sites that signed up, we usually made them slower and offered little additional protection. But, over time, and with the patience of our first users, we incorporated everything we learned from each new site and built something great.

Together We Grow Stronger

The core value proposition of CloudFlare has always been that the system gets smarter and faster with each new website that joins. In that sense, CloudFlare is a community. When an attack is launched against any one site, knowledge about that attack is immediately shared across the rest of the network. Similarly, we use data from the performance of sites on CloudFlare to help tune optimizations for each new site that joins.

Onward and Upward

Today CloudFlare's community is made up of hundreds of thousands of sites, and each new site that joins continues to make the system better. Together we have brought the resources previously reserved only for the Internet giants to the rest of the web, and we've grown into a giant ourselves. We now power more page views per month than Twitter, Amazon.com, Wikipedia, Zynga, AOL, Apple, and Bing — combined. We have big plans for tomorrow and ways we are continuing to work to save the web, but we'll always remember that we couldn't have done it without you.

From the whole CloudFlare team, thank you!

PS - Want to have an even bigger impact? We're always hiring and we usually get our best candidates from our existing users.

CloudFlare at Parallels Summit

Kristin Tarr — Wed, 08 Feb 2012 21:40:00 -0800

CloudFlare is going to be at the Parallels Summit in Orlando next week, February 14-16. We are offering complimentary limousine transfers from the Orlando International Airport (MCO) to the Gaylord Palms Resort. Ride in style and reserve your spot today!

CloudFlare co-founders Matthew Prince and Michelle Zatlyn will both be giving presentations during Parallels Summit. Matthew is speaking on Thursday at 4:45pm and Michelle is speaking on Wednesday at 3:30pm.

Be sure to stop by our exhibit booth (#311) to introduce yourself and get a CloudFlare tshirt.

We are looking forward to seeing current partners and potential new partners during the event!