CloudFlare's Network Engineer, Tom Paseka, will be in Hong Kong at the end of August. During his time there, we wanted to host a meetup for CloudFlare users and anyone interested in web performance and security.

The event will include a short presentation on what's new at CloudFlare, features we've added, and a preview of what's on the roadmap. Afterwards, Tom will answer any questions that you may have.

If you're a CloudFlare user, come introduce yourself to Tom. If you're not a CloudFlare customer, but interested in web performance and security, then you are welcome to join as well. We look forward to meeting our users and friends from China!

• Saturday, August 25

• 5:00pm

• Hong Kong (exact location TBD)

RSVP & More information: http://www.meetup.com/CloudFlare-Meetups/events/77779312/

Note: We are still confirming our venue. If you have a suggestion,please let us know.

To understand the problem, you need to understand a bit about how Internet routing works. The Internet is a massively interconnected network. Networks send packets to each other across routes. These routes are set for each network by routers. A route defines the path for packets to take to get to a particular IP address. One network will announce that it is responsible for a particular set of IP addresses. That fact is then shared to upstream routers so if they see a packet bound for a particular IP they can send it in the correct direction.

Routers exchange routes between each other using something called Border Gateway Protocol (BGP). When two networks interconnect, they generally trust each other's routes. If a routing change is announced by one router, the immediately connected upstream routers will pickup the routing change. They will subsequently pass the change on to other routers that are further upstream.

Today we had a scheduled maintenance for our Hong Kong data center while its systems were being upgraded. The data center was taken offline by shutting down all the in-bound Anycast routes. This, as we intended, caused all traffic that would have gone to that data center to hit the next closest facility (either Singapore or Tokyo).

While the systems were being upgraded, our network team worked to optimize some of the routing in Hong Kong. At some point, the out-bound routes were entered into the in-bound interface. The out-bound routes describe our entire net range so the net effect was the router in Hong Kong was announcing that it was the correct place to send all traffic bound for CloudFlare's IP space.

Our upstream provider trusts our routes so, via BGP, they were quickly relayed throughout their network and to their upstreams. The result was traffic from around the world was directed to the Hong Kong data center, which was offline. We realized the issue and announced the corrected routes. It took approximately 15 minutes from the beginning of the problem to when the routes were corrected network wide. About 25% of CloudFlare's in-bound traffic comes from direct peers. This traffic was not affected by the routing because the direct peers trusted our routes more than the ones they were receiving from other upstreams.

We are implementing systems to run all routing changes through a verification layer that double check before any routes are announced. We are also talking with all our upstream providers to enable additional checks on their networks that do not automatically propogate major routing changes without confirmation.

This is only the second significant outage in CloudFlare's history (here's our post mortem from the other). Any period of downtime is completely unacceptable to us. On behalf of our whole team, I apologize for the problem. We have learned from this experience and are already implementing the safeguards to ensure it will not happen again.

CloudFlare is now connected to HKIX (Hong Kong Internet Exchange). HKIX is the largest Internet Exchange in Asia transferring around 200Gbit to its 160 ISP, Carrier and Content networks. HKIX is fundamental to the local Hong Kong ISP Market, with every provider in Hong Kong connected, as well as excellent regional coverage with networks from Thailand to Japan to Australia.

So, what does a connection to HKIX mean for CloudFlare users? Faster performance for your Hong Kong web surfers. Being a part of the HKIX allows us to deliver traffic to all Hong Kong web surfers within Hong Kong. This will improve web browsing performance by keeping the traffic local and the latency low.

Connecting to HKIX is one of the steps CloudFlare is working on to bring the Internet closer to you. Watch for more news to come over 2012!