Think Global, Peer Local. Peer with CloudFlare at 100 Internet Exchange Points

Internet Exchange Points (IXPs) or Network Access Points (NAPs) facilities are where networks meet, participating in what's known as peering, which interconnects various parts of the global Internet.

At CloudFlare we are dedicated to peering. So much so that we just joined our 100th Internet Exchange point!

Image courtesy of Martin Levy

According to Wikipedia:

“In computer networking, peering is a voluntary interconnection of administratively separate Internet networks for the purpose of exchanging traffic between the users of each network”

In reality this normally means a physical place where two different networks (they could be backbones, CDNs, mobile networks or broadband ISPs) connect their respective networks together to exchange traffic. Over the last fifteen years, there has been a major expansion in network interconnections, running parallel to the enormous expansion of the global Internet. This expansion includes new data centre facilities being developed to house network equipment. Some of those data centres have attracted massive numbers of networks, in no small part due to the thriving Internet Exchanges Points (both new and existing) that operate within them. London with the LINX and LONAP exchanges, Amsterdam with AMS-IX and NL-IX exchanges, Frankfurt with DE-CIX and ECIX exchanges are good examples for Europe. In North America there are plenty of IXPs run by Equinix, TelX, Coresite and a myriad of independent operators. The same can be said for Asia Pacific, South America, Oceania and more recently the African continent. Over the last fifteen years IXPs have been deployed in nearly every corner of the globe.

Peering has become a core requirement of the global Internet. With around 54,000 unique Autonomous Networks (ASNs) making up the global Internet, nearly 6,000 of those networks participate in the peering ecosystem; some big, some small.

A volunteer-based organisation runs PeeringDB, a database of peering networks, peering points, peering locations and more. It’s 100% user-driven data and provides networks with a directory of IXPs and contact information for networks willing to peer. As of January 2016 there’s around 5,700 networks and around 630 IXPs listed in the database. Keeping your PeeringDB information maintained is a must for anybody wanting to peer; here are some tips.

There’s almost nothing stopping any network from peering, especially if there’s another network within its physical reach. Once you have two, three (or more) networks in the same locale, then it makes perfect sense to start an IXP.

CloudFlare operates 75+ PoPs globally and nearly all of them are located close to where a local Internet Exchange exists. By connecting to the local IXP (or multiple local IXPs) we provide our customers an unprecedented level of web and DNS service delivery.

At their core, IXPs are large Layer 2 LANs. Sometimes built with one Ethernet switch, sometimes many switches interconnected together across one or more physical buildings. With either configuration the goal is to connect networks together physically. An IXP is no different in basic concept to your home network, with the only real difference being scale. IXPs can range from 100s of Megabits/second to many Terabits/second of exchanged traffic. Independent of size, their primary goal is to make sure that many networks’ routers are connected together cleanly and efficiently. In comparison, at home you would normally only have one router and many computers or mobile devices.

Across the IXP's local network those different ISPs are able to create one-to-one connections using the BGP protocol. This protocol was created to allow disparate networks to announce to each other their own IP addresses plus those that they have provided connectivity to downstream (i.e. their customers). Once two networks set up a BGP session, their respective routes are exchanged and hence traffic can flow directly between them. Any previous intermediary network or networks are removed from the equation.

BGP allows groups of IP addresses to be announced as a single entity (called a CIDR block). An IP address is a 32 bit number for IPv4 and 128 bit number for IPv6. As the global Internet is made up of billions and billions of IP addresses, the BGP protocol consolidates IP addresses into CIDR network addresses. Within the IPv4 Internet these networks could contain anywhere from 256 contiguous IP addresses (called a /24) up to 16,777,216 contiguous addresses (a /8). A /24 means 2^(32-24) addresses. In IPv6, the smallest network block is either a /64 (18,446,744,073,709,551,616 IP addresses - or 2^(128-64) addresses) or a /48 (a staggering 1,208,925,819,614,629,174,706,176 individual addresses). It’s best to think of a /48 as just 65,536 /64s - the basic building block of IPv6 networking.

Getting back to BGP and the global Internet, the reason it works is that cooperative networks announce their IP address blocks (CIDR blocks) to peering neighbours. Here’s an example of one of those BGP announcements:

8.8.8.0/24         *[BGP/170] 15:29:05, MED 0, localpref 200, from 80.249.208.247
                     AS path: 15169 I, validation-state: unverified
                     to 80.249.208.247 via ae3.0
                   > to 80.249.209.100 via ae3.0

CloudFlare can see this route within its network because our network peers directly with the source of that specific CIDR block. What’s going on here is the other ISP’s router (aka our peering partner) is telling our router that our network can send traffic towards 8.8.8.0/24 (the network block that encompasses Google’s 8.8.8.8 DNS resolver service) to the IXP fabric. Our router sends traffic to the IP addresses of 80.249.208.247 or 80.249.209.100 as the next hop on the exchange fabric. (Fabric is a fancy word for Ethernet switch). The AS path shows that it’s a direct hop to AS15169/Google, meaning we have direct connectivity to Google over this specific Internet Exchange Point without an intermediary network. In reality, CloudFlare and Google interconnect via peering points in every continent (nearly 65+ locations globally).

This single example is repeated again-and-again with 1,000s of other networks across all of CloudFlare’s locations. Each peering session and peering point are dedicated to improve CloudFlare’s network offering towards its customers.

Without IXPs, traffic going from one ISP to another would rely on an intermediary backbone ISP to carry the traffic from source to destination. In some situations there’s no problem with doing this: it’s how a large portion of international Internet traffic flows, as it’s cost prohibitive to maintain direct connections to each-and-every ISP in the world. However, relying on a backbone ISP to carry local traffic can be adverse to performance, sometimes due to the backbone carrier being connected to another backbone whom it hands the traffic to in a completely different city. This situation can lead to what’s known as tromboning, where traffic from one city destined to another ISP in the same city can travel vast distances to be exchanged and then return again. In many developing countries this is a very common.

By connecting to a local Internet Exchange Point and peering with other local ISPs, traffic becomes finely tuned and traffic is exchanged locally and directly. This gives the best performance for our web and DNS services towards those destination networks that peer with us. At some of our PoPs we serve more than 90% of outgoing traffic over Internet Exchange Points, helping to keep latency for network traffic as low a possible.

As well as a performance benefit, exchanging traffic directly over IXPs improves redundancy. Because CloudFlare can setup peering with those other ISPs over more than one exchange, this means both networks have possibly diverse paths in which to exchange traffic.

If for example an ISP only has connectivity to one or two backbones and one of those backbones suffers an outage, traffic destined for ISPs on the Internet Exchange would not be affected as it is exchanged directly.

CloudFlare operates a network that protects its customers from DDoS attacks and alas that DDoS traffic (created somewhere on the global Internet) ends up flowing towards CloudFlare’s network. As we have written about before here and here, we have built a massive network to address DDoS traffic and filter it prior to reaching our customers’ web services. Peering plays a big part in that story. In the same way as we want to optimise the path out of our network towards the end user (providing the best experience when viewing content delivered by our network), we also want to optimise the path of nefarious DDoS traffic inbound to our network.

Why? Removing an intermediary network that’s in the path of a DDoS attack can actually help keep the Internet operating cleanly. We've built a network that will handle DDoS attack traffic, if that traffic reaches us via direct peering sessions then experience shows we have the ability to handle the traffic levels (and packet-per-second levels) associated with it.

If there’s a intermediary network in that path we sometimes observe collateral congestion or packet loss within that network. Not a good thing. A direct path (over a peering connection) means there’s the shortest path between the nefarious source and our filtering and sinking of that DDoS traffic. Without peering, the Internet could not handle the types of DDoS attacks that are much too common these days.

Just about anyone! There’s 100s and 100s of IXPs operating globally. Some countries only have one IXP, some (such as Brazil) have nearly 50 individual IXPs. Alas there are quite a few countries where no IXP exists at all.

Many operating models exist for IXPs. CloudFlare connects to IXPs with a variety of these models.

The membership model (preferred in Europe) centres around a group of networks that are members of a non-profit organisation that operates an IXP in one or more cities. Membership payments along with a per-connection fee keep the IXP operating. With a membership based IXP, the fee structure is publicly published and all members pay equally. This model does exist in some North American cities, though to a much lesser extent. It also exists in most other continents.

Next up is a purely commercial model where the operator of an IXP is a commercial for-profit entity. With the commercial model, pricing can be negotiated and in many cases the commercial entity also operates a data centre facility where costs for colocation (space, power, security, remote-hands, cross-connects et al.) eclipse the cost of the IXP itself, Negotiating downwards towards no cost is quite common. After all in the commercial world, a loss-leader mentality can be common.

Sometimes there are 100% volunteer-based IXPs with very little structure. These exist in many cities as a secondary IXP to a primary (read: larger) IXP. Sometimes a friendly group of ISPs get together (over a few beers?) and just decide it would be fun to operate an IXP. We love these people. You’ve read many times about our love of innovation, entrepreneurship, random hacking and experimenting for the good of the Internet. That also happens in the IXP space. Sometimes these IXPs get big and have to convert to a paid-for model. That’s success in its rawest form!

Not be outdone by commercial, membership or even volunteer models, some governments insist on getting into the IXP game. In most cases this is in countries where an excessive number of telecommunication regulations exist. As you can imagine (if you've read this blog before), we aren’t big fans of telecommunication regulations. That said, some of these IXPs surprisingly work quite well. Egypt’s regulatory body (NTRA – National Telecommunications Regulatory Authority) operates CAIX (the Cairo Internet eXchange). It’s a success story for the Egyptian Internet. India’s various Internet Exchanges have not been as successful. These seven IXP locations are operated by the Indian government’s NIXI (National Internet Exchange of India) company. When reading the statistics, from a distance, it shows bandwidth levels very low when compared with the size of the Indian Internet itself. CloudFlare operates three PoPs in India and by design they are not connected to the NIXI exchanges.

Finally, there’s the Pseudo-IXP. This is an exchange operated by a telecom operator (sometimes the incumbent) whose sole purpose is to provide connectivity back to its own network and charge accordingly for the pleasure. Thailand has nearly a dozen of these. Each telecom company or mobile operation has its own IXP, which misses the point of interconnects! We aren't a fan of this model, and most forward-thinking networks aren't fans of this model either.

When a network wants to expand its footprint and connect to an IXP in another city or country (or even continent), it was normal practice to extend the Layer 3 network by placing a router in that remote location. There was expense associated with those implementations. A router has to be purchased and shipped to the remote site. The remote site has to be acquired (renting space, power, etc.) and some form of remote-hands are needed to rack and stack the network equipment. Finally, there needed to be a contract with the IXP and a connection engineered and paid for.

Then along came the concept of remote peering, which extends the Layer 2 switch fabric (in a very controlled manner) via a third party telecom operator to another city, country or even continent, allowing remote networks to become true members of an IXP. Europe took this concept and went wild with it. The main IXPs in London, Amsterdam & Frankfurt got telecom partners signed up to provide remote peering. Their number of connected networks (and traffic levels) increased instantly. Other parts of the world got the remote peering bug and in this day and age you can virtually (pun intended) connect to IXPs all over the world via remote Layer 2 connections.

Remote peering is an obvious idea and to be honest it helps many ISPs reach peers in neighbouring cities or countries via a direct peering relationship. For CloudFlare our aim is to serve traffic as locally as possible and hence we don’t use remote peering. We try to not peer with remotely connected networks if we have a PoP closer to them. We are relentless in this desire to keep traffic local.

Similar to the concept of remote peering, some IXPs expand their footprint across a country or in some cases across multiple countries, this means an ISP can peer with another ISP in a different country via a local connection to the IXP. This is for practical reasons the same as remote peering, with the difference that the long-range link is between the IXP switches, instead of the connection between the ISP and the IXP. There are a few examples of where CloudFlare peers on distributed exchanges, in order to reach both local participants and ISPs in other surrounding countries that we’ve not yet finished building PoPs in.

The world isn’t perfect and in some markets local peering cannot be achieved (sometimes due to a lack of any local IXPs). In situations where we cannot peer locally on an IXP we aim to connect to ISPs privately. This is the same idea as peering over an IXP except we connect directly to the ISP’s router from our router.

In some cases CloudFlare will help foster the creation of a new IXP or IXPs, such as the many new MegaIX exchange points coming to North America. After all, when you improve local Internet connectivity, you actually improve global connectivity. The Internet is a global entity and every part plays an important role.

While some marketplaces have only one IXP (for example Dublin in Ireland with INEX), some markets have seen redundancy, duplication or fragmentation between IXPs. This is in most cases a good thing. Redundancy or even competition between commercial entities always makes the Internet stronger. The Internet was designed to handle disruptions to its underlying infrastructure. IXPs are just one more part of that infrastructure. We will sometimes connect to three (or four) IXPs in a single city. Four may seem excessive, but for the sake of good redundancy, four is a great number.

When there is more than one IXP in a city there’s a tendency for some ISPs to prefer one IXP over another, or in some cases even running their own. By CloudFlare being present at many exchanges, it gives us a unique and redundant view of the local Internet, it also allows us to have the best coverage of that local market. It brings the added advantage of being able to peer with certain regional networks over multiple exchanges and hence allowing for the best routing redundancy.

After five years of operating a network that's grown to over 75 PoPs globally we have joined our 100th Internet Exchange Point. We have a network that’s growing each and every day. Reaching 100 may be just a number; however, only one or two networks have had the ability to grow to that size. Our customers reap the benefits of that global reach. We won’t stop at 100. Far from it! Want to help us grow our network and increase our global peering? Check out available roles here.

Are you an ISP? Do you want to peer with us? Visit as13335.peeringdb.com or talk to us about an on-net cache.