This blog focuses on IPv4. We provide four IPs (two for each IP address family) in order to provide a path toward the DNS resolver independent of IPv4 or IPv6 reachability.

1.0.0.0/8 was assigned in 2010 to APNIC, before this time it was unassigned space.

% IANA WHOIS server
% for more information on IANA, visit http://www.iana.org
% This query returned 1 object

inetnum:      1.0.0.0 - 1.255.255.255
organisation: APNIC
status:       ALLOCATED

whois:        whois.apnic.net

changed:      2010-01
source:       IANA

https://www.iana.org/whois?q=1.0.0.0%2F8

Unassigned, however is not the same as reserved for private use!

To put it simply, 1.1.1.1 was BROKEN! The good news is, for most users 1.1.1.1 is now reachable. We’ve worked hard to ensure that issues get resolved and continue to contact operators to resolve issues quickly. We’re confident we can get everything cleaned up, but this is a stark reminder that you shouldn’t hijack IP addresses not assigned to you. We found over 1,000 probes out of just over 10,000 were unable to make DNS queries to 1.1.1.1 successfully. Some of this was due to single large networks having reachability issues, for example a large operator in Germany has nearly 350 probes connected, all of them failing. The methodology for testing was very simple:

1. Run a DNS lookup measurement towards 1.1.1.1

2. Find probes where the lookup fails

3. Run a traceroute with affected probes

4. Analyse the result

The results were quite mixed, but fell into three main causes:

1. Built-in 1.1.1.1 ISP routers using 1.1.1.1 as an internal IP address, preventing queries from reaching the real 1.1.1.1

2. Blackholing 1.1.1.1 ISPs statically configuring a route for 1.1.1.1 inside their networks, preventing traffic leaving their network, either through routing internally, or by sending the packets to null0

3. Filtering 1.1.1.1 ISPs dropping packets on ingress or egress to/from their network when sourced/destined from/to 1.1.1.1

Of these three main causes, the majority of cases were either 1, 2 or both! Several ISPs even had route loops internally, where they were advertising 1.1.1.1 inside their network, but had no actually path to it, so packets loop around and around.

Once we had narrowed it down for each group of probes we began contacting ISPs for clarification on what was happening, several networks responded very quickly reporting they had removed an internal route to 1.1.1.1, which in most cases was the beginning and end of the matter. There were plenty of networks which took their time to respond, but in the end did the same, removing the internal routes left there for legacy testing reasons.

All of those fixes were great to make, but most were quite uninteresting, what was more interesting was finding cases from issue number 1, CPEs (customer premises equipment) aka home routers, gateways and wireless access points. With the help from the folks at Sonic we were quickly able to identify that the Pace 5268 a common xDSL modem deployed primarily in the United States (including wide usage on AT&T) uses 1.1.1.0/29 for internal communication. We requested comment from AT&T’s noc, but have not heard anything from them. We did however receive a response from them via social media:

Independent investigation confirms the findings:

The same finding was made with the D-Link DMG-6661, which was reported to us by a user from Brazil connected to Vivo FTTH.

Another user in Argentina connected to Telefónica found the issue on the Mitrastar GPT-2541GNAC.

It appears this CPE has been deployed in many of Telefónica's networks internationally.

We noticed similar behaviour on a large portion of probes connected to Orange France, we contacted them and received a swift response that the CPE team was investigating the issue. After providing more details they came back to us with a statement.

We have escalated your alert inside ORANGE France.Our CPE team has analyzed the issue which is now well understood. This problem only impacts a subset of our CPEs. The commitment of the fix is currently on going and a deployment on our CPEs will follow.In case of complaints notified by our customers in the meantime we have prepared a communication to inform them that we are currently fixing the issue.

The CPE in question is the Livebox, although it's not clear which versions are affected, it should be resolved by Orange across all affected devices. Users in Poland reported the same issues as users in France, likely due to Orange deploying the same models across multiple networks.

By far my favourite response was from the friendly folks at Telenor:

I have corrected all routers in our network now that had an awful old solution that is now obsolete. Thank you Cloudflare for the help to get it done!

Obsolescence is inevitable, but the desire to speedily fix such occurrences is great to see.

These are by no means all devices that have issues, but some of the wider deployed ones. The current list we have of affected devices is:

• Pace (Arris) 5268

• D-Link DMG-6661

• Technicolor C2100 Series

• Mitrastar GPT-2541GNAC

• Askey RTF3507VW-N1

• Calix GigaCenter

• Nomadix (model(s) unknown)

• Xerox Phaser multi-function printer

• See below :)

If you have a device that is affected, please let us know in the comments. A good example of this is a super-low latency with only 1 hop to 1.1.1.1:

Traceroute to 1.1.1.1 (1.1.1.1), 48 byte packets

1 1.1.1.1 1dot1dot1dot1.cloudflare-dns.com AS13335 8.301ms 1.879ms 1.836ms

Using the RIPE Atlas probes gives excellent visibility into residential and business internet connections, however they’re connected via a cable, so this rules out another use-case, WiFi access points. After very little research we quickly came across Cisco mis-using 1.1.1.1, a quick search for “cisco 1.1.1.1” brought up numerous articles where Cisco are squatting on 1.1.1.1 for their Wireless LAN Controllers (WLC). It’s unclear if Cisco officially regards 1.0.0.0/8 as bogon space, but there are lots of examples that can be found on their community websites giving example bogon lists that include the /8. It mostly seems to be used for captive portal when authenticating to the wireless access point, often found in hotels, cafés and other public WiFi hotspot locations.

Here are some interesting statistics from before we started contacting operators and after we have fixed many issues. It’s staggering to see how fixing some key networks increased availability by almost 20% in Europe & North America!

We began testing the availability of 1.1.1.1 on the 23rd of March, in Europe and North America it was only around 91%.

1.1.1.1 availability from Europe and North America, 23rd of March

By the 3rd of April, our work cleaning up the space had pushed the availability up to 97%.

1.1.1.1 availability from Europe and North America, 3rd of April

For the rest of the world, excluding Europe and North America, availability to 1.1.1.1 was only 73% on the 23d of March.

1.1.1.1 availability for the World (Europe and North America excluded), 23rd of March

By the 3rd of April we've made a tonne of progress and managed to clean up enough of the bad routing that availability was up to 85%.

1.1.1.1 availability for the World (Europe and North America excluded), 3rd of April

We are continuing to work with ISPs and CPE manufacturers to clean up bad routing globally. Our goal is for 1.1.1.1 to be properly routed and available for 100% of Internet users.

Above images from Catchpoint analytics

The last public analysis was done in 2010 by RIPE and APNIC. At the time, 1.1.1.0/24 was 100 to 200Mb/s of traffic, most of it being audio traffic. In March, when Cloudflare announced 1.0.0.0/24 and 1.1.1.0/24, ~10Gbps of unsolicited background traffic appeared on our interfaces.

The most targeted IPs were 1.1.1.1, 1.1.1.8, 1.1.1.10, 1.0.0.19. When searching these IPs, we notice it is usually misconfiguration or hardcoded IPs.

The destination port is usually 80/443, but also other variants of HTTP ports (8000, 8080, et al.) using UDP and TCP, seemingly trying to setup a proxy connection. Some of the traffic was also DHCP/BOOTP, iperf and syslog.

Some IPs are also the target of DDoS attacks (even before we announced the new service publicly). Analysing by source port we saw NTP and memcached, usually reaching 5Gbps for a few minutes. The short duration of the attack shows that could be a hardcoded IP in a botnet before it starts sending traffic to a specific target.

We also noticed daily patterns where 4 IPs receive the same amount of traffic (±50Mbps).

All of this bad traffic is unrelated to DNS, simple, unsolicited background traffic.

It was clear from the start that we’d have our work cut out, especially with CPE vendors, where a firmware update would be required. What was impressive was the willingness of operators to collaborate with us to clean up the legacy misconfiguration. It was clear that 1.1.1.1 needed a lot of cleaning in order to be globally accessible. We decided six months before the 1st of April release date we'd commit the network and support resources to that task. Now that 1.1.1.1 is live, we’re thankful to all the networks and hardware companies who have assisted us in this effort. We’re not done, nor are others.

The RIPE Atlas project was immensely useful in testing reachability from as many networks around the world as possible. If you’d like to help the project, please consider hosting a probe. Some networks are not covered with at least one probe, you can see if your ISP has a probe here, sorted by country.

Particular thanks to the following operators who were responsive and helped clean up issues quickly.

• Airtel

• Beirut-IX

• BHTelecom

• Comcast

• ITC

• Fastweb

• Kazakhtelecom

• Level(3)

• LG Telecom

• Liquid Telecom

• MTN

• Omantel

• Rostelecom

• SKBB

• SFR

• STC

• Tata

• Telecom Italia

• Telenor

• Telus

• Turk Telekom

• Turkcell

• Voo

• XS4ALL

• Ziggo

We still have work to do, contacting operators that we see issues with, in the meantime you should be able to use our second IP address of 1.0.0.1, which has far fewer issues. Don't forget, both of our IPv6 addresses too: 2606:4700:4700::1001 and 2606:4700:4700::1111.

Do you still have reachability issues to 1.1.1.1? You can find more information at our community forum. We’d also recommend reporting such issues to your ISP, they may already be aware of issues, or they may need you to report it to them to start investigating. Whichever is true, making them aware is especially helpful, operators are not always receptive to reports from external parties.

Our newest data center in Edinburgh expands our current total to 139 cities globally with a Cloudflare deployment. It also brings our UK total to 3 cities, after London and Manchester.

Edinburgh is the capital of Scotland, located in the Lothian region. It has been recognised as the capital since the 15th century, and is the home of the Scottish government, parliament and supreme courts. It is a city of many hills, with important landmarks such as Edinburgh Castle being built at the top of a hill.

Photo by Nicola Gadler / Unsplash

One of the most famous events held each year in Edinburgh is the Fringe Festival, which is reported to be the world's largest arts festival. Many famous comedians have made their big break at this very festival.

The internet community in the UK is very well established, but mostly concentrated on London and Manchester, with a heavy emphasis on London. By deploying in Edinburgh we're encouraging ISPs to regionalise their traffic, away from just London. We're connected to IXScotland, and are actively seeking to peer with other connected networks, to help build the peering community.

Can you guess three more cities we'll announce deployments to in Europe? Comment your answers to be in with a chance of winning some Cloudflare swag.

This map reflects the network as of the publish date of this blog post. For the most up to date directory of locations please refer to our Network Map on the Cloudflare site.

Iceland is a small country in Northern Europe, a land of active volcanoes and boiling hot geysers. The geology and climate creates unique conditions for running compute power. With an abundance of green electricity and natural cooling, many companies are placing high power machines in Iceland to run power intensive, heat generating operations. Reykjavík is our 125th location globally.

A unique aspect about Iceland relates to how it connects to the Internet, being situated on the Mid-Atlantic Ridge means submarine cables are necessary to reach networks in other countries. Iceland has three active fibre optic submarine cables that land on its shores: FARICE-1, DANICE and Greenland Connect. Due to the distance, latency to the nearest Cloudflare locations in London and Copenhagen starts at 35 milliseconds. By deploying in Reykjavík, we're able to drive down latency even further to a minimum of under 1 millisecond.

Iceland is unique in many ways, but is no different from other countries when it comes to exchanging internet traffic. ISNIC, Iceland's Network Information Centre runs RIX, the Reykjavík Internet Exchange. Cloudflare is the only CDN network connected to RIX, allowing traffic to flow directly to Icelandic networks.

Photo by Frank Denney / Unsplash

Iceland pulls above its small weight with a population of only 330K. That is 1/1000th population of the United States (but guess which one qualified for the football World Cup this year).

Iceland's Parliament is the oldest in the world, tracing its history to 930 AD.

Like many of its fellow Scandinavian countries it is a progressive country in many ways, ranked number one by the World Economic Forum for women for nine straight years.

Iceland was the first country to elect a female president in 1980.

Iceland has become a hotspot for tourism with over 2 million visiting last year, now Cloudflare is present in Iceland you can can reach websites on Cloudflare with speed and reliability. Thanks to ubiquitous cellular coverage across the country, you're never far away from a website using Cloudflare.

لبنان بلد تاريخي، موطن مدينتين من بين أقدم المدن في العالم. هناك مزيج كبير من التأثيرات من الشرق والغرب. كما أنه أصغر .بلد في آسيا القارية

CC-BY-SA Gregor Rom

Lebanon is a little different to most other countries when it comes to the internet, with all connectivity to the outside world flowing via a single network, Ogero. Traffic to Lebanon was previously served from our existing deployments in Marseille and Paris, due to where Ogero connects to the rest of the internet. By deploying locally in Beirut, round-trip latency is cut by around 50 milliseconds. This might seem like almost nothing, but it adds up when you factor in a DNS lookup and 3-way handshake required to open a TCP connection. Internet penetration in Lebanon according to different sources is around 75%, which is quite high. However, the speed available to end users is low, typically in single digit megabits per second.

The Ministry of Telecommunications has an ambitious plan to improve the connectivity available in Lebanon by 2020, a big part of this involves deploying fiber optic cabling to homes and businesses throughout the country. This will inevitably help to boost the level of traffic we see today coming from Lebanon. Comparing Lebanon to Denmark where the population is only a few thousand lower there is 7x more traffic served to Denmark than to Lebanon.

اتصال لبنان بالإنترنت

لبنان يختلف قليلا عن معظم البلدان الأخرى عندما يتعلق الأمر بالإنترنت، فكل اتصال إلى العالم الخارجي يتدفق عبر شبكة واحدة، أوجيرو. كانت حركة مرور الانترنت إلى لبنان في السابق من عمليات النشر الحالية لدينا في مرسيليا وباريس، ويرجع ذلك إلى حيث تتصل أوجيرو ببقية الإنترنت. من خلال النشر محليا في بيروت، أصبح وقت الإستجابة ذهابا وإيابا أقل من 50 ميلي ثانية واحدة.

قد يبدو هذا لا شيء تقريبا، لكنه يصبح ذو معنى عندما تحسب بحث نظام أسماء النطاقات (DNS) ومصافحة ثلاثية الطرق المطلوبة لفتح بروتوكول التحكم بالإرسال .(TCP) ويبلغ انتشار الإنترنت في لبنان وفقا لمصادر مختلفة حوالي ٧٥%، وهو رقم مرتفع جدا. ومع ذلك، فإن السرعة المتاحة للمستخدمين منخفضة، وعادة تكون رقم مفرد من الميغابتس في الثانية الواحدة.

قامت وزارة الاتصالات بعرض خطة طموحة لتحسين الاتصال المتوفر في لبنان بحلول عام ٢٠٢٠، جزء كبير من هذا ينطوي على نشر كابلات الألياف البصرية للمنازل والشركات في جميع أنحاء البلد. وهذا سيساعد حتما على تعزيز مستوى حركة المرور التي نراها اليوم قادمة من لبنان.وبمقارنة لبنان بالدانمرك حيث يبلغ عدد سكانه بضعة آلاف فقط أقل من لبنان، هناك ٧ أضعاف حركة المرور إلى الدنمارك أكثر من لبنان.

The Internet exchange in Beirut is no exception, with fibre access not possible in Lebanon, ISPs reach the IX by microwave. To give the best access from all around Beirut it is situated at the top of a hill. With most Internet exchanges, line of sight isn’t a concern as fibre is available. Our deployment connected to Beirut IX brings over 7 million websites closers to ISPs connected, making the Internet faster and safer for users in Lebanon.

(Beirut IX) تبادل الإنترنت في بيروت

إن تبادل الإنترنت في بيروت ليس استثناء، إذ لا يوجد صلة للالياف الضوئية في لبنان، فإن مزودي خدمة الإنترنت يصلون إلى نقطة تبادل الإنترنت (IX) عن طريق موجات الميكرويف. لإعطاء أفضل وصول من جميع أنحاء بيروت أنها تقع في أعلى تلة. مع معظم تبادلات الإنترنت، خط الأفق ليس مصدر قلق بما أن صلة الألياف متاحة.

إن نشرنا المتصل بتبادل الإنترنت (IX) في بيروت يجلب أكثر من ٧ ملايين موقع على شبكة الإنترنت، مما يجعل الإنترنت أسرع وأكثر أمانا للمستخدمين في لبنان.

Thank you to Layal Jebran for the translation.

At CloudFlare we are dedicated to peering. So much so that we just joined our 100th Internet Exchange point!

Image courtesy of Martin Levy

According to Wikipedia:

“In computer networking, peering is a voluntary interconnection of administratively separate Internet networks for the purpose of exchanging traffic between the users of each network”

In reality this normally means a physical place where two different networks (they could be backbones, CDNs, mobile networks or broadband ISPs) connect their respective networks together to exchange traffic. Over the last fifteen years, there has been a major expansion in network interconnections, running parallel to the enormous expansion of the global Internet. This expansion includes new data centre facilities being developed to house network equipment. Some of those data centres have attracted massive numbers of networks, in no small part due to the thriving Internet Exchanges Points (both new and existing) that operate within them. London with the LINX and LONAP exchanges, Amsterdam with AMS-IX and NL-IX exchanges, Frankfurt with DE-CIX and ECIX exchanges are good examples for Europe. In North America there are plenty of IXPs run by Equinix, TelX, Coresite and a myriad of independent operators. The same can be said for Asia Pacific, South America, Oceania and more recently the African continent. Over the last fifteen years IXPs have been deployed in nearly every corner of the globe.

Peering has become a core requirement of the global Internet. With around 54,000 unique Autonomous Networks (ASNs) making up the global Internet, nearly 6,000 of those networks participate in the peering ecosystem; some big, some small.

A volunteer-based organisation runs PeeringDB, a database of peering networks, peering points, peering locations and more. It’s 100% user-driven data and provides networks with a directory of IXPs and contact information for networks willing to peer. As of January 2016 there’s around 5,700 networks and around 630 IXPs listed in the database. Keeping your PeeringDB information maintained is a must for anybody wanting to peer; here are some tips.

There’s almost nothing stopping any network from peering, especially if there’s another network within its physical reach. Once you have two, three (or more) networks in the same locale, then it makes perfect sense to start an IXP.

CloudFlare operates 75+ PoPs globally and nearly all of them are located close to where a local Internet Exchange exists. By connecting to the local IXP (or multiple local IXPs) we provide our customers an unprecedented level of web and DNS service delivery.

At their core, IXPs are large Layer 2 LANs. Sometimes built with one Ethernet switch, sometimes many switches interconnected together across one or more physical buildings. With either configuration the goal is to connect networks together physically. An IXP is no different in basic concept to your home network, with the only real difference being scale. IXPs can range from 100s of Megabits/second to many Terabits/second of exchanged traffic. Independent of size, their primary goal is to make sure that many networks’ routers are connected together cleanly and efficiently. In comparison, at home you would normally only have one router and many computers or mobile devices.

Across the IXP's local network those different ISPs are able to create one-to-one connections using the BGP protocol. This protocol was created to allow disparate networks to announce to each other their own IP addresses plus those that they have provided connectivity to downstream (i.e. their customers). Once two networks set up a BGP session, their respective routes are exchanged and hence traffic can flow directly between them. Any previous intermediary network or networks are removed from the equation.

BGP allows groups of IP addresses to be announced as a single entity (called a CIDR block). An IP address is a 32 bit number for IPv4 and 128 bit number for IPv6. As the global Internet is made up of billions and billions of IP addresses, the BGP protocol consolidates IP addresses into CIDR network addresses. Within the IPv4 Internet these networks could contain anywhere from 256 contiguous IP addresses (called a /24) up to 16,777,216 contiguous addresses (a /8). A /24 means 2^(32-24) addresses. In IPv6, the smallest network block is either a /64 (18,446,744,073,709,551,616 IP addresses - or 2^(128-64) addresses) or a /48 (a staggering 1,208,925,819,614,629,174,706,176 individual addresses). It’s best to think of a /48 as just 65,536 /64s - the basic building block of IPv6 networking.

Getting back to BGP and the global Internet, the reason it works is that cooperative networks announce their IP address blocks (CIDR blocks) to peering neighbours. Here’s an example of one of those BGP announcements:

8.8.8.0/24         *[BGP/170] 15:29:05, MED 0, localpref 200, from 80.249.208.247
                     AS path: 15169 I, validation-state: unverified
                     to 80.249.208.247 via ae3.0
                   > to 80.249.209.100 via ae3.0

CloudFlare can see this route within its network because our network peers directly with the source of that specific CIDR block. What’s going on here is the other ISP’s router (aka our peering partner) is telling our router that our network can send traffic towards 8.8.8.0/24 (the network block that encompasses Google’s 8.8.8.8 DNS resolver service) to the IXP fabric. Our router sends traffic to the IP addresses of 80.249.208.247 or 80.249.209.100 as the next hop on the exchange fabric. (Fabric is a fancy word for Ethernet switch). The AS path shows that it’s a direct hop to AS15169/Google, meaning we have direct connectivity to Google over this specific Internet Exchange Point without an intermediary network. In reality, CloudFlare and Google interconnect via peering points in every continent (nearly 65+ locations globally).

This single example is repeated again-and-again with 1,000s of other networks across all of CloudFlare’s locations. Each peering session and peering point are dedicated to improve CloudFlare’s network offering towards its customers.

Without IXPs, traffic going from one ISP to another would rely on an intermediary backbone ISP to carry the traffic from source to destination. In some situations there’s no problem with doing this: it’s how a large portion of international Internet traffic flows, as it’s cost prohibitive to maintain direct connections to each-and-every ISP in the world. However, relying on a backbone ISP to carry local traffic can be adverse to performance, sometimes due to the backbone carrier being connected to another backbone whom it hands the traffic to in a completely different city. This situation can lead to what’s known as tromboning, where traffic from one city destined to another ISP in the same city can travel vast distances to be exchanged and then return again. In many developing countries this is a very common.

By connecting to a local Internet Exchange Point and peering with other local ISPs, traffic becomes finely tuned and traffic is exchanged locally and directly. This gives the best performance for our web and DNS services towards those destination networks that peer with us. At some of our PoPs we serve more than 90% of outgoing traffic over Internet Exchange Points, helping to keep latency for network traffic as low a possible.

As well as a performance benefit, exchanging traffic directly over IXPs improves redundancy. Because CloudFlare can setup peering with those other ISPs over more than one exchange, this means both networks have possibly diverse paths in which to exchange traffic.

If for example an ISP only has connectivity to one or two backbones and one of those backbones suffers an outage, traffic destined for ISPs on the Internet Exchange would not be affected as it is exchanged directly.

CloudFlare operates a network that protects its customers from DDoS attacks and alas that DDoS traffic (created somewhere on the global Internet) ends up flowing towards CloudFlare’s network. As we have written about before here and here, we have built a massive network to address DDoS traffic and filter it prior to reaching our customers’ web services. Peering plays a big part in that story. In the same way as we want to optimise the path out of our network towards the end user (providing the best experience when viewing content delivered by our network), we also want to optimise the path of nefarious DDoS traffic inbound to our network.

Why? Removing an intermediary network that’s in the path of a DDoS attack can actually help keep the Internet operating cleanly. We've built a network that will handle DDoS attack traffic, if that traffic reaches us via direct peering sessions then experience shows we have the ability to handle the traffic levels (and packet-per-second levels) associated with it.

If there’s a intermediary network in that path we sometimes observe collateral congestion or packet loss within that network. Not a good thing. A direct path (over a peering connection) means there’s the shortest path between the nefarious source and our filtering and sinking of that DDoS traffic. Without peering, the Internet could not handle the types of DDoS attacks that are much too common these days.

Just about anyone! There’s 100s and 100s of IXPs operating globally. Some countries only have one IXP, some (such as Brazil) have nearly 50 individual IXPs. Alas there are quite a few countries where no IXP exists at all.

Many operating models exist for IXPs. CloudFlare connects to IXPs with a variety of these models.

The membership model (preferred in Europe) centres around a group of networks that are members of a non-profit organisation that operates an IXP in one or more cities. Membership payments along with a per-connection fee keep the IXP operating. With a membership based IXP, the fee structure is publicly published and all members pay equally. This model does exist in some North American cities, though to a much lesser extent. It also exists in most other continents.

Next up is a purely commercial model where the operator of an IXP is a commercial for-profit entity. With the commercial model, pricing can be negotiated and in many cases the commercial entity also operates a data centre facility where costs for colocation (space, power, security, remote-hands, cross-connects et al.) eclipse the cost of the IXP itself, Negotiating downwards towards no cost is quite common. After all in the commercial world, a loss-leader mentality can be common.

Sometimes there are 100% volunteer-based IXPs with very little structure. These exist in many cities as a secondary IXP to a primary (read: larger) IXP. Sometimes a friendly group of ISPs get together (over a few beers?) and just decide it would be fun to operate an IXP. We love these people. You’ve read many times about our love of innovation, entrepreneurship, random hacking and experimenting for the good of the Internet. That also happens in the IXP space. Sometimes these IXPs get big and have to convert to a paid-for model. That’s success in its rawest form!

Not be outdone by commercial, membership or even volunteer models, some governments insist on getting into the IXP game. In most cases this is in countries where an excessive number of telecommunication regulations exist. As you can imagine (if you've read this blog before), we aren’t big fans of telecommunication regulations. That said, some of these IXPs surprisingly work quite well. Egypt’s regulatory body (NTRA – National Telecommunications Regulatory Authority) operates CAIX (the Cairo Internet eXchange). It’s a success story for the Egyptian Internet. India’s various Internet Exchanges have not been as successful. These seven IXP locations are operated by the Indian government’s NIXI (National Internet Exchange of India) company. When reading the statistics, from a distance, it shows bandwidth levels very low when compared with the size of the Indian Internet itself. CloudFlare operates three PoPs in India and by design they are not connected to the NIXI exchanges.

Finally, there’s the Pseudo-IXP. This is an exchange operated by a telecom operator (sometimes the incumbent) whose sole purpose is to provide connectivity back to its own network and charge accordingly for the pleasure. Thailand has nearly a dozen of these. Each telecom company or mobile operation has its own IXP, which misses the point of interconnects! We aren't a fan of this model, and most forward-thinking networks aren't fans of this model either.

When a network wants to expand its footprint and connect to an IXP in another city or country (or even continent), it was normal practice to extend the Layer 3 network by placing a router in that remote location. There was expense associated with those implementations. A router has to be purchased and shipped to the remote site. The remote site has to be acquired (renting space, power, etc.) and some form of remote-hands are needed to rack and stack the network equipment. Finally, there needed to be a contract with the IXP and a connection engineered and paid for.

Then along came the concept of remote peering, which extends the Layer 2 switch fabric (in a very controlled manner) via a third party telecom operator to another city, country or even continent, allowing remote networks to become true members of an IXP. Europe took this concept and went wild with it. The main IXPs in London, Amsterdam & Frankfurt got telecom partners signed up to provide remote peering. Their number of connected networks (and traffic levels) increased instantly. Other parts of the world got the remote peering bug and in this day and age you can virtually (pun intended) connect to IXPs all over the world via remote Layer 2 connections.

Remote peering is an obvious idea and to be honest it helps many ISPs reach peers in neighbouring cities or countries via a direct peering relationship. For CloudFlare our aim is to serve traffic as locally as possible and hence we don’t use remote peering. We try to not peer with remotely connected networks if we have a PoP closer to them. We are relentless in this desire to keep traffic local.

Similar to the concept of remote peering, some IXPs expand their footprint across a country or in some cases across multiple countries, this means an ISP can peer with another ISP in a different country via a local connection to the IXP. This is for practical reasons the same as remote peering, with the difference that the long-range link is between the IXP switches, instead of the connection between the ISP and the IXP. There are a few examples of where CloudFlare peers on distributed exchanges, in order to reach both local participants and ISPs in other surrounding countries that we’ve not yet finished building PoPs in.

The world isn’t perfect and in some markets local peering cannot be achieved (sometimes due to a lack of any local IXPs). In situations where we cannot peer locally on an IXP we aim to connect to ISPs privately. This is the same idea as peering over an IXP except we connect directly to the ISP’s router from our router.

In some cases CloudFlare will help foster the creation of a new IXP or IXPs, such as the many new MegaIX exchange points coming to North America. After all, when you improve local Internet connectivity, you actually improve global connectivity. The Internet is a global entity and every part plays an important role.

While some marketplaces have only one IXP (for example Dublin in Ireland with INEX), some markets have seen redundancy, duplication or fragmentation between IXPs. This is in most cases a good thing. Redundancy or even competition between commercial entities always makes the Internet stronger. The Internet was designed to handle disruptions to its underlying infrastructure. IXPs are just one more part of that infrastructure. We will sometimes connect to three (or four) IXPs in a single city. Four may seem excessive, but for the sake of good redundancy, four is a great number.

When there is more than one IXP in a city there’s a tendency for some ISPs to prefer one IXP over another, or in some cases even running their own. By CloudFlare being present at many exchanges, it gives us a unique and redundant view of the local Internet, it also allows us to have the best coverage of that local market. It brings the added advantage of being able to peer with certain regional networks over multiple exchanges and hence allowing for the best routing redundancy.

After five years of operating a network that's grown to over 75 PoPs globally we have joined our 100th Internet Exchange Point. We have a network that’s growing each and every day. Reaching 100 may be just a number; however, only one or two networks have had the ability to grow to that size. Our customers reap the benefits of that global reach. We won’t stop at 100. Far from it! Want to help us grow our network and increase our global peering? Check out available roles here.

Are you an ISP? Do you want to peer with us? Visit as13335.peeringdb.com or talk to us about an on-net cache.

As a Technical Support Engineer I get to work with many different members of the CloudFlare family and with customers from all around the world. Each day is very different to the next, and of course, some days stand out more than others.

Recently, I spent a bit of time working with a customer on an issue that was causing severe performance degradation on their website. The initial inquiry described a website that was yet to be publicised so it wasn’t seeing very much traffic. However, pages on the site were taking almost a minute to load. Throughout the day I worked with both the customer and other members of the support team to eliminate all the possible causes of the performance degradation. The main areas we looked at were:

• Did the customer’s server run out of resources?

• Was there anything preventing assets from being cached?

• Had the ISP of the customer caused traffic to be routed strangely?

• Were there any parts of the site that were noticeably slower than others?

• Did performance change when browsing the customer’s server directly?

After eliminating each of those points, it was unclear to us what was causing the issue. At this point we decided to make some configuration changes within the customer’s CloudFlare account. The main thing we did was to set up a few Page Rules to force everything under a certain URL pattern to be cached. The motivation for doing this was to ensure the site was fast for visitors while we continued to work on the performance issue behind the scenes. Another feature enabled for this CloudFlare Business customer was Railgun, to improve performance on any pages that we couldn’t set a Cache Everything page rule on.

After a careful examination of exactly what was running on the customer’s server we managed to find what was causing the issues — it was an installation of a caching programme on the origin that was taking up too much CPU, causing the web server to have to wait a long time before it could respond to any incoming requests. Upon disabling the programme the CPU load on the server dropped dramatically, this hugely improved performance.

By the end of the day the issue was solved and the customer had some extra CloudFlare features enabled to fully optimise the performance of their website — a great day working with a customer.

About Marty

Though he’d love to be at a Formula 1 race track any day of the week, we have convinced Marty to hang out in our UK office and help our customers. Previously a developer, Marty came to CloudFlare in the most awesome of ways: he was a customer. He’s now excited to be behind the scenes making the CloudFlare experience better for everyone. On his off days, he’ll be traveling between Formula 1 race tracks throughout the world.

The London Support Team (Marty is on the far left)

Do you have the enthusiasm to work with a technically motivated team? Do you enjoy working with a diverse global customer base? Are you somebody who uses their intuition to solve problems? CloudFlare is hiring Technical Support Engineers in London and San Francisco.