One problem arises from this design: what protects against a malevolent peer who decides to announce incorrect information? The damage caused by route hijacks can be major.

Routing Public Key Infrastructure (RPKI) is a framework created in 2008. Its goal is to provide a source of truth for Internet Resources (IP addresses) and ASes in signed cryptographically signed records called Route Origin Objects (ROA).

Recently, we’ve seen the significant threshold of two hundred thousand ROAs being passed. This represents a big step in making the Internet more secure against accidental and deliberate BGP tampering.

We have talked about RPKI in the past, but we thought it would be a good time for an update.

In a more technical context, the RPKI framework consists of two parts:

• IP addresses need to be cryptographically signed by their owners in a database managed by a Trust Anchor: Afrinic, APNIC, ARIN, LACNIC and RIPE NCC. Those five organizations are in charge of allocating Internet resources. The ROA indicates which Network Operator is allowed to announce the addresses using BGP.

• Network operators download the list of ROAs, perform the cryptographic checks and then apply filters on the prefixes they receive: this is called BGP Origin Validation.

The launch of the website isbgpsafeyet.com to test if your ISP correctly performs BGP Origin Validation was a success. Since launch, it has been visited more than five million times from over 223 countries and 13,000 unique networks (20% of the entire Internet), generating half a million BGP Origin Validation tests.

Many providers subsequently indicated on social media (for example, here or here) that they had an RPKI deployment in the works. This increase in Origin Validation by networks is increasing the security of the Internet globally.

The site’s test for Origin Validation consists of queries toward two addresses, one of which is behind an RPKI invalid prefix and the other behind an RPKI valid prefix. If the query towards the invalid succeeds, the test fails as the ISP does not implement Origin Validation. We counted the number of queries that failed to reach invalid.cloudflare.com. This also included a few thousand RIPE Atlas tests that were started by Cloudflare and various contributors, providing coverage for smaller networks.

Every month since launch we’ve seen that around 10 to 20 networks are deploying RPKI Origin Validation. Among the major providers we can build the following table:

With the help of many contributors, we have compiled a list of network operators and public statements at the top of the isbgpsafeyet.com page.

We excluded providers that manually blocked the traffic towards the prefix instead of using RPKI. Among the techniques we see are firewall filtering and manual prefix rejection. The filtering is often propagated to other customer ISPs. In a unique case, an ISP generated a “more-specific” blackhole route that leaked to multiple peers over the Internet.

The deployment of RPKI by major transit providers, also known as Tier 1, such as Cogent, GTT, Hurricane Electric, NTT and Telia made many downstream networks more secure without them having them deploying validation software.

Overall, we looked at the evolution of the successful tests per ASN, and we noticed a steady increase over the recent months of 8%.

Furthermore, when we probed the entire IPv4 space this month, using a similar technique to the isbgpsafeyet.com test, many more networks were not able to reach an RPKI invalid prefix than compared to the same period last year. This confirms an increase of RPKI Origin Validation deployment across all network operators. The picture below shows the IPv4 space behind a network with RPKI Origin Validation enabled in yellow and the active space in blue. It uses a Hilbert Curve to efficiently plot IP addresses: for example one /20 prefix (4096 IPs) is a pixel, a /16 prefix (65536 IPs) will form a 4x4 pixels square.

The more the yellow spreads, the safer the Internet becomes.

What does it mean exactly? If you were hijacking a prefix, the users behind the yellow space would likely not be affected. This also applies if you miss-sign your prefixes: you would not be able to reach the services or users behind the yellow space. Once RPKI is enabled everywhere, there will only be yellow squares.

Owners of IP addresses indicate the networks allowed to announce them. They do this by signing prefixes: they create Route Origin Objects (ROA). As of today, there are more than 200,000 ROAs. The distribution shows that the RIPE region is still leading in ROA count, then followed by the APNIC region.

2020 started with 172,000 records and the count is getting close to 200,000 at the beginning of November, approximately a quarter of all the Internet routes. Since last year, the database of ROAs grew by more than 70 percent, from 100,000 records, an average pace of 5% every month.

On the following graph of unique ROAs count per day, we can see two points that were followed by a change in ROA creation rate: 140/day, then 231/day, and since August, 351 new ROAs per day.

It is not yet clear what caused the increase in August.

In 2018 and 2019, Cloudflare was impacted by BGP route hijacks. Both could have been avoided with RPKI. Not long after the first incident, we started signing prefixes and developing RPKI software. It was necessary to make BGP safer, and we wanted to do more than talk about it. But we also needed enough networks to be deploying RPKI as well. By making deployment easier for everyone, we hoped to increase adoption.

The following is a reminder of what we built over the years around RPKI and how it grew.

OctoRPKI is Cloudflare’s open source RPKI Validation software. It periodically generates a JSON document of validated prefixes that we pass onto our routers using GoRTR. It generates most of the data behind the graphs here.

The latest version, 1.2.0, of OctoRPKI was released at the end of October. It implements important security fixes, better memory management and extended logging. This is the first validator to provide detailed information around cryptographically invalid records into Sentry and performance data in distributed tracing tools.GoRTR remains heavily used in production, including by transit providers. It can natively connect to other validators like rpki-client.

When we released our public rpki.json endpoint in early 2019, the idea was to enable anyone to see what Cloudflare was filtering.

The file is also used as a bootstrap by GoRTR, so that users can test a deployment. The file is cached on more than 200 data centers, ensuring quick and secure delivery of a list of valid prefixes, making RPKI more accessible for smaller networks and developers.

Between March 2019 and November 2020, the number of queries more than doubled and there are five times more networks querying this file.

The growth of queries follows approximately the rate of ROA creation (~5% per month).

A public RTR server is also available on rtr.rpki.cloudflare.com. It includes a plaintext endpoint on port 8282 and an SSH endpoint on port 8283. This allows us to test new versions of GoRTR before release.

Later in 2019, we also built a public dashboard where you can see in-depth RPKI validation. With a GraphQL API, you can now explore the validation data, test a list of prefixes, or see the status of the current routing table.

Currently, the API is used by BGPalerter, an open-source tool that detects routing issues (including hijacks!) from a stream of BGP updates.

Additionally, starting in November, you can access the historical data from May 2019. Data is computed daily and contains the unique records. The team behind the dashboard worked hard to provide a fast and accurate visualization of the daily ROA changes and the volumes of files changed over the day.

We believe RPKI is going to continue growing, and we would like to thank the hundreds of network engineers around the world who are making the Internet routing more secure by deploying RPKI.

25% of routes are signed and 20% of the Internet is doing origin validation and those numbers grow every day. We believe BGP will be safer before reaching 100% of deployment; for instance, once the remaining transit providers enable Origin Validation, it is unlikely a BGP hijack will make it to the front page of world news outlets.

While difficult to quantify, we believe that critical mass of protected resources will be reached in late 2021.

We will keep improving the tooling; OctoRPKI and GoRTR are open-source, and we welcome contributions. In the near future, we plan on releasing a packaged version of GoRTR that can be directly installed on certain routers. Stay tuned!

The Internet is too vital to allow this known problem to continue any longer. It's time networks prevented leaks and hijacks from having any impact. It's time to make BGP safe. No more excuses.

Border Gateway Protocol (BGP), a protocol to exchange routes has existed and evolved since the 1980s. Over the years it has had security features. The most notable security addition is Resource Public Key Infrastructure (RPKI), a security framework for routing. It has been the subject of a few blog posts following our deployment in mid-2018.

Today, the industry considers RPKI mature enough for widespread use, with a sufficient ecosystem of software and tools, including tools we've written and open sourced. We have fully deployed Origin Validation on all our BGP sessions with our peers and signed our prefixes.

However, the Internet can only be safe if the major network operators deploy RPKI. Those networks have the ability to spread a leak or hijack far and wide and it's vital that they take a part in stamping out the scourge of BGP problems whether inadvertent or deliberate.

Many like AT&T and Telia pioneered global deployments of RPKI in 2019. They were successfully followed by Cogent and NTT in 2020. Hundreds networks of all sizes have done a tremendous job over the last few years but there is still work to be done.

If we observe the customer-cones of the networks that have deployed RPKI, we see around 50% of the Internet is more protected against route leaks. That's great, but it's nothing like enough.

Today, we are releasing isBGPSafeYet.com, a website to track deployments and filtering of invalid routes by the major networks.

We are hoping this will help the community and we will crowdsource the information on the website. The source code is available on GitHub, we welcome suggestions and contributions.

We expect this initiative will make RPKI more accessible to everyone and ultimately will reduce the impact of route leaks. Share the message with your Internet Service Providers (ISP), hosting providers, transit networks to build a safer Internet.

Additionally, to monitor and test deployments, we decided to announce two bad prefixes from our 200+ data centers and via the 233+ Internet Exchange Points (IXPs) we are connected to:

• 103.21.244.0/24

• 2606:4700:7000::/48

Both these prefixes should be considered invalid and should not be routed by your provider if RPKI is implemented within their network. This makes it easy to demonstrate how far a bad route can go, and test whether RPKI is working in the real world.

A Route Origin Authorization for 103.21.244.0/24 on rpki.cloudflare.com

In the test you can run on isBGPSafeYet.com, your browser will attempt to fetch two pages: the first one valid.rpki.cloudflare.com, is behind an RPKI-valid prefix and the second one, invalid.rpki.cloudflare.com, is behind the RPKI-invalid prefix.

The test has two outcomes:

• If both pages were correctly fetched, your ISP accepted the invalid route. It does not implement RPKI.

• If only valid.rpki.cloudflare.com was fetched, your ISP implements RPKI. You will be less sensitive to route-leaks.

a simple test of RPKI invalid reachability

We will be performing tests using those prefixes to check for propagation. Traceroutes and probing helped us in the past by creating visualizations of deployment.

A simple indicator is the number of networks sending the accepted route to their peers and collectors:

Routing status from online route collection tool RIPE Stat

In December 2019, we released a Hilbert curve map of the IPv4 address space. Every pixel represents a /20 prefix. If a dot is yellow, the prefix responded only to the probe from a RPKI-valid IP space. If it is blue, the prefix responded to probes from both RPKI valid and invalid IP space.

To summarize, the yellow areas are IP space behind networks that drop RPKI invalid prefixes. The Internet isn't safe until the blue becomes yellow.

Hilbert Curve Map of IP address space behind networks filtering RPKI invalid prefixes

Last but not least, we would like to thank every network that has already deployed RPKI and every developer that contributed to validator-software code bases. The last two years have shown that the Internet can become safer and we are looking forward to the day where we can call route leaks and hijacks an incident of the past.

Traffic towards video conferencing, streaming services and news, e-commerce websites has surged. We've seen growth in traffic from residential broadband networks, and a slowing of traffic from businesses and universities.

The Cloudflare team is fully operational and the Network Operating Center (NOC) is watching the changing traffic patterns in the more than 200 cities in which we operate hardware.

Big changes in Internet traffic aren't unusual. They often occur around large sporting events like the Olympics or World Cup, cultural events like the Eurovision Song Contest and even during Ramadan at the breaking of the fast each day.

The Internet was built to cope with an ever changing environment. In fact, it was literally created, tested, debugged and designed to deal with changing load patterns.

Over the last few weeks, the Cloudflare Network team has noticed some new patterns and we wanted to share a few of them with you.

Last Friday evening, the US President announced a State of Emergency in the United States. Not so long after, our US data centers served 20% more traffic than usual. The red line shows Friday, the grey lines the preceding days for comparison.

On the Sunday, March 15, the Dutch government announced on the radio at 1730 local time closures of the non-essential business (1630 UTC). A sharp dip in the regular Sunday traffic followed:

The French president made two national announcements, on March 12 (pink curve) and March 16 (red curve) at 2000 local time (1900 UTC). The lockdown announcement on March 16 caused French traffic to dip by half followed by a spike:

Italy has seen a 20-40% increase in daily traffic since the lockdown:

With universities closing, some national research networks are remaining (almost) as quiet as a weekend (in purple). Current day in red and previous days in grey (overlaps with previous week):

The Internet Exchange Points, a key part of the Internet infrastructure, where Internet service providers and content providers can exchange data directly (rather than via a third party) have also seen spikes in traffic. Many provide public traffic graphs.

In Amsterdam (AMS-IX), London (LINX) and Frankfurt (DE-CIX), around 10-20% increase is seen around March 9th:

In Milan (MXP-IX), the Exchange point shows a 40% increase on Wednesday, 9th of March 2020, the day of the quarantine:

In Asia, in Hong Kong (HKIX), we can observe a faster increase since the end of January which likely corresponds to the Hubei lockdown on the January 23:

The emergency has a non-negligible impact on Internet services and our lives. Although it is difficult to quantify exactly the increase, we observe numbers from 10% to 40% depending on the region and the state of government action in those regions.

Even though from time to time individual services, such as a web site or an app, have outages the core of the Internet is robust. Traffic is shifting from corporate and university networks to residential broadband, but the Internet was designed for change.

Check back on the Cloudflare blog for further updates and insights.

Today’s article is going to cover our experience and the tools we are using. As a brief reminder, RPKI is a framework that allows networks to deploy route filtering using cryptography-validated information. Picture TLS certificates for IP addresses and Autonomous System Numbers (ASNs)

We validate our IP routes. This means, as a 1.1.1.1 DNS resolver user, you are less likely to be victim of cache poisoning. We signed our IP routes. This means a user browsing the websites on Cloudflare’s network are unlikely to experience route hijacks.

All our Points of Presence which have a router compatible with The Resource Public Key Infrastructure (RPKI) to Router Protocol (RTR protocol) are connected to our custom software called GoRTR and are now filtering invalid routes. The deployment amounts to around 70% of our network.

We received many questions regarding the amount of invalid traffic. Our estimates go from 100Mbps to 2Gbps. The spread remains high due to the difficulty of accounting the traffic that would go towards an invalid route. At our scale, it is a drop in the ocean of packets.

It is worth mentioning that one provider experienced a substantial traffic shift due to many regional IPs announced as smaller subnets and not included in the Route Origin Attestation (ROA), a key resource of the RPKI environment. We noticed many important networks announcing invalids on Internet Exchange Points. We emailed Network Operating Centers and were happy to see the records got corrected in a matter of days.

There are two components: GoRTR and OctoRPKI. The big picture of our setup.

OctoRPKI and GoRTR ecosystem diagram

As our network keeps growing, we needed a scalable solution to send the list of validated ROAs to our edge. Ideally using our CDN for caching the resources. We created GoRTR, a small Go application which will fetch a JSON file using a format used by RPKI Validators. We made sure that it could be used with various servers and implemented cryptographic checks to ensure an untampered distribution. By default, it will fetch the list of prefixes available at rpki.cloudflare.com/rpki.json.

You do not need to run your own validation software, just plug your RPKI-enabled router to GoRTR and start filtering. GoRTR will also expose a web server and Prometheus-type metrics. Nonetheless, if you want to run your own validation software, keep reading because OctoRPKI might be the solution for you. You can start GoRTR with Docker and plug your router to the port 8282 to receive a RTR feed:

$ docker run -ti -p 8282:8282 cloudflare/gortr

If you want to control the validation of routes, we got you covered!

After a few months work, we released OctoRPKI. Behind an octopus logo, there is a RPKI Relying Party written in Go.

The current ecosystem did not bring us joy, we needed RPKI Repository Delta Protocol (RRDP), a fully-fledged monitored validator subsystem and the ability to push only the final computed data to our storage cluster. We decided to build it in Go for multiple reasons. In exchange of a slightly increased overhead, we gained flexibility. There is a great amount of cryptographic resources and libraries for the Go language. And we are hoping this would benefit an already important community.

The repository contains OctoRPKI plus a library to decode certificates, ROAs, manifests, etc. The code can start a synchronization with rsync and/or RRDP.

Similarly to GoRTR, it embeds an API to get the latest results of the synchronization and a Prometheus metrics endpoint to deploy alerting. Internally, we monitor the state of the validation using Prometheus. It is also providing data for a GraphQL API that will be released soon.

The docker image provides a way to instantly run OctoRPKI and comes complete with four of the five Trust Anchor Locator (TAL) files needed to operate (AFRINIC, APNIC, LACNIC, and RIPE). The fifth is the ARIN TAL, so don’t forget to add the TAL from ARIN. You can start the process of downloading it at the following address: www.arin.net/resources/rpki/tal.html.

$ docker run -ti \
     -p 8080:8080 \
     -v $PWD/cache:/cache \
     -v path_to_arin_tal:/tals/arin.tal 
   cloudflare/octorpki

The results will be available on http://localhost:8080/output.json. To plug GoRTR to this file, use the -cache URL argument (you will need to load the public key which signs some fields in JSON).

$ gortr -cache http://localhost:8080/output.json -verify.key path-to-key

In conclusion, we hope these tools will help you deploy RPKI easily. Routing security is an important subject that should not be avoided. Looking forward to more networks joining this initiative and making the Internet more secure.We learned a lot about RPKI, from signing the routes on the five registries to validating cryptographically more than 60000 resources in a few seconds. We will keep working on making OctoRPKI and GoRTR always more stable and reliable. We look forward to your feedback!

We're withholding answers until the start of the new year, to give you a chance to solve them without spoilers. Before we reveal the answers; if you manage to solve them, we'll be giving the first 5 people to get the answers right some Cloudflare swag. Fill out your answers and details using this form so we know where to send it.

Have fun!

UPDATE: This quiz is now closed. Thank you to everyone who's played. We have received many responses, 15 of which got all the answers right; we will shortly be sending out some swag to those who got the answers right.

NOTE: Hints, now followed with solutions, are below the questions, avoid scrolling too far if you want to avoid any spoilers.

Client says hello, as follows:

00000c07ac01784f437dbfc70800450000f2560140004006db58ac1020c843c7f80cd1f701bbc8b2af3449b598758018102a72a700000101080a675bce16787abd8716030100b9010000b503035c1ea569d5f64df3d8630de8bdddd1152e75f528ae577d2436949ce8deb7108600004400ffc02cc02bc024c023c00ac009c008c030c02fc028c027c014c013c012009f009e006b0067003900330016009d009c003d003c0035002f000a00af00ae008d008c008b010000480000000b000900000663666c2e7265000a00080006001700180019000b00020100000d0012001004010201050106010403020305030603000500050100000000001200000017000052305655494338795157524c656d6443436c5246574651675430346754456c4f52564d674e434242546b51674e513d3d

[Raw puzzle without text wrap]

A user has an authenticator device to generate one time passwords for logins to their banking website. The implementation contains a fatal flaw.

At the following times, the following codes are generated (all in GMT/UTC):

• Friday, 21 December 2018 16:29:28 - 084342

• Saturday, 22 December 2018 13:11:53 - 411907

• Tuesday, 25 December 2018 12:15:03 - 617041

What code will be generated at precisely midnight of the 1st of January 2019?

At Cloudflare, we just setup RPKI: we signed a few hundred prefixes in order to reduce route leaks. But some of the prefixes hide a secret message. Find the ROAs that look different, decode the word!

This challenge has 3 hints, as follows:

• Challenge is based on a network capture

• https://blog.cloudflare.com/encrypted-sni/

• What's weird about the Frame?

The Time-Based One-Time Password Algorithm is described in RFC 6238, which was based of RFC4226 (providing an algorithm for HOTP). The TOTP algorithm requires input of two important parameters, the time and a shared secret - could one be missing?

The implementation used to generate the TOTP codes for the challenge uses SHA-1 as a digest algorithm.

Note: This challenge will no longer be valid after mid-January 2019.

This challenge has 4 hints, as follows:

• Hint #0: Four or six? Probably six.

• Hint #1: If only there was a way of listing only our IPs!

• Hint #2: What is the only part of the ROA where we can hide information into

• Hint #3: Subtract the reserve, the char will show itself

If you prefer video form, someone has created a YouTube video of the how to solve the problems, else the written solutions are below:

The string was (mostly) a capture from Wireshark of a Client Hello frame in TLS 1.2 handshake; as such, it reveals the Server Name where the connection is intended to go; in this case cfl.re.

There is a string suffixed to this hex stream which shouldn't be there; it's a base64 encoded string 




R0VUIC8yQWRLemdCClRFWFQgT04gTElORVMgNCBBTkQgNQ==. Decoding this string reveals:

GET /2AdKzgBTEXT ON LINES 4 AND 5

Accordingly; https://cfl.re/2AdKzgB redirects to https://www.cloudflare.com/robots.txt; on lines 4 and 5 is the phrase: "Dear robot be nice".

The GET request would obviously ordinarily not be appended to the Client Hello like this; however SNI information would be. You can find more about the work Cloudflare is doing to encrypt such information, so attackers cannot see which site you're visiting, in the following post: Encrypting SNI: Fixing One of the Core Internet Bugs

For this part, I'm going to use the pyotp library to demonstrate how the challenge is set-up:

>>> import pyotp>>> totp = pyotp.TOTP('')>>> print totp.at(1545409768)084342>>> print totp.at(1545484313)411907>>> print totp.at(1545740103)617041

Note that the argument to the TOTP function is set to an empty string, this means that there is no secret in place; and the one time passwords are generated solely from a hash of the time. Accordingly, a TOTP with the timestamp generated at midnight on New Year is 301554.

Whilst this may seem like a somewhat incredulous position for a developer to end up in - searching GitHub, I was even able to find implementations that used the default secret (base32secret3232) for all users wanting to authenticate to a website. This means that any other user's One Time Password is valid for any other account, and the secret could likely be breached fairly easily (as it isn't randomly generated).

Cloudflare can only generate ROAs based on their prefixes. The IPv6 prefixes are listed here: https://cloudflare.com/ips-v6.

Using any RPKI validated prefix list (https://rpki.cloudflare.com/rpki.json, or using the GUI of the RIPE’s RPKI Validator), test out our IPv6 prefixes. Some of them will appear coming from Reserved ASNs for Private Use:

• 2803:f800:cfcf:cfcf:cfcf:cfcf:cfcf:1 - B

• 2803:f800:cfcf:cfcf:cfcf:cfcf:cfcf:2 - R

• 2803:f800:cfcf:cfcf:cfcf:cfcf:cfcf:3 - A

• 2803:f800:cfcf:cfcf:cfcf:cfcf:cfcf:4 - V

• 2803:f800:cfcf:cfcf:cfcf:cfcf:cfcf:5 - O

Subtract 4200000000, it will give you one byte for each character of the secret word.

Repeat until the word is decoded.

Interested in helping build a better internet and drive security online? Cloudflare is hiring.

This article will talk about our approach to network security using technologies like RPKI to sign Internet routes and protect our users and customers from route hijacks and misconfigurations. We are proud to announce we have started deploying active filtering by using RPKI for routing decisions and signing our routes.

Back in April, articles including our blog post on BGP and route-leaks were reported in the news, highlighting how IP addresses can be redirected maliciously or by mistake. While enormous, the underlying routing infrastructure, the bedrock of the Internet, has remained mostly unsecured.

At Cloudflare, we decided to secure our part of the Internet by protecting our customers and everyone using our services including our recursive resolver 1.1.1.1.

A prefix is a range of IP addresses, for instance, 10.0.0.0/24, whose first address is 10.0.0.0 and the last one is 10.0.0.255. A computer or a server usually have one. A router creates a list of reachable prefixes called a routing table and uses this routing table to transport packets from a source to a destination.  

On the Internet, network devices exchange routes via a protocol called BGP (Border Gateway Protocol). BGP enables a map of the interconnections on the Internet so that packets can be sent across different networks to reach their final destination. BGP binds the separate networks together into the Internet.

This dynamic protocol is also what makes Internet so resilient by providing multiple paths in case a router on the way fails. A BGP announcement is usually composed of a prefix which can be reached at a destination and was originated by an Autonomous System Number (ASN).

IP addresses and Autonomous Systems Numbers are allocated by five Regional Internet Registries (RIR): Afrinic for Africa, APNIC for Asia-Pacific, ARIN for North America, LACNIC for Central and South America and RIPE for Europe, Middle-East and Russia. Each one operates independently.

With more than 700,000 IPv4 routes and 40,000 IPv6 routes announced by all Internet actors, it is difficult to know who owns which resource.

There is no simple relationship between the entity that has a prefix assigned, the one that announces it with an ASN and the ones that receive or send packets with these IP addresses. An entity owning 10.0.0.0/8 may be delegating a subset 10.1.0.0/24 of that space to another operator while being announced through the AS of another entity.

Thereby, a route leak or a route hijack is defined as the illegitimate advertisement of an IP space. The term route hijack implies a malicious purpose while a route leak usually happens because of a misconfiguration.

A change in route will cause the traffic to be redirected via other networks. Unencrypted traffic can be read and modified. HTTP webpages and DNS without DNSSEC are sensitive to these exploits.

You can learn more about BGP Hijacking in our Learning Center.

When an illegitimate announcement is detected by a peer, they usually notify the origin and reconfigure their network to reject the invalid route.Unfortunately, the time to detect and act upon may take from a few minutes up to a few days, more than enough to steal cryptocurrencies, poison a DNS cache or make a website unavailable.

A few systems exist to document and prevent illegitimate BGP announcements.

The Internet Routing Registries (IRR) are semi-public databases used by network operators to register their assigned Internet resources. Some database maintainers do not check whether the entry was actually made by the owner, nor check if the prefix has been transferred to somebody else. This makes them prone to error and not completely reliable.

Resource Public Key Infrastructure (RPKI) is similar to the IRR “route” objects, but adding the authentication with cryptography.

Here’s how it works: each RIR has a root certificate. They can generate a signed certificate for a Local Internet Registry (LIR, a.k.a. a network operator) with all the resources they are assigned (IPs and ASNs). The LIR then signs the prefix containing the origin AS that they intend to use: a ROA (Route Object Authorization) is created. ROAs are just simple X.509 certificates.

If you are used to SSL/TLS certificates used by browsers to authenticate the holder of a website, then ROAs are the equivalent in the Internet routing world.

Each network operator owning and managing Internet resources (IP addresses, Autonomous System Numbers) has access to their Regional Internet Registry portal. Signing their prefixes through the portal or the API of their RIR is the easiest way to start with RPKI.

Because of our global presence, Cloudflare has resources in each of the 5 RIR regions. With more than 800 prefix announcements over different ASNs, the first step was to ensure the prefixes we were going to sign were correctly announced.

We started by signing our less used prefixes, checked if the traffic levels remained the same and then signed more prefixes. Today about 25% of Cloudflare prefixes are signed. This includes our critical DNS servers and our public 1.1.1.1 resolver.

Signing the prefixes is one thing. But ensuring that the prefixes we receive from our peers match their certificates is another.

The first part is validating the certificate chain. It is done by synchronizing the RIR databases of ROAs through rsync (although there are some new proposals regarding distribution over HTTPS), then check the signature of every ROA against the RIR’s certificate public key. Once the valid records are known, this information is sent to the routers.

Major vendors support a protocol called RPKI to Router Protocol (abbreviated as RTR). This is a simple protocol for passing a list of valid prefixes with their origin ASN and expected mask length. However, while the RFC defines 4 different secure transport methods, vendors have only implemented the insecure one. Routes sent in clear text over TCP can be tampered with.

With more than 150 routers over the globe, it would be unsafe to rely on these cleartext TCP sessions over the insecure and lossy Internet to our validator. We needed local distribution on a link we know secure and reliable.

One option we considered was to install an RPKI RTR server and a validator in each of our 150+ datacenters, but doing so would cause a significant increase in operational cost and reduce debugging capabilities.

We needed an easier way of passing an RPKI cache securely. After some system design sessions, we settled on distributing the list of valid routes from a central validator securely, distribute it via our own Content Delivery Network and use a lightweight local RTR server. This server fetches the cache file over HTTPS and passes the routes over RTR.

Rolling out this system on all our PoPs using automation was straightforward and we are progressively moving towards enforcing strict validation of RPKI signed routes everywhere.

To encourage adoption of Route Origin Validation on the Internet, we also want to provide this service to everyone, for free. You can already download our RTR server with the cache behind Cloudflare. Just configure your Juniper or Cisco router. And if you do not want to use our file of prefixes, it is compatible with the RIPE RPKI Validator Export format.

We are also working on providing a public RTR server using our own Spectrum service so that you will not have to install anything, just make sure you peer with us! Cloudflare is present on many Internet Exchange Points so we are one hop away from most routers.

A few months ago, Nick Sullivan introduced our new Nimbus Certificate Transparency Log.

In order to track the emitted certificates in the RPKI, our Crypto team created a new Certificate Transparency Log called Cirrus which includes the five RIRs root certificates as trust anchors. Certificate transparency is a great tool for detecting bad behavior in the RPKI because it keeps a permanent record of all valid certificates that are submitted to it in an append-only database that can’t be modified without detection. It also enables users to download the entire set of certificates via an HTTP API.

We use services like BGPmon and other public observation services extensively to ensure quick action if some of our prefixes are leaked. We also have internal BGP and BMP collectors, aggregating more than 60 millions routes and processing live updates.

Our filters make use of this live feed to ensure we are alerted when a suspicious route appears.

The latest statistics suggest that around 8.7% of the IPv4 Internet routes are signed with RPKI, but only 0.5% of all the networks apply strict RPKI validation.Even with RPKI validation enforced, a BGP actor could still impersonate your origin AS and advertise your BGP route through a malicious router configuration.

However that can be partially solved by denser interconnections, that Cloudflare already has through an extensive network of private and public interconnections.To be fully effective, RPKI must be deployed by multiple major network operators.

As said by Job Snijders from NTT Communications, who’s been at the forefront of the efforts of securing Internet routing:

If the Internet's major content providers use RPKI and validate routes, the impact of BGP attacks is greatly reduced because protected paths are formed back and forth. It'll only take a small specific group of densely connected organizations to positively impact the Internet experience for billions of end users.

RPKI is not a bullet-proof solution to securing all routing on the Internet, however it represents the first milestone in moving from trust based to authentication based routing. Our intention is to demonstrate that it can be done simply and cost efficiently. We are inviting operators of critical Internet infrastructure to follow us in a large scale deployment.

Subscribe to the blog for daily updates on our announcements.

Cloudflare’s mission is to help build a better Internet. To do that we operate datacenters across the globe. By having datacenters close to end user we provide a fast, secure experience for everyone. Today I’d like to talk about our datacenters in Africa and our plans to serve a population of 1.2 billion people over 58 countries.

Internet penetration in developed countries skyrocketed since the 2000s, Internet usage is growing rapidly across Africa. We are seeing a 4% to 7% increase in traffic month on month. As of July 2018, we have 8 datacenters on the African Continent:

• Angola

• Djibouti

• Egypt

• Kenya

• Mauritius

• Three in South Africa

While we see changes on the horizon, the majority of Internet content providers are located in North America and Western Europe. This means that only the billion people living in those areas close to the content they are trying to reach. When it comes to Africa, submarine cables will usually bring back the packets to Europe to hubs like Marseille, Paris, London, Lisbon and sometimes Frankfurt or Amsterdam, adding precious milliseconds, slowing down communications.

By setting up datacenters on the African continent, Cloudflare is able to serve content locally, increasing download speed in the region, improving the end-user experience, and ultimately increasing Internet usage.

We wondered if the Internet usage increased when we set-up a datacenter in a region which was previously not well served.

It was surprising to see a fast growth in the following months in each of the countries we turn on our equipment. We took into account bandwidth and also quantity of information exchanged. The increase in bandwidth usually leads to an increase in usage.

Bandwidth is the maximum rate information can be transferred over a link. The interpretation of maximum depends on the type of transmission.

The explanation why the rate is dependent on latency comes from the TCP protocol specification on which run most of the web applications. Every transmission ends with an acknowledgement, to determine if the data was correctly received. The next transmission will begin after the sender receives the acknowledgement. This means while the acknowledgement is transmitting, nothing is actually being received. The difference of data received over time is the transfer rate.

This is summarized in the following diagram:

With the most common algorithms, the amount of data sent before an acknowledgment increases until an error appears. Any link will drop packets: whether there is a transmission issue (wireless and perturbations) or a processing issue (router dropping packets to reduce load). This contributes never reaching the maximum bandwidth of a link. This is why above 80 milliseconds latency, performances start to worsen drastically. Coast to coast in the USA is around 70 milliseconds. Paris to London is 10 milliseconds. Satellites connections implement proxies to allow sustainable throughput despite 600 milliseconds round-time-trip.

The explanation is behavioral. If an Internet connection is slow and frustrating, chances are we will use it only if we are forced to. While getting access to the content immediately, people are likely to fetch more content and click on links.

The following chart shows the growth of requests from a country after every datacenter turn up. Traffic is normalized on the latest value (July 2018).

Djibouti shows the biggest increase in traffic after the launch.

For the rest of the continent, we are seeing a steady increase in delivered traffic over the last four years.

Overall, Sierra Leone is the country which grew the most at around 8% per month. At the opposite scale, Algeria only increased its traffic by 2% per month. The mean of all those countries is 6.2% per month, which is also the Internet traffic growth of South Africa.

In comparison to Europe, North America, it will take, at the current rate, 4 years and 3 months for Africa to reach today’s traffic levels of those two continents. However, if Europe, North America keeps its current 4% growth rate, then the African continent will take approximately eight to twelve years to catch up.

Please note these estimations are not perfectly representative as Cloudflare only see a part of the Internet and the numbers also includes our growing base of customers.

The units on the vertical axis represent the growth based on the initial ratio between Europe/USA/Canada and Africa.

We analyzed the latencies between Europe and African IP addresses that hit our edge. On the following Demer map from Vasco Asturiano, the area of the country is exponentially proportional to the response time in milliseconds.

All the coastal Africa is well connected with submarines cables. The only exception is Eritrea, where the main provider (EriTel) which uses two satellite providers for its outbound links.

The island of Saint Helena is on the path of the future SAex cable, but the only connection at the moment for its 5000 inhabitants is through satellite, causing a latency of 600ms.Central African Republic also uses satellite or connections through Cameroon.

On average a packet round trip from Northern Africa to Europe will take around 50-100ms while a round trip from the South will take 250ms.

Regarding inter-African routing, Africa has only a limited number of cities where interconnection happens successfully: Johannesburg in South Africa, Nairobi in Kenya, and Djibouti with datacenters and internet exchange points. The rest of the continent is mostly split between providers which will interconnect in Europe or Asia. Chances are, a user in Cameroon talking to a user in Ghana will likely go through Paris.

Using RIPE Atlas, we are able to show inter-provider routing usually happens in Europe. Only one traceroute showed packets being exchanged in South Africa.

Traceroute to two Ghana Atlas probes from other Atlas Probes

To Ghana only via Europe

To Ghana with one through South Africa

Cloudflare and many network operators are using extensively RIPE Atlas to determine performance and troubleshoot issues. If you want to help us improve Internet quality in Africa, become a probe host here.

What you may be wondering is: what are the countries in Europe that route the African traffic? We are using an anycast network ; when a user fetch a website on Cloudflare, the packets will take the shortest path seen by the router to reach its destination. In telecommunications, the shortest path does necessarily mean geographical proximity. The selection metric of a path usually involve cost and performance.

An anycast address will have the same metric everywhere on our side. This mean a service provider having links to London and Paris will see our IP addresses originating from London datacenter and Paris datacenter. The choice between the two will only be depending on metrics set by the provider: will it cost more to reach London?

Cloudflare has a significant number of points of presence to be able to show small metric differences. Due to the density of its network in Europe, it is common to have datacenters one hop away from each others (London and Paris, Paris and Amsterdam). Our analysis of the next-hop will show a provider’s preference.

In the case of Africa, a provider can rent capacity on submarine cable to a city, for instance Lisbon, Paris, London. Ideally, the provider wants to maximize the resources it can obtain without adding more hops (costlier).

Paris, London, Amsterdam and Frankfurt have a lot of content providers and interconnections options, the differences are going to be on the content. One major bias will be the language: France will have more francophone content hosted in Parisian datacenters, it also helps running operations to if both parties speak French.

As an example case: if Paris is chosen, for the content that can only be found in London, another provider will carry the packets to London from Paris, adding one hop to the path, reducing preference. We will see the landing point in Paris.

Why isn’t the provider getting a link to London? On thing to know is any Internet link has a flat rate price plus a variable rate based on consumption.

Bandwidth within major European cities is among the cheapest in the world. The flat rate for submarine capacity is high, as a result, the quantity of content that is available from London for cheaper than Paris has to make up the difference.

If you are interested to know more about the costs of bandwidth around the world, check out this article by Nitin Rao.

The following map represents the most common Cloudflare datacenter reached for each country. The countries filled with color are where Cloudflare datacenters are. The border color indicates the destination country.

As aforementioned, we can notice most of the French-speaking countries will tend to go towards our datacenter in Paris. South Africa remains well connected with its neighbors.

Previously, we mentioned there are some content providers in South Africa and Djibouti. We took a look at the popular news and banking websites.

Over the top 200 origins of the websites (Alexa ranking) for African countries in 2017, only 42 were attached to an African country and only 10 among those were serving non-education/government content (news websites, banks).They were located mostly in South Africa and Zimbabwe.

We also took a look at our 10 millions zones behind Cloudflare. The ones that are using Afrinic IPs are hosted in the following countries:

Only four countries out of sixty four are using IPv6 in a measurable way. Egypt (1%), Kenya (2%), Gabon (2%) and Zimbabwe (9%). In each of these countries we found that only one provider that deployed IPv6 at any significant scale. Since the beginning of the year, IPv6 usage in Egypt has doubled. In Gabon, a major provider rolled IPv6 at the end of December 2017.

As Mathieu Paonessa from Group Vivendi Afrique told us, “Gabon went from 0% to 2% IPv6 in only 9 months, following the opening of our CanalBox Gabon FTTH service.”

The timeline is similar with a small ISP in Kenya and has been increasing steadily. We can maybe hope they double their IPv6 usage by the end of 2018.

Compared to the rest of the world, Belgium is still ahead with 37%, followed by India at 35%. Most European countries are above 10%. USA is 22% and Canada is 15%.

One explanation for the small number of deployments is the size of the remaining IPv4 pool of addresses.

As of July 2018, there were 36,245 /24 available (approximately 9.2 millions IPs). The total Afrinic size is 6 /8 (approximately 100 millions IPs). 9% are left.

The following graphs shows the allocations and usage of each /24 in the Afrinic allocations.

This is the Hilbert graph of Afrinic IPv4 exhaustion. It was inspired by Ben Cartwright-Cox’s blog post and used masscan and ipv4-heatmap.

Prefix

Announced (yellow) vs Available (green) vs Reserved (red)

Response to ICMP (blue = low reply, red = all reply)

41.0.0.0/8

102.0.0.0/8

105.0.0.0/8

154.0.0.0/8

196.0.0.0/8

197.0.0.0/8

We notice that even if allocated, some blocks remain dark. 102.0.0.0/8 remains the range with the most IP space left in the world.

Africa is the largest continent on earth, yet it is very lagging behind on Internet connectivity and traffic levels. Historically it's been entirely relying on its European interconnections. However we predict that its traffic will outgrow EU and US by 2027. This will be enabled by the fast deployment of local content, IPv6, and alternative submarine cables.

We are working on deploying even more points of presence in Africa and increasing our current capacity in the existing ones. Fifteen new locations in the African continent are part of our global expansion plans. These include: Algeria (Algiers), Cameroon (Yaoundé), Congo (Kinshasa), Côte d’Ivoire (Abidjan), Egypt (Alexandria), Ghana (Accra), Kenya (Nairobi), La Réunion (Sainte-Marie), Madagascar (Antananarivo), Morocco (Casablanca), Nigeria (Lagos), Tanzania (Dar es Salaam), Tunisia (Tunis), Uganda (Kampala), and Zimbabwe (Harare)

Header image source

CC BY 2.0 image by elhombredenegro

The Internet is composed of routes. For our DNS resolver 1.1.1.1 , we tell the world that all the IPs in the range 1.1.1.0 to 1.1.1.255 can be accessed at any Cloudflare PoP.

For the people who do not have a direct link to our routers, they receive the route via transit providers, who will deliver packets to those addresses as they are connected to Cloudflare and the rest of the Internet.

This is the normal way the Internet operates.

There are authorities (Regional Internet Registries, or RIRs) in charge of distributing IP addresses in order to avoid people using the same address space. Those are IANA, RIPE, ARIN, LACNIC, APNIC and AFRINIC.

CC BY 2.0 image by Magnus D

The broad definition of a BGP leak would be IP space that is announced by somebody not allowed by the owner of the space. When a transit provider picks up Cloudflare's announcement of 1.1.1.0/24 and announces it to the Internet, we allow them to do so. They are also verifying using the RIR information that only Cloudflare can announce it to them.

Although it can get tricky checking all the announcements. Especially when there are 700,000+ routes on the Internet and chains of providers exchanging traffic between each other.

By their nature, route leaks are localized. The more locally connected you are, the smaller the risk of accepting a leaked route is.In order to be accepted over a legitimate route, the route has to be either:

• A smaller prefix (10.0.0.1/32 = 1 IP vs 10.0.0.0/24 = 256 IPs)

• Have better metrics than a prefix with the same length (shorter path)

The cause of a BGP leak is usually a configuration mistake: a router suddenly announces the IPs it learned. Or smaller prefixes used internally for traffic engineering suddenly becoming public.

But sometimes it is done with a malicious intent. The prefix can be re-routed through in order to passively analyze the data. Or somebody can also set-up a service to reply illegitimately instead. This has happened before.

Cloudflare maintains a range of BGP collectors gathering BGP information from hundreds of routers around the planet.

Between approximately 11:05:00 UTC and 12:55:00 UTC today we saw the following announcements:

BGP4MP|04/24/18 11:05:42|A|205.251.199.0/24|10297
BGP4MP|04/24/18 11:05:42|A|205.251.197.0/24|10297
BGP4MP|04/24/18 11:05:42|A|205.251.195.0/24|10297
BGP4MP|04/24/18 11:05:42|A|205.251.193.0/24|10297
BGP4MP|04/24/18 11:05:42|A|205.251.192.0/24|10297
...
BGP4MP|04/24/18 11:05:54|A|205.251.197.0/24|4826,6939,10297

Those are more specifics announcements of the ranges:

• 205.251.192.0/23

• 205.251.194.0/23

• 205.251.196.0/23

• 205.251.198.0/23

This IP space is allocated to Amazon (AS16509). But the ASN that announced it was eNet Inc (AS10297) to their peers and forwarded to Hurricane Electric (AS6939).

Those IPs are for Route53 Amazon DNS servers. When you query for one of their client zones, those servers will reply.

During the two hours leak the servers on the IP range only responded to queries for myetherwallet.com. As some people noticed SERVFAIL.

Any DNS resolver that was asked for names handled by Route53 would ask the authoritative servers that had been taken over via the BGP leak. This poisoned DNS resolvers whose routers had accepted the route.

This included Cloudflare DNS resolver 1.1.1.1. We were affected in Chicago, Sydney, Melbourne, Perth, Brisbane, Cebu, Bangkok, Auckland, Muscat, Djibouti and Manilla. In the rest of the world, 1.1.1.1 worked normally.

BGP hijack this morning affected Amazon DNS. eNet (AS10297) of Columbus, OH announced the following more-specifics of Amazon routes from 11:05 to 13:03 UTC today:205.251.192.0/24205.251.193.0/24205.251.195.0/24205.251.197.0/24205.251.199.0/24

— InternetIntelligence (@InternetIntel) April 24, 2018

Correction: the BGP hijack this morning was against AWS DNS not Google DNS. https://t.co/gp3VLbImpX

— InternetIntelligence (@InternetIntel) April 24, 2018

For instance, the following query will return you legitimate Amazon IPs:

$ dig +short myetherwallet.com @205.251.195.239
54.192.146.xx

But during the hijack, it returned IPs associated with a Russian provider (AS48693 and AS41995). You did not need to accept the hijacked route to be victim of the attack, just use a DNS resolver that had been poisoned.

If you were using HTTPS, the fake website would display a TLS certificate signed by an unknown authority (the domain listed in the certificate was correct but it was self-signed). The only way for this attack to work would be to continue and accept the wrong certificate. From that point on, everything you send would be encrypted but the attacker had the keys.

If you were already logged-in, your browser will send the login information in the cookie. Otherwise, your username and password would be sent if you typed them in on a login page.

Once the attacker got the login information, it used them on the legitimate website to transfer and steal Ethereum.

Normal case

After a BGP route leak

As previously mentioned, AS10279 announced this route. But only some regions got affected. Hurricane Electric has a strong presence Australia, mostly due to Internet costs. Chicago was affected because AS10279 has a physical presence resulting in direct peering.

The following graph displays the number of packets received in the affected regions and unaffected regions (Y axis normalized). The drop is due to the authoritative server not responding to our requests anymore (it only responded for the one website and all other Amazon domains were ignored).

The others transits used by eNet (CenturyLink, Cogent and NTT) did not seem to have accepted this route: a reason could be that they have filters and/or Amazon as a customer.eNet provides IP services, so one of their clients could have been announcing it.

As there are many actors involved, it is hard to determine fault. Actors involved:

• The ISP that announced a subnet it did not own.

• The transit providers that did not check the announcement before relaying it.

• The ISPs that accepted the route.

• The lack of protection on the DNS resolvers and authority.

• The phishing website hosted on providers in Russia.

• The website that did not enforce legitimate TLS certificates.

• The user that clicked continue even though the TLS certificate was invalid.

Just like a blockchain, a network change is usually visible and archived. RIPE maintains a database for this use.

This is a difficult question to answer. There are proposals for securing BGP:Some terms can be added to the RIR databases, so a list of allowed sources can be generated:

$ whois -h whois.radb.net ' -M 205.251.192.0/21' | egrep '^route:|^origin:|source:' | paste - - - | sort
route:      205.251.192.0/23	origin:     AS16509	source:     RADB
route:      205.251.192.0/23	origin:     AS16509	source:     REACH

Setting up RPKI/ROA records with the RIR as a source of truth regarding to the path of a route, although not everyone create those records or validate them. IP and BGP were created a few decades ago, with different requirements in mind regarding integrity and authenticity (less routes).

A few things can be done on the upper levels of the network stack.

On DNS, you can use DNSSEC to sign your records. The IPs returned by the fake DNS would not have been signed as they do not have the private keys.If you use Cloudflare as a DNS, you can enable DNSSEC within a few clicks in the panel.

On HTTPS, your browser will check the TLS certificate provided by the website. If HSTS is enabled, the browser will require a valid certificate all the time. The only way to generate a legitimate TLS certificate for a domain would be to poison the cache of a non-DNSSEC DNS resolver of the Certificate Authority.

DANE provides a way of pinning certificates to a domain name using DNS.

DNS over HTTPS would also allow to validate you are talking to the correct resolver in case the leak would be on the DNS resolvers instead of the DNS authority.

There is no perfect and unique solution. The more of these protections are in place, the harder it will be for a malicious actor to perform this kind of attack.

On Tuesday, September 5th, the government of Togo decided to restrict Internet access in the country following political protests. The government blocked social networks and rate-limited traffic, which had an impact on Cloudflare.

This adds Togo to the list of countries like Syria (twice), Iraq, Turkey, Libya, Tunisia, etc that have restricted or revoked Internet access.

The second event happened on Wednesday, September 6th, when a category 5 hurricane ravaged the Caribbean Islands.

The affected countries at the moment are:

• Anguilla

• Antigua and Barbuda

• British Virgin Islands

• Puerto Rico

• Saint Barthelemy

• Saint Kitts and Nevis

• Saint Martin

• Sint Maarten

• U.S. Virgin Islands

Most of the network cables are buried underground or laying at the bottom of the oceans but the hardware which relies on electricity is the first one to go down.

Cell towers sometime have their own power source thus allowing local phone calls but without a backbone no outside access is possible.

The Réseaux IP Européens (RIPE) is an Internet registry that collects routes announced over the Internet and can display the reachability of a country.A few networks went black in Saint Martin island. Orange Caraïbes announced in a press release that they repaired their network at 14:00 local time (18:00 UTC). On the graph we can see that it recovered at 19:00 UTC.

While the routers were going back online, the end-user traffic was severely reduced on the edge of Cloudflare.

Saint Kitts did not go offline, but traffic was reduced at 08:00 am local time on Wednesday.

Both Saint Martin and Sint Maarten share the same island, which was one of the most affected by the hurricane. According to the New York Times, the island was badly hit at around 10:00 am local time on September 6th.

Antigua and Barbuda started losing connectivity when the storm approached at 10:00 pm local time on Tuesday 5th September.

The U.S. Virgin Islands and Puerto Rico were also strongly affected.

This final image shows the different times when the hurricane reaches every island:

While the Internet was designed to sustain a nuclear attack and can re-route within seconds, islands are often isolated with fewer routing options for provider than on the mainland (3 or 4 instead of 10 or 15).During natural events like this one, the Internet is crucial as a resource for communication, since people use it to tell their family abroad they are safe and to find shelter, food and clean water.

If you live in these islands and have Internet access, Google lists the shelters and has public alerts.