RPKI and BGP: our path to securing Internet Routing

This article will talk about our approach to network security using technologies like RPKI to sign Internet routes and protect our users and customers from route hijacks and misconfigurations. We are proud to announce we have started deploying active filtering by using RPKI for routing decisions and signing our routes.

Back in April, articles including our blog post on BGP and route-leaks were reported in the news, highlighting how IP addresses can be redirected maliciously or by mistake. While enormous, the underlying routing infrastructure, the bedrock of the Internet, has remained mostly unsecured.

At Cloudflare, we decided to secure our part of the Internet by protecting our customers and everyone using our services including our recursive resolver 1.1.1.1.

A prefix is a range of IP addresses, for instance, 10.0.0.0/24, whose first address is 10.0.0.0 and the last one is 10.0.0.255. A computer or a server usually have one. A router creates a list of reachable prefixes called a routing table and uses this routing table to transport packets from a source to a destination.  

On the Internet, network devices exchange routes via a protocol called BGP (Border Gateway Protocol). BGP enables a map of the interconnections on the Internet so that packets can be sent across different networks to reach their final destination. BGP binds the separate networks together into the Internet.

This dynamic protocol is also what makes Internet so resilient by providing multiple paths in case a router on the way fails. A BGP announcement is usually composed of a prefix which can be reached at a destination and was originated by an Autonomous System Number (ASN).

IP addresses and Autonomous Systems Numbers are allocated by five Regional Internet Registries (RIR): Afrinic for Africa, APNIC for Asia-Pacific, ARIN for North America, LACNIC for Central and South America and RIPE for Europe, Middle-East and Russia. Each one operates independently.

With more than 700,000 IPv4 routes and 40,000 IPv6 routes announced by all Internet actors, it is difficult to know who owns which resource.

There is no simple relationship between the entity that has a prefix assigned, the one that announces it with an ASN and the ones that receive or send packets with these IP addresses. An entity owning 10.0.0.0/8 may be delegating a subset 10.1.0.0/24 of that space to another operator while being announced through the AS of another entity.

Thereby, a route leak or a route hijack is defined as the illegitimate advertisement of an IP space. The term route hijack implies a malicious purpose while a route leak usually happens because of a misconfiguration.

A change in route will cause the traffic to be redirected via other networks. Unencrypted traffic can be read and modified. HTTP webpages and DNS without DNSSEC are sensitive to these exploits.

You can learn more about BGP Hijacking in our Learning Center.

When an illegitimate announcement is detected by a peer, they usually notify the origin and reconfigure their network to reject the invalid route.Unfortunately, the time to detect and act upon may take from a few minutes up to a few days, more than enough to steal cryptocurrencies, poison a DNS cache or make a website unavailable.

A few systems exist to document and prevent illegitimate BGP announcements.

The Internet Routing Registries (IRR) are semi-public databases used by network operators to register their assigned Internet resources. Some database maintainers do not check whether the entry was actually made by the owner, nor check if the prefix has been transferred to somebody else. This makes them prone to error and not completely reliable.

Resource Public Key Infrastructure (RPKI) is similar to the IRR “route” objects, but adding the authentication with cryptography.

Here’s how it works: each RIR has a root certificate. They can generate a signed certificate for a Local Internet Registry (LIR, a.k.a. a network operator) with all the resources they are assigned (IPs and ASNs). The LIR then signs the prefix containing the origin AS that they intend to use: a ROA (Route Object Authorization) is created. ROAs are just simple X.509 certificates.

If you are used to SSL/TLS certificates used by browsers to authenticate the holder of a website, then ROAs are the equivalent in the Internet routing world.

Each network operator owning and managing Internet resources (IP addresses, Autonomous System Numbers) has access to their Regional Internet Registry portal. Signing their prefixes through the portal or the API of their RIR is the easiest way to start with RPKI.

Because of our global presence, Cloudflare has resources in each of the 5 RIR regions. With more than 800 prefix announcements over different ASNs, the first step was to ensure the prefixes we were going to sign were correctly announced.

We started by signing our less used prefixes, checked if the traffic levels remained the same and then signed more prefixes. Today about 25% of Cloudflare prefixes are signed. This includes our critical DNS servers and our public 1.1.1.1 resolver.

Signing the prefixes is one thing. But ensuring that the prefixes we receive from our peers match their certificates is another.

The first part is validating the certificate chain. It is done by synchronizing the RIR databases of ROAs through rsync (although there are some new proposals regarding distribution over HTTPS), then check the signature of every ROA against the RIR’s certificate public key. Once the valid records are known, this information is sent to the routers.

Major vendors support a protocol called RPKI to Router Protocol (abbreviated as RTR). This is a simple protocol for passing a list of valid prefixes with their origin ASN and expected mask length. However, while the RFC defines 4 different secure transport methods, vendors have only implemented the insecure one. Routes sent in clear text over TCP can be tampered with.

With more than 150 routers over the globe, it would be unsafe to rely on these cleartext TCP sessions over the insecure and lossy Internet to our validator. We needed local distribution on a link we know secure and reliable.

One option we considered was to install an RPKI RTR server and a validator in each of our 150+ datacenters, but doing so would cause a significant increase in operational cost and reduce debugging capabilities.

Introducing GoRTR

We needed an easier way of passing an RPKI cache securely. After some system design sessions, we settled on distributing the list of valid routes from a central validator securely, distribute it via our own Content Delivery Network and use a lightweight local RTR server. This server fetches the cache file over HTTPS and passes the routes over RTR.

Rolling out this system on all our PoPs using automation was straightforward and we are progressively moving towards enforcing strict validation of RPKI signed routes everywhere.

To encourage adoption of Route Origin Validation on the Internet, we also want to provide this service to everyone, for free. You can already download our RTR server with the cache behind Cloudflare. Just configure your Juniper or Cisco router. And if you do not want to use our file of prefixes, it is compatible with the RIPE RPKI Validator Export format.

We are also working on providing a public RTR server using our own Spectrum service so that you will not have to install anything, just make sure you peer with us! Cloudflare is present on many Internet Exchange Points so we are one hop away from most routers.

A few months ago, Nick Sullivan introduced our new Nimbus Certificate Transparency Log.

In order to track the emitted certificates in the RPKI, our Crypto team created a new Certificate Transparency Log called Cirrus which includes the five RIRs root certificates as trust anchors. Certificate transparency is a great tool for detecting bad behavior in the RPKI because it keeps a permanent record of all valid certificates that are submitted to it in an append-only database that can’t be modified without detection. It also enables users to download the entire set of certificates via an HTTP API.

We use services like BGPmon and other public observation services extensively to ensure quick action if some of our prefixes are leaked. We also have internal BGP and BMP collectors, aggregating more than 60 millions routes and processing live updates.

Our filters make use of this live feed to ensure we are alerted when a suspicious route appears.

The latest statistics suggest that around 8.7% of the IPv4 Internet routes are signed with RPKI, but only 0.5% of all the networks apply strict RPKI validation.Even with RPKI validation enforced, a BGP actor could still impersonate your origin AS and advertise your BGP route through a malicious router configuration.

However that can be partially solved by denser interconnections, that Cloudflare already has through an extensive network of private and public interconnections.To be fully effective, RPKI must be deployed by multiple major network operators.

As said by Job Snijders from NTT Communications, who’s been at the forefront of the efforts of securing Internet routing:

If the Internet's major content providers use RPKI and validate routes, the impact of BGP attacks is greatly reduced because protected paths are formed back and forth. It'll only take a small specific group of densely connected organizations to positively impact the Internet experience for billions of end users.

RPKI is not a bullet-proof solution to securing all routing on the Internet, however it represents the first milestone in moving from trust based to authentication based routing. Our intention is to demonstrate that it can be done simply and cost efficiently. We are inviting operators of critical Internet infrastructure to follow us in a large scale deployment.

Subscribe to the blog for daily updates on our announcements.