Today’s Internet requires stronger protection within its core routing system and as we have already said: it's high time to stop BGP route leaks and hijacks by deploying operationally-excellent RPKI!
Luckily, over the last year plus a lot of good work has happened in this arena. If you’ve been following the growth of RPKI’s validation data, then you’ll know that more and more networks are signing their routes and creating ROA’s or Route Origin Authorizations. These are cryptographically-signed assertions of the validity of an announced IP block and contribute to the further securing of the global routing table that makes for a safer Internet.
The protocol that we have not written much about is RTR. The Resource Public Key Infrastructure (RPKI) to Router Protocol - or RTR Protocol for short. Today we’re fixing that.
RPKI rewind
We have written a few times about RPKI (here and here). We have written about how Cloudflare both signs its announced routes and filters its routing inbound from other networks (both transits and peers) using RPKI data. We also added our efforts in the open-source software space with the release of the Cloudflare RPKI Toolkit.
The primary part of the RPKI (Resource Public Key Infrastructure) system is a cryptographically-signed database which is read and processed by a RPKI validator. The validator works with the published ROAs to build a list of validated routes. A ROA consists of an IP address block plus an ASN (Autonomous System Number) that together define who can announce which IP block.
After that step, it is then the job of that validator (or some associated software module) to communicate its list of valid routes to an Internet router. That’s where the RTR protocol (the RPKI to Router Protocol) comes in. Its job is to communicate between the validator and device in charge of allowing or rejecting routes in its table.
RTR
The IETF defines the RTR protocol in RFC 8210. This blog post focuses on version 1 and ignores previous versions.
In order to verifiably validate the origin Autonomous Systems and Autonomous System Paths of BGP announcements, routers need a simple but reliable mechanism to receive Resource Public Key Infrastructure (RFC 6480) prefix origin data and router keys from a trusted cache. This document describes a protocol to deliver them.
This document describes version 1 of the RPKI-Router protocol.
The Internet’s routers are, to put it bluntly, not the best place to run a routing table’s cryptographic processing. The RTR protocol allows the heavy lifting to be done outside of the valuable processing modules that routers have. RTR is a very lightweight protocol with a low memory footprint. The router simply decides yay-or-nay when a route is received (called “announce” in BGP speak) and hence the router never needs to touch the complex cryptographic validation algorithms. In many cases, it also provides some isolation between the outside world, where certificates need to be fetched from across the globe and then stored, checked, processed, and databased locally. In many cases the control plane (where RTR communication happens) exists on private or protected networks. Separation is a good thing.
Cloudflare’s open-source software for RPKI validation also includes GoRTR, an implementation of the RTR protocol. As mentioned, in Cloudflare’s operational model, we separate the validation (done with OctoRPKI) from the RTR process.
RTR protocol implementations are also provided in other RPKI validation software packages. In fact, RPKI is unable to filter routes without the final step of running RTR (or something similar - should it exist). Here’s a current list of RPKI software packages that either validate or validate and run RTR.
- Cloudflare RPKI Validator Tools and Libraries (OctoRPKI & GoRTR).
- Dragon Research Labs RPKI Toolkit.
- NIC Mexico and LACNIC FORT project including the FORT validator.
- NLnet Labs Routinator 3000.
- RIPE NCC RPKI Validator version 2 (deprecated).
- RIPE NCC RPKI Validator version 3.
- rpki-client by OpenBSD
Each of these open source software packages has its own specific database model and operational methods. Because GoRTR reads a somewhat common JSON file format, you can mix and match between different validators and GoRTR’s code.
The RTR protocol
The protocol's core is all about synchronizing a database between a validator and a router. This is done using serial-numbers and session-ids.
It’s kicked off with a router setting up a TCP connection towards a backend RTR server followed by a series of serial-number exchanges and data records exchanges such that a cache on the validator (or RTR server) can be synced fully with a cache on the router. As mentioned, the lightweight protocol is void of all the cryptographic data that RPKI is built upon and simply deals with the validated routing list, which consists of CIDRs, ASNs and maybe a MaxLength parameter.
Here’s a simple Cisco configuration for enabling RTR on a router:
router bgp 65001
rpki server 192.168.1.100
transport tcp port 8282
!
!
The configuration can take additional parameters in order to enable SSH or similar transport options. Other platforms (such as Juniper, Arista, Bird 2.0, etc) have their own specific configuration language.
The RTR protocol supports IPv4 and IPv6 routing information (as you would expect).
Being specified as a lightweight protocol, RTR allows the data to be transferred quickly. With a session-id created by the RTR cache server plus serial-numbers exchanged between cache servers and routers, there’s the solid ability for route authentication data on the router to be kept fresh with a minimum amount of actual data being transferred. Remember, as we said above, the router has much better things to do with its control plane processor like running the BGP convergence algorithm, or SRv6, or ISIS, or any of the protocols needed to manage routing tables.
Is RTR a weak link in the RPKI story?
All aspects of RPKI data processing are built around solid cryptographic principles. The five RIRs each hold a root key called a Trust Anchor (TA). Each publishes data fully signed up/down so that every piece of information can be proven to be correct and without tampering. A validators job is to do that processing and spit out (or store) a list of valid ROAs (Route Origin Authorizations) that are assertions traceable back to a known source. If you want to study this protocol, you can start with RFC6480 and work forward through all the other relevant RFCs (Hint: It’s at least thirty more RFCs from RFC6483 thru RFC8210 and counting).
However, RTR does not carry that trust through to the Internet router. All that complexity (and hence assertions) are stripped away before a router sees anything. It is 100% up to the network operator to build a reliable and secure path between validator or RTR cache and router so that this lightweight transfer is still trusted.
RTR helps somewhat in this space. It provides more than one way to communicate between cache server and router. The RFC defines various methods to communicate.
- A plain TCP connection (which is clearly insecure). In this case the RFC states: “the cache and routers MUST be on the same trusted and controlled network.”.
- A TCP connection with TCP-AO transport.
- A Secure Shell version 2 (SSHv2) transport.
- A TCP connection with TCP MD5 transport (which is already obsoleted by TCP-AO).
- A TCP connection over IPsec transport.
- Transport Layer Security (TLS) transport.
This plethora of options is all well and good; however, there’s no useful implementation of TCP-AO out in the production world and hence (ironically) a lot of early implementations are living with plain-text communications. SSH and TLS are much better options; however, this comes with classic operational problems to solve. For example, in SSH’s case, the RFC states:
It is assumed that the router and cache have exchanged keys out of band by some reasonably secured means.
For a TLS connection, there’s also some worthwhile security setup mentioned in the RFC. It starts off as follows:
Client routers using TLS transport MUST present client-side certificates to authenticate themselves to the cache in order to allow the cache to manage the load by rejecting connections from unauthorized routers.
Then the RFC continues with enough information to secure the connection fully. If implemented correctly, then security is correctly provided between RTR cache and router such that no MITM attack can take place.
Assuming that these operational issues are handled fully then the RTR protocol is a perfect protocol for operationally implementing RPKI’s final linkage into the routers.
Testing the RTR protocol and open-source rpki-rtr-client
A modern router software stack can be configured to run RTR against a cache. If you have a test lab (as most modern networks do); then you have all you need to see RPKI route filtering (and the dropping of invalid routes).
However, if you are without a router and want to see RTR in action, Cloudflare has just placed rpki-rtr-client on GitHub. This software, written in Python, performs the router portion of the RTR protocol and comes with enough debug output that it can also be used to help write new RTR caches, or test existing code bases. The code was written directly from the RFC and then tested against a public RTR cache that Cloudflare operates.
$ pip3 install netaddr
...
$ git clone https://github.com/cloudflare/rpki-rtr-client.git
...
$ cd rpki-rtr-client
$
Operating the client is easy (and doubly-so if you use the Cloudflare provided cache).
$ ./rtr_client.py -h rtr.rpki.cloudflare.com -p 8282
...
^C
$
As there is no router (and hence no dropping of invalids) this code simply creates data files for later review. See the README file for more information.
$ ls -lt data/2020-02
total 21592
-rw-r--r-- 1 martin martin 5520676 Feb 16 18:22 2020-02-17-022209.routes.00000365.json
-rw-r--r-- 1 martin martin 5520676 Feb 16 18:42 2020-02-17-024242.routes.00000838.json
-rw-r--r-- 1 martin martin 412 Feb 16 19:56 2020-02-17-035645.routes.00000841.json
-rw-r--r-- 1 martin martin 272 Feb 16 20:16 2020-02-17-041647.routes.00000842.json
-rw-r--r-- 1 martin martin 643 Feb 16 20:36 2020-02-17-043649.routes.00000843.json
$
As the RTR protocol communicates and increments its serial-number, the rpki-rtr-client software writes the routing information is a fresh file for later review.
$ for f in data/2020-02/*.json ; do echo "$f `jq -r '.routes.announce[]|.ip' < $f | wc -l` `jq -r '.routes.withdraw[]|.ip' < $f | wc -l`" ; done
data/2020-02/2020-02-17-022209.routes.00000365.json 128483 0
data/2020-02/2020-02-17-024242.routes.00000838.json 128483 0
data/2020-02/2020-02-17-035645.routes.00000841.json 3 6
data/2020-02/2020-02-17-041647.routes.00000842.json 5 0
data/2020-02/2020-02-17-043649.routes.00000843.json 9 5
$
Valid ROAs are listed as follows:
$ jq -r '.routes.announce[]|.ip,.asn,.maxlen' data/2020-02/*0838.json | paste - - - | sort -V | head
1.0.0.0/24 13335 null
1.1.1.0/24 13335 null
1.9.0.0/16 4788 24
1.9.12.0/24 65037 null
1.9.21.0/24 24514 null
1.9.23.0/24 65120 null
1.9.31.0/24 65077 null
1.9.65.0/24 24514 null
1.34.0.0/15 3462 24
1.36.0.0/16 4760 null
$
The code can also dump the raw binary protocol and then replay that data to debug the protocol.
As the code is on GitHub, any protocol developer can feel free to expand on the code.
Future of RTR protocol
The present RFC defines version 1 of the protocol and it is expected that this lightweight protocol will progress to also include additional functions, but stay lightweight. RPKI is a Route Origin Validation protocol (i.e. mapping an IP route or CIDR to an ASN). It does not provide support for validating the AS-PATH. Neither does it provide any support for IRR databases (which are non-cryptographically-signed routing definitions). Presently IRR data is the primary method used for filtering routing on the global Internet. Today that is done by building massive filter lists within a router's configuration file and not via a lightweight protocol like RTR.
At the present time there’s an IETF proposal for RTR version 2. This is draft, work alongside the ASPA (Autonomous System Provider Authorization) draft and draft work. These draft documents from Alexander Azimov et al. define ASPA extending the RPKI data structures to handle BGP path information. The version 2 of RTR protocol should provide the required messaging in order to move ASPA data into the router.
Additionally, RPKI is going to potentially further expand, at some point, from today's singular data type (the ROA object). Just like with the ASPA draft, RTR will need to advance in lock-step. Hopefully the open-source code we have published will help this effort.
Some final thoughts on RTR and RPKI
If RPKI is to become ubiquitous, then RTR support in all BGP speaking Internet routers is going to be required. Vendors need to complete their RTR software delivery and additionally support some of the more secure transport definitions from the RFC. Additionally, should the protocol advance, then timely support for the new version will be needed.
Cloudflare continues to be committed to a secure Internet; so should you also have the same thoughts and you like what you’ve read here or elsewhere on our blog; then please take a look at our jobs page. We have software and network engineering open roles in many of our offices around the world.