A few months ago, we made a first then a second announcement about Cloudflare’s involvement in Resource Public Key Infrastructure (RPKI), and our desire to make BGP Internet routing more secure. Our mission is to build a safer Internet. We want to make it easier for network operators to deploy RPKI.
Today’s article is going to cover our experience and the tools we are using. As a brief reminder, RPKI is a framework that allows networks to deploy route filtering using cryptography-validated information. Picture TLS certificates for IP addresses and Autonomous System Numbers (ASNs)
What it means for you:
We validate our IP routes. This means, as a 1.1.1.1 DNS resolver user, you are less likely to be victim of cache poisoning. We signed our IP routes. This means a user browsing the websites on Cloudflare’s network are unlikely to experience route hijacks.
All our Points of Presence which have a router compatible with The Resource Public Key Infrastructure (RPKI) to Router Protocol (RTR protocol) are connected to our custom software called GoRTR and are now filtering invalid routes. The deployment amounts to around 70% of our network.
We received many questions regarding the amount of invalid traffic. Our estimates go from 100Mbps to 2Gbps. The spread remains high due to the difficulty of accounting the traffic that would go towards an invalid route. At our scale, it is a drop in the ocean of packets.
It is worth mentioning that one provider experienced a substantial traffic shift due to many regional IPs announced as smaller subnets and not included in the Route Origin Attestation (ROA), a key resource of the RPKI environment. We noticed many important networks announcing invalids on Internet Exchange Points. We emailed Network Operating Centers and were happy to see the records got corrected in a matter of days.
Let’s talk about tooling!
There are two components: GoRTR and OctoRPKI. The big picture of our setup.
OctoRPKI and GoRTR ecosystem diagram
As our network keeps growing, we needed a scalable solution to send the list of validated ROAs to our edge. Ideally using our CDN for caching the resources. We created GoRTR, a small Go application which will fetch a JSON file using a format used by RPKI Validators. We made sure that it could be used with various servers and implemented cryptographic checks to ensure an untampered distribution. By default, it will fetch the list of prefixes available at rpki.cloudflare.com/rpki.json.
You do not need to run your own validation software, just plug your RPKI-enabled router to GoRTR and start filtering. GoRTR will also expose a web server and Prometheus-type metrics. Nonetheless, if you want to run your own validation software, keep reading because OctoRPKI might be the solution for you. You can start GoRTR with Docker and plug your router to the port 8282 to receive a RTR feed:
$ docker run -ti -p 8282:8282 cloudflare/gortrIf you want to control the validation of routes, we got you covered!
After a few months work, we released OctoRPKI. Behind an octopus logo, there is a RPKI Relying Party written in Go.
Why did we build a validator?
The current ecosystem did not bring us joy, we needed RPKI Repository Delta Protocol (RRDP), a fully-fledged monitored validator subsystem and the ability to push only the final computed data to our storage cluster. We decided to build it in Go for multiple reasons. In exchange of a slightly increased overhead, we gained flexibility. There is a great amount of cryptographic resources and libraries for the Go language. And we are hoping this would benefit an already important community.
The repository contains OctoRPKI plus a library to decode certificates, ROAs, manifests, etc. The code can start a synchronization with rsync and/or RRDP.
Similarly to GoRTR, it embeds an API to get the latest results of the synchronization and a Prometheus metrics endpoint to deploy alerting. Internally, we monitor the state of the validation using Prometheus. It is also providing data for a GraphQL API that will be released soon.
You can start it with Docker.
The docker image provides a way to instantly run OctoRPKI and comes complete with four of the five Trust Anchor Locator (TAL) files needed to operate (AFRINIC, APNIC, LACNIC, and RIPE). The fifth is the ARIN TAL, so don’t forget to add the TAL from ARIN. You can start the process of downloading it at the following address: www.arin.net/resources/rpki/tal.html.
$ docker run -ti \
-p 8080:8080 \
-v $PWD/cache:/cache \
-v path_to_arin_tal:/tals/arin.tal
cloudflare/octorpkiThe results will be available on http://localhost:8080/output.json. To plug GoRTR to this file, use the -cache URL argument (you will need to load the public key which signs some fields in JSON).
$ gortr -cache http://localhost:8080/output.json -verify.key path-to-keyIn conclusion, we hope these tools will help you deploy RPKI easily. Routing security is an important subject that should not be avoided. Looking forward to more networks joining this initiative and making the Internet more secure.We learned a lot about RPKI, from signing the routes on the five registries to validating cryptographically more than 60000 resources in a few seconds. We will keep working on making OctoRPKI and GoRTR always more stable and reliable. We look forward to your feedback!