Protecting Cloudflare Customers from BGP Insecurity with Route Leak Detection

This post is also available in 日本語, Bahasa Indonesia, ไทย.

Border Gateway Protocol (BGP) route leaks and hijacks can ruin your day — BGP is insecure by design, and incorrect routing information spreading across the Internet can be incredibly disruptive and dangerous to the normal functioning of customer networks, and the Internet at large. Today, we're excited to announce Route Leak Detection, a new network alerting feature that tells customers when a prefix they own that is onboarded to Cloudflare is being leaked, i.e., advertised by an unauthorized party. Route Leak Detection helps protect your routes on the Internet: it tells you when your traffic is going places it’s not supposed to go, which is an indicator of a possible attack, and reduces time to mitigate leaks by arming you with timely information.

In this blog, we will explain what route leaks are, how Cloudflare Route Leak Detection works, and what we are doing to help protect the Internet from route leaks.

A route leak occurs when a network on the Internet tells the rest of the world to route traffic through their network, when the traffic isn’t supposed to go there normally. A great example of this and the impact it can cause was an incident in June 2019, where a small ISP in Pennsylvania started advertising routes for part of the Internet including Cloudflare, Amazon, and Linode. A significant portion of traffic destined for those networks was incorrectly routed to the network, leaking Cloudflare, Amazon, and Linode’s prefixes, and causing congestion and unreachable network errors for end users. Route leaks tend to happen because of a misconfigured peering session or customer router, a software bug in a customer or third party router, a man-in-the-middle attack, or a malicious customer or third party.

Some route leaks are innocuous. But some route leaks can be malicious, and can have very real security impact. An attacker can advertise specific routes for the express purpose of directing users to their network to do things like steal cryptocurrencies and other important data, or attempt to issue SSL/TLS certificates that can be used to impersonate domains. By advertising more specific routes, an attacker can trick you into accessing a site that you don’t intend to, and if it looks exactly like the site you expect, you may unwittingly enter in personal data and be at risk for an attack. Here’s a diagram representing traffic without a route leak:

And here’s traffic after a route leak:

So in addition to making users unhappy because a lot of Internet traffic is going through paths that can’t handle it, route leaks can have very real data leak implications.

Cloudflare’s Route Leak Detection allows you to get notified quickly when your routes are leaking so that you know when a potential attack is happening.

In order to configure Route Leak Detection, you must be a Cloudflare customer who has “brought your own IP” (BYOIP) addresses—this includes Magic Transit (L3), Spectrum (L4), and WAF (L7) customers. Only prefixes advertised by Cloudflare qualify for Route Leak Detection.

Configuring Route Leak Detection can be done by setting up a message in the Notifications tab in your account.

Cloudflare will then begin monitoring all of your onboarded prefixes for leaks and hijacks and will send you alerts when they occur via email or specialized on call tools like PagerDuty.

Cloudflare’s alert notification system supports webhooks, email, and PagerDuty, so your teams are kept up to date across their desired medium with changes in network routes and so that they can respond and take corrective action when necessary.

A malicious party attempting to use routes to gain access to customer data starts advertising a subnet of onboarded prefixes for one of our Magic Transit customers. This attack, if not found and remediated quickly, could have serious impact on the customer. When the attacker begins to advertise the prefix without the customer’s knowledge, BGP updates and route changes start occurring rapidly in the global routing table—typically within 60 seconds.

Let’s walk through how a customer might deploy Route Leak Detection. Customer Acme Corp. owns the IP prefix 203.0.113.0/24. Acme has onboarded 203.0.113.0/24 to Cloudflare, and Cloudflare tells the rest of the Internet that this prefix is reachable through Cloudflare’s network.

Once Acme has enabled Route Leak Detection, Cloudflare continuously monitors routing information on the Internet for 203.0.113.0/24. Our goal is to detect leaks within five minutes of the erroneous routing information propagating on the Internet.

Let’s go back to the attack scenario. A malicious party attempting to attack Acme’s network hijacks the advertisement for 203.0.113.0/24, diverting legitimate users from the intended network path to Acme (though Cloudflare’s network) and instead to a facsimile of Acme’s network intended to capture information from unwitting users.

Because Acme has enabled Route Leak Detection, an alert is sent to Acme’s administrators.

The alert includes all of the ASNs that are seeing the prefix being advertised by the potentially malicious party. Acme is able to warn their users that they may be at risk of a data exfiltration attack, and they should be on the lookout for suspicious behavior.

Acme is also able to quickly contact the service providers listed in the alert to stop honoring the smaller routes. Currently, the process of mitigating a route leak is a highly manual process, requiring contacting service providers directly using contact information published in public databases. In the future, we plan to build features to automate this outreach and mitigation process to further drive downtime to mitigation for route leak events that may impact our customers.

Cloudflare uses several sources of routing data to create a synthesis of how the Internet sees routes to our BYOIP customers. Cloudflare then watches these views to track any sudden changes that occur on the Internet. If we can correlate those changes to actions we have taken, then we know the change is benign, and it’s business as usual. However, if we haven’t made any changes, we quickly take action to tell you that your routes and your users may be at risk.

Cloudflare’s main source of data is from externally maintained repositories such as RIPE’s RIS feed, RouteViews, and Caida’s public BMP feed. It is important to use multiple external views of the Internet routing table to be as accurate as possible when making inferences about the state of the Internet. Cloudflare makes API calls to those sources to ingest data and analyze changes in BGP routes. These feeds allow us to ingest routing data for the whole Internet. Cloudflare filters all of that down to your prefixes that you have previously onboarded to Cloudflare.

Once this data is ingested and filtered, Cloudflare begins cross-referencing updates to the global routing table with metrics that indicate possible hijacks, such as the number of ASNs that directly see your routes, the number of BGP updates that occur over short periods of time, and how many subnets are being advertised. If the number of ASNs that directly see your routes or the number of updates change drastically, it could mean that your prefixes are being leaked. If a subnet of the prefixes you are advertising are seeing drastic amounts of change in the global routing table, it’s likely that your prefixes are being leaked somewhere.

Cloudflare already has this configured on our own prefixes today. Here’s an example of what we see when our system determines that something is wrong:

Cloudflare owns the prefix range 2606:4700:50::/44, as it is a subnet of one of the ranges listed on our site here. For a period of an hour, we noticed that someone tried to advertise a subnet of that range to 38 other networks. Fortunately, because we have deployed RPKI, we know that most networks will reject rather than honor these route advertisements from attackers.

The best way to prevent route leaks is to deploy RPKI in your network, and urge your Internet providers to do so as well. RPKI allows you and your providers to sign routes that you advertise to the Internet, so that no one else can steal them. If someone is advertising your RPKI routes, any providers that support RPKI will not forward those routes to other customers, ensuring that the attempted leak is contained as close to the attacker as possible.

Cloudflare’s continued advocacy for RPKI has yielded fruits in the past three months alone. Providers such as Amazon, Google, Telstra, Cogent, and even Netflix have started supporting RPKI and are filtering and dropping invalid prefixes. In fact, over 50% of the top Internet providers now support RPKI in some fashion:

Cloudflare’s Route Leak Detection combined with more providers implementing RPKI are helping to ensure that data loss and downtime from route leaks become a thing of the past. If you’re a Cloudflare Magic Transit or BYOIP customer, try configuring a route leak alert in your dash today. If you’re not a Magic Transit or BYOIP customer, reach out to our sales team to get started in the process to keep your network safe — even the routes.