
<rss version="2.0" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:media="http://search.yahoo.com/mrss/">
    <channel>
        <title><![CDATA[ The Cloudflare Blog ]]></title>
        <description><![CDATA[ Get the latest news on how products at Cloudflare are built, technologies used, and join the teams helping to build a better Internet. ]]></description>
        <link>https://blog.cloudflare.com</link>
        <atom:link href="https://blog.cloudflare.com/" rel="self" type="application/rss+xml"/>
        <language>en-us</language>
        <image>
            <url>https://blog.cloudflare.com/favicon.png</url>
            <title>The Cloudflare Blog</title>
            <link>https://blog.cloudflare.com</link>
        </image>
        <lastBuildDate>Sun, 05 Apr 2026 17:04:05 GMT</lastBuildDate>
        <item>
            <title><![CDATA[Bringing more transparency to post-quantum usage, encrypted messaging, and routing security]]></title>
            <link>https://blog.cloudflare.com/radar-origin-pq-key-transparency-aspa/</link>
            <pubDate>Fri, 27 Feb 2026 06:00:00 GMT</pubDate>
            <description><![CDATA[ Cloudflare Radar has added new tools for monitoring PQ adoption, KT logs for messaging, and ASPA routing records to track the Internet's migration toward more secure encryption and routing standards.  ]]></description>
            <content:encoded><![CDATA[ <p></p><p>Cloudflare Radar already offers a wide array of <a href="https://radar.cloudflare.com/security/"><u>security insights</u></a> — from application and network layer attacks, to malicious email messages, to digital certificates and Internet routing.</p><p>And today we’re introducing even more. We are launching several new security-related data sets and tools on Radar: </p><ul><li><p>We are extending our post-quantum (PQ) monitoring beyond the client side to now include origin-facing connections. We have also released a new tool to help you check any website's post-quantum encryption compatibility. </p></li><li><p>A new Key Transparency section on Radar provides a public dashboard showing the real-time verification status of Key Transparency Logs for end-to-end encrypted messaging services like WhatsApp, showing when each log was last signed and verified by Cloudflare's Auditor. The page serves as a transparent interface where anyone can monitor the integrity of public key distribution and access the API to independently validate our Auditor’s proofs. </p></li><li><p>Routing Security insights continue to expand with the addition of global, country, and network-level information about the deployment of ASPA, an emerging standard that can help detect and prevent BGP route leaks. </p></li></ul>
    <div>
      <h2>Measuring origin post-quantum support</h2>
      <a href="#measuring-origin-post-quantum-support">
        
      </a>
    </div>
    
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/2gs0x3zMZTxios168jT9xW/179d8959b5e0939835cf6facef797457/1.png" />
          </figure><p>Since <a href="https://x.com/CloudflareRadar/status/1788277817362329983"><u>April 2024</u></a>, we have tracked the aggregate growth of client support for post-quantum encryption on Cloudflare Radar, chronicling its global growth from <a href="https://radar.cloudflare.com/adoption-and-usage?dateStart=2024-01-01&amp;dateEnd=2024-01-31#post-quantum-encryption-adoption"><u>under 3% at the start of 2024</u></a>, to <a href="https://radar.cloudflare.com/adoption-and-usage?dateStart=2026-02-01&amp;dateEnd=2026-02-28#post-quantum-encryption-adoption"><u>over 60% in February 2026</u></a>. And in October 2025, <a href="https://blog.cloudflare.com/pq-2025/#what-you-can-do-today-to-stay-safe-against-quantum-attacks"><u>we added the ability</u></a> for users to <a href="https://radar.cloudflare.com/adoption-and-usage#browser-support"><u>check</u></a> whether their browser supports <a href="https://developers.cloudflare.com/ssl/post-quantum-cryptography/pqc-support/#x25519mlkem768"><code><u>X25519MLKEM768</u></code></a> — a hybrid key exchange algorithm combining classical <a href="https://www.rfc-editor.org/rfc/rfc8410"><code><u>X25519</u></code></a> with <a href="https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.203.pdf"><u>ML-KEM</u></a>, a lattice-based post-quantum scheme standardized by NIST. This provides security against both classical and quantum attacks. </p><p>However, post-quantum encryption support on user-to-Cloudflare connections is only part of the story.</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/67cvSmOaISIHjrKKRHKPzg/e0ccf032658904fd6beaa7de7340b561/2.png" />
          </figure><p>For content not in our CDN cache, or for uncacheable content, Cloudflare’s edge servers establish a separate connection with a customer’s origin servers to retrieve it. To accelerate the transition to quantum-resistant security for these origin-facing fetches, we <a href="https://blog.cloudflare.com/post-quantum-to-origins/"><u>previously introduced an API</u></a> allowing customers to opt in to preferring post-quantum connections. Today, we’re making post-quantum compatibility of origin servers visible on Radar.</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/6KvV2meYLEPbNIQyHP6yji/9477a134c8f5f6a7aaecd6257cd59981/3.png" />
          </figure><p>The new origin post-quantum support graph on Radar illustrates the share of customer origins supporting <code>X25519MLKEM768</code>. This data is derived from <a href="https://blog.cloudflare.com/automatically-secure/"><u>our automated TLS scanner,</u></a> which probes TLS 1.3-compatible origins and aggregates the results daily. It is important to note that our scanner tests for support rather than the origin server's specific preference. While an origin may support a post-quantum key exchange algorithm, its local TLS key exchange preference can ultimately dictate the encryption outcome.</p><p>While the headline graph focuses on post-quantum readiness, the scanner also evaluates support for classical key exchange algorithms. Within the Radar <a href="https://radar.cloudflare.com/explorer?dataSet=post_quantum.origin&amp;groupBy=key_agreement#result"><u>Data Explorer view</u></a>, you can also see the full distribution of these supported TLS key exchange methods.</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/5PBOoQSCcIAQrYezKp1pJU/d4218aba59deef6c21df53856a93040a/4.png" />
          </figure><p>As shown in the graphs above, approximately 10% of origins could benefit from a post-quantum-preferred key agreement today. This represents a significant jump from less than 1% at the start of 2025 — <a href="https://radar.cloudflare.com/explorer?dataSet=post_quantum.origin&amp;groupBy=key_agreement&amp;dt=2025-01-01_2025-12-31"><u>a 10x increase in just over a year</u></a>. We expect this number to grow steadily as the industry continues its migration. This upward trend likely accelerated in 2025 as many server-side TLS libraries, such as <a href="https://openssl-library.org/post/2025-04-08-openssl-35-final-release/"><u>OpenSSL 3.5.0+</u></a>,<a href="https://www.gnutls.org/"><u> GnuTLS 3.8.9+</u></a>, and <a href="https://go.dev/doc/go1.24#cryptotlspkgcryptotls"><u>Go 1.24+</u></a>, enabled hybrid post-quantum key exchange by default, allowing platforms and services to support post-quantum connections simply by upgrading their cryptographic library dependencies.</p><p>In addition to the Radar and Data Explorer graphs, the <a href="https://developers.cloudflare.com/api/resources/radar/subresources/post_quantum/subresources/origin/"><u>origin readiness data is available through the Radar API</u></a> as well.</p><p>As an additional part of our efforts to help the Internet transition to post-quantum cryptography, we are also launching <a href="https://radar.cloudflare.com/post-quantum#website-support"><u>a tool to test whether a specific hostname supports post-quantum encryption</u></a>. These tests can be run against any publicly accessible website, as long as they allow connections from Cloudflare’s <a href="https://www.cloudflare.com/ips/"><u>egress IP address ranges</u></a>. </p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/5dgwK3i7IeLLSUt5xnk4lf/276e25dda3389f6e0ad83a26acd08fec/5.png" />
          </figure><p><sub><i>A screenshot of the tool in Radar to test whether a hostname supports post-quantum encryption.</i></sub></p><p>The tool presents a simple form where users can enter a hostname (such as <a href="https://radar.cloudflare.com/post-quantum?host=cloudflare.com%3A443"><code><u>cloudflare.com</u></code></a> or <a href="https://radar.cloudflare.com/post-quantum?host=www.wikipedia.org%3A443"><code><u>www.wikipedia.org</u></code></a>) and optionally specify a custom port (the default is <a href="https://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.xhtml?search=443"><u>443, the standard HTTPS port</u></a>). After clicking "Test", the result displays a tag indicating PQ support status alongside the negotiated TLS key exchange algorithm. If the server prefers PQ secure connections, a green "PQ" tag appears with a message confirming the connection is "post-quantum secure." Otherwise, a red tag indicates the connection is "not post-quantum secure", showing the classical algorithm that was negotiated.</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/3rfEG4dMlwR4FJkaKXTRWF/8cab135242057ce57f3b0e4a92be4cec/6.png" />
          </figure>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/PXu3kjzwhVkb29kIFREOn/41785c06297e0667ff9e2b261ae9b819/7.png" />
          </figure><p>Under the hood, this tool uses <a href="https://developers.cloudflare.com/containers/"><u>Cloudflare Containers</u></a> — a new capability that allows running container workloads alongside Workers. Since the Workers runtime is not exposed to details of the underlying TLS handshake, Workers cannot initiate TLS scans. Therefore, we created a Go container that leverages the <a href="https://pkg.go.dev/crypto/tls"><code><u>crypto/tls</u></code></a> package's support for post-quantum compatibility checks. The container runs on-demand and performs the actual handshake to determine the negotiated TLS key exchange algorithm, returning results through the <a href="https://developers.cloudflare.com/api/resources/radar/subresources/post_quantum/subresources/tls/methods/support/"><u>Radar API</u></a>.</p><p>With the addition of these origin-facing insights, complementing the existing client-facing insights, we have moved all the post-quantum content to <a href="https://radar.cloudflare.com/post-quantum"><u>its own section on Radar</u></a>. </p>
    <div>
      <h2>Securing E2EE messaging systems with Key Transparency</h2>
      <a href="#securing-e2ee-messaging-systems-with-key-transparency">
        
      </a>
    </div>
    
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/71b8HJK1iT0udJscvkqqI4/778efb329047fca017ff2cf4153330ad/8.png" />
          </figure><p><a href="https://www.cloudflare.com/learning/privacy/what-is-end-to-end-encryption/"><u>End-to-end encrypted (E2EE)</u></a> messaging apps like WhatsApp and Signal have become essential tools for private communication, relied upon by billions of people worldwide. These apps use <a href="https://www.cloudflare.com/learning/ssl/how-does-public-key-encryption-work/"><u>public-key cryptography</u></a> to ensure that only the sender and recipient can read the contents of their messages — not even the messaging service itself. However, there's an often-overlooked vulnerability in this model: users must trust that the messaging app is distributing the correct public keys for each contact.</p><p>If an attacker were able to substitute an incorrect public key in the messaging app's database, they could intercept messages intended for someone else — all without the sender knowing.</p><p>Key Transparency addresses this challenge by creating an auditable, append-only log of public keys — similar in concept to <a href="https://radar.cloudflare.com/certificate-transparency"><u>Certificate Transparency</u></a> for TLS certificates. Messaging apps publish their users' public keys to a transparency log, and independent third parties can verify and vouch that the log has been constructed correctly and consistently over time. In September 2024, Cloudflare <a href="https://blog.cloudflare.com/key-transparency/"><u>announced</u></a> such a Key Transparency auditor for WhatsApp, providing an independent verification layer that helps ensure the integrity of public key distribution for the messaging app's billions of users.</p><p>Today, we're publishing Key Transparency audit data in a new <a href="https://radar.cloudflare.com/key-transparency"><u>Key Transparency section</u></a> on Cloudflare Radar. This section showcases the Key Transparency logs that Cloudflare audits, giving researchers, security professionals, and curious users a window into the health and activity of these critical systems.</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/1LZ1DUzv0SCgBa0XqDURKP/26ccd8b0741073895cbb52aa7f1d5643/image11.png" />
          </figure><p>The new page launches with two monitored logs: WhatsApp and Facebook Messenger Transport. Each monitored log is displayed as a card containing the following information:</p><ul><li><p><b>Status:</b> Indicates whether the log is online, in initialization, or disabled. An "online" status means the log is actively publishing key updates into epochs that Cloudflare audits. (An epoch represents a set of updates applied to the key directory at a specific time.)</p></li><li><p><b>Last signed epoch:</b> The most recent epoch that has been published by the messaging service's log and acknowledged by Cloudflare. By clicking on the eye icon, users can view the full epoch data in JSON format, including the epoch number, timestamp, cryptographic digest, and signature.</p></li><li><p><b>Last verified epoch:</b> The most recent epoch that Cloudflare has verified. Verification involves checking that the transition of the transparency log data structure from the previous epoch to the current one represents a valid tree transformation — ensuring the log has been constructed correctly. The verification timestamp indicates when Cloudflare completed its audit.</p></li><li><p><b>Root:</b> The current root hash of the <a href="https://github.com/facebook/akd"><u>Auditable Key Directory (AKD)</u></a> tree. This hash cryptographically represents the entire state of the key directory at the current epoch. Like the epoch fields, users can click to view the complete JSON response from the auditor.</p></li></ul><p>The data shown on the page is also available via the Key Transparency Auditor API, with endpoints for <a href="https://developers.cloudflare.com/key-transparency/api/auditor-information/"><u>auditor information</u></a> and <a href="https://developers.cloudflare.com/key-transparency/api/namespaces/"><u>namespaces</u></a>.</p><p>If you would like to perform audit proof verification yourself, you can follow the instructions in our <a href="https://blog.cloudflare.com/key-transparency/"><u>Auditing Key Transparency blog post</u></a>. We hope that these use cases are the first of many that we publish in this Key Transparency section in Radar — if your company or organization is interested in auditing for your public key or related infrastructure, you can <a href="https://www.cloudflare.com/lp/privacy-edge/"><u>reach out to us here</u></a>.</p>
    <div>
      <h2>Tracking RPKI ASPA adoption</h2>
      <a href="#tracking-rpki-aspa-adoption">
        
      </a>
    </div>
    
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/2LAbrwY9ziVbe1BzfUyl7K/821a40f86c62dd9b44f7bcaee018dd28/10.png" />
          </figure><p>While the <a href="https://www.cloudflare.com/learning/security/glossary/what-is-bgp/"><u>Border Gateway Protocol (BGP)</u></a> is the backbone of Internet routing, it was designed without built-in mechanisms to verify the validity of the paths it propagates. This inherent trust has long left the global network vulnerable to route leaks and hijacks, where traffic is accidentally or maliciously detoured through unauthorized networks.</p><p>Although <a href="https://en.wikipedia.org/wiki/Resource_Public_Key_Infrastructure"><u>RPKI</u></a> and <a href="https://www.arin.net/resources/manage/rpki/roas/"><u>Route Origin Authorizations (ROAs)</u></a> have successfully hardened the origin of routes, they cannot verify the path traffic takes between networks. This is where <a href="https://datatracker.ietf.org/doc/draft-ietf-sidrops-aspa-verification/"><u>ASPA (Autonomous System Provider Authorization)</u></a><b> </b>comes in. ASPA extends RPKI protection by allowing an <a href="https://www.cloudflare.com/learning/network-layer/what-is-an-autonomous-system/"><u>Autonomous System (AS)</u></a> to cryptographically sign a record listing the networks authorized to propagate its routes upstream. By validating these Customer-to-Provider relationships, ASPA allows systems to detect invalid path announcements with confidence and react accordingly.</p><p>While the specific IETF standard remains <a href="https://datatracker.ietf.org/doc/draft-ietf-sidrops-aspa-verification/"><u>in draft</u></a>, the operational community is moving fast. Support for creating ASPA objects has already landed in the portals of Regional Internet Registries (RIRs) like <a href="https://www.arin.net/announcements/20260120/"><u>ARIN</u></a> and <a href="https://labs.ripe.net/author/tim_bruijnzeels/aspa-in-the-rpki-dashboard-a-new-layer-of-routing-security/"><u>RIPE NCC</u></a>, and validation logic is available in major software routing stacks like <a href="https://www.undeadly.org/cgi?action=article;sid=20231002135058"><u>OpenBGPD</u></a> and <a href="https://bird.network.cz/?get_doc&amp;v=20&amp;f=bird-5.html"><u>BIRD</u></a>.</p><p>To provide better visibility into the adoption of this emerging standard, we have added comprehensive RPKI ASPA support to the <a href="https://radar.cloudflare.com/routing"><u>Routing section</u></a> of Cloudflare Radar. Tracking these records globally allows us to understand how quickly the industry is moving toward better path validation.</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/6SI6A5vd2bAp3QnBAsJFmZ/24e11445eb0309252d759e88dbf2ba62/11.png" />
          </figure><p>Our new ASPA deployment view allows users to examine the growth of ASPA adoption over time, with the ability to visualize trends across the five <a href="https://en.wikipedia.org/wiki/Regional_Internet_registry"><u>Regional Internet Registries</u></a> (RIRs) based on AS registration. You can view the entire history of ASPA entries, dating back to October 1, 2023, or zoom into specific date ranges to correlate spikes in adoption with industry events, such as the introduction of ASPA features on ARIN and RIPE NCC online dashboards.</p><p>Beyond aggregate trends, we have also introduced a granular, searchable explorer for real-time ASPA content. This table view allows you to inspect the current state of ASPA records, searchable by AS number, AS name, or by filtering for only providers or customer ASNs. This allows network operators to verify that their records are published correctly and to view other networks’ configurations.</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/K97G5TC7O1MYwkvFbrdTl/85b27f807401f85d2bbe140f1611a034/12.png" />
          </figure><p>We have also integrated ASPA data directly into the country/region routing pages. Users can now track how different locations are progressing in securing their infrastructure, based on the associated ASPA records from the customer ASNs registered locally.</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/6mhZyfrHexdo1GDAoKZEd7/44b63675595a01939fa4748210d8c482/13.png" />
          </figure><p>On individual AS pages, we have updated the Connectivity section. Now, when viewing the connections of a network, you may see a visual indicator for "ASPA Verified Provider." This annotation confirms that an ASPA record exists authorizing that specific upstream connection, providing an immediate signal of routing hygiene and trust.</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/3lVJY4fZWv3KaFdKwLHfAV/aeb2bc27bdccb6a9025345dbaed5b762/14.png" />
          </figure><p>For ASes that have deployed ASPA, we now display a complete list of authorized provider ASNs along with their details. Beyond the current state, Radar also provides a detailed timeline of ASPA activity involving the AS. This history distinguishes between changes initiated by the AS itself ("As customer") and records created by others designating it as a provider ("As provider"), allowing users to immediately identify when specific routing authorizations were established or modified.</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/ZIlAn2l0sDTLCyEMMcBI9/871b8d7abffe17b3aee060502eaa4c1c/15.png" />
          </figure><p>Visibility is an essential first step toward broader adoption of emerging routing security protocols like ASPA. By surfacing this data, we aim to help operators deploy protections and assist researchers in tracking the Internet's progress toward a more secure routing path. For those who need to integrate this data into their own workflows or perform deeper analysis, we are also exposing these metrics programmatically. Users can now access ASPA content snapshots, historical timeseries, and detailed changes data using the newly introduced endpoints in the<a href="https://developers.cloudflare.com/api/resources/radar/subresources/bgp/subresources/rpki/subresources/aspa/"> <u>Cloudflare Radar API</u></a>.</p>
    <div>
      <h2>As security evolves, so does our data</h2>
      <a href="#as-security-evolves-so-does-our-data">
        
      </a>
    </div>
    <p>Internet security continues to evolve, with new approaches, protocols, and standards being developed to ensure that information, applications, and networks remain secure. The security data and insights available on Cloudflare Radar will continue to evolve as well. The new sections highlighted above serve to expand existing routing security, transparency, and post-quantum insights already available on Cloudflare Radar. </p><p>If you share any of these new charts and graphs on social media, be sure to tag us: <a href="https://x.com/CloudflareRadar"><u>@CloudflareRadar</u></a> (X), <a href="https://noc.social/@cloudflareradar"><u>noc.social/@cloudflareradar</u></a> (Mastodon), and <a href="https://bsky.app/profile/radar.cloudflare.com"><u>radar.cloudflare.com</u></a> (Bluesky). If you have questions or comments, or suggestions for data that you’d like to see us add to Radar, you can reach out to us on social media, or contact us via <a><u>email</u></a>.</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/5jAzDXss7PvszWkwGC0q2g/df14de40bf268052fac11239952fc1ed/16.png" />
          </figure><p></p> ]]></content:encoded>
            <category><![CDATA[Radar]]></category>
            <category><![CDATA[Security]]></category>
            <category><![CDATA[Privacy]]></category>
            <category><![CDATA[Post-Quantum]]></category>
            <category><![CDATA[Routing]]></category>
            <category><![CDATA[Research]]></category>
            <guid isPermaLink="false">1Iy1Qvw9TsOhRwgjUYqFxO</guid>
            <dc:creator>David Belson</dc:creator>
            <dc:creator>Mingwei Zhang</dc:creator>
            <dc:creator>André Jesus</dc:creator>
            <dc:creator>Suleman Ahmad</dc:creator>
            <dc:creator>Sabina Zejnilovic</dc:creator>
            <dc:creator>Thibault Meunier</dc:creator>
            <dc:creator>Mari Galicer</dc:creator>
        </item>
        <item>
            <title><![CDATA[ASPA: making Internet routing more secure]]></title>
            <link>https://blog.cloudflare.com/aspa-secure-internet/</link>
            <pubDate>Fri, 27 Feb 2026 06:00:00 GMT</pubDate>
            <description><![CDATA[ ASPA is the cryptographic upgrade for BGP that helps prevent route leaks by verifying the path network traffic takes. New features in Cloudflare Radar make tracking its adoption easy. ]]></description>
            <content:encoded><![CDATA[ <p>Internet traffic relies on the <a href="https://www.cloudflare.com/learning/security/glossary/what-is-bgp/"><u>Border Gateway Protocol (BGP)</u></a> to find its way between networks. However, this traffic can sometimes be misdirected due to configuration errors or malicious actions. When traffic is routed through networks it was not intended to pass through, it is known as a <a href="https://datatracker.ietf.org/doc/html/rfc7908"><u>route leak</u></a>. We have <a href="https://blog.cloudflare.com/bgp-route-leak-venezuela/"><u>written on our blog</u></a> <a href="https://blog.cloudflare.com/cloudflare-1111-incident-on-june-27-2024/"><u>multiple times</u></a> about <a href="https://blog.cloudflare.com/route-leak-detection-with-cloudflare-radar/"><u>BGP route leaks</u></a> and the impact they have on Internet routing, and a few times we have even alluded to a future of path verification in BGP. </p><p>While the network community has made significant progress in verifying the final destination of Internet traffic, securing the actual path it takes to get there remains a key challenge for maintaining a reliable Internet. To address this, the industry is adopting a new cryptographic standard called <a href="https://datatracker.ietf.org/doc/draft-ietf-sidrops-aspa-verification/"><u>ASPA (Autonomous System Provider Authorization)</u></a>, which is designed to validate the entire path of network traffic and prevent route leaks.</p><p>To help the community track the rollout of this standard, Cloudflare Radar has introduced a new ASPA deployment monitoring feature. This view allows users to observe ASPA adoption trends over time across the five<a href="https://en.wikipedia.org/wiki/Regional_Internet_registry"><u> Regional Internet Registries (RIRs)</u></a>, and view ASPA records and changes over time at the<a href="https://www.cloudflare.com/learning/network-layer/what-is-an-autonomous-system/"><u> Autonomous System (AS)</u></a> level.</p>
    <div>
      <h2>What is ASPA?</h2>
      <a href="#what-is-aspa">
        
      </a>
    </div>
    <p>To understand how ASPA works, it is helpful to look at how the Internet currently secures traffic destinations.</p><p>Today, networks use a secure infrastructure system called <a href="https://en.wikipedia.org/wiki/Resource_Public_Key_Infrastructure"><u>RPKI (Resource Public Key Infrastructure)</u></a>, which has seen <a href="https://blog.apnic.net/2026/02/20/rpkis-2025-year-in-review/"><u>significant deployment growth</u></a> over the past few years. Within RPKI, networks publish specific cryptographic records called ROAs (Route Origin Authorizations). A ROA acts as a verifiable digital ID card, confirming that an Autonomous System (AS) is officially authorized to announce specific IP addresses. This addresses the "origin hijacks" issue, where one network attempts to impersonate another.</p><p><a href="https://datatracker.ietf.org/doc/draft-ietf-sidrops-aspa-verification/"><u>ASPA (Autonomous System Provider Authorization)</u></a> builds directly on this foundation. While a ROA verifies the <i>destination</i>, an ASPA record verifies the <i>journey</i>.</p><p>When data travels across the Internet, it keeps a running log of every network it passes through. In BGP, this log is known as the <a href="https://datatracker.ietf.org/doc/html/rfc4271#section-5.1.2"><code><u>AS_PATH</u></code></a> (Autonomous System Path). ASPA provides networks with a way to officially publish a list of their authorized upstream providers within the RPKI system. This allows any receiving network to look at the <code>AS_PATH</code>, check the associated ASPA records, and verify that the traffic only traveled through an approved chain of networks.</p><p>A ROA helps ensure the traffic arrives at the correct destination, ASPA ensures the traffic takes an intended, authorized route to get there. Let’s take a look at how path evaluation actually works in practice.</p>
    <div>
      <h2>Route leak detection with ASPA</h2>
      <a href="#route-leak-detection-with-aspa">
        
      </a>
    </div>
    <p>How does ASPA know if a route is a <i>detour</i>? It relies on the hierarchy of the Internet.</p><p>In a healthy Internet routing topology (e.g. <a href="https://ieeexplore.ieee.org/document/6363987"><u>“valley-free” routing</u></a>), traffic generally follows a specific path: it travels "up" from a customer to a large provider (like a major ISP), optionally crosses over to another big provider, and then flows "down" to the destination. You can visualize this as a “mountain” shape:</p><ol><li><p><b>The Up-Ramp:</b> Traffic starts at a Customer and travels "up" through larger and larger Providers (ISPs), where ISPs pay other ISPs to transit traffic for them.</p></li><li><p><b>The Apex:</b> It reaches the top tier of the Internet backbone and may cross a single peering link.</p></li></ol><p><b>The Down-Ramp:</b> It travels "down" through providers to reach the destination Customer.</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/1VGuSHfq6GcQZUYLGmoDH3/a1486f40c16e568f32ca2fa81d58ac41/1.png" />
          </figure><p><sup><i>A visualization of "valley-free" routing. Routes propagate up to a provider, optionally across one peering link, and down to a customer.</i></sup></p><p>In this model, a route leak is like a valley, or dip. One type of such leak happens when traffic goes down to a customer and then unexpectedly tries to go back <i>up</i> to another provider. </p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/6eoaEpdIJpCCLbMnNZD5ob/7ceca2a98f2252e8161915b942bf7dbd/2.png" />
          </figure><p>This "down-and-up" movement is undesirable as customers aren't intended nor equipped to transit traffic between two larger network providers.</p>
    <div>
      <h4>How ASPA validation works</h4>
      <a href="#how-aspa-validation-works">
        
      </a>
    </div>
    <p>ASPA gives network operators a cryptographic way to declare their <i>authorized providers,</i> enabling receiving networks to verify that an AS path follows this expected structure.</p><p>ASPA validates AS paths by checking the “chain of relationships” from both ends of the routes propagation:</p><ul><li><p><b>Checking the Up-Ramp:</b> The check starts at the origin and moves forward. At every hop, it asks: <i>"Did this network authorize the next network as a Provider?"</i> It keeps going until the chain stops.</p></li><li><p><b>Checking the Down-Ramp:</b> It does the same thing from the destination of a BGP update, moving backward.</p></li></ul><p>If the "Up" path and the "Down" path overlap or meet at the top, the route is <b>Valid</b>. The mountain shape is intact.</p><p>However, if the two valid paths <b>do not meet</b>, i.e. there is a gap in the middle where authorization is missing or invalid, ASPA reports such paths as problematic. That gap represents the "valley" or the leak.</p>
    <div>
      <h4>Validation process example</h4>
      <a href="#validation-process-example">
        
      </a>
    </div>
    <p>Let’s look at a scenario where a network (AS65539) receives a bad route from a customer (AS65538).</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/7kn8W6c7CaPcMLMycjS5NO/59036dc52a942870e9bb0e377f235dd4/3.png" />
          </figure><p>The customer (AS65538) is trying to send traffic received from one provider (AS65537) "up" to another provider (AS65539), acting like a bridge between providers. This is a <a href="https://datatracker.ietf.org/doc/html/rfc7908#autoid-4"><u>classic route leak</u></a>. Now let’s walk the ASPA validation process.</p><ol><li><p>We check the <b>Up-Ramp</b>: The original source (AS65536) authorizes its provider. (Check passes).</p></li><li><p>We check the <b>Down-Ramp</b>: We start from the destination and look back. We see the customer (AS65538).</p></li><li><p><b>The Mismatch:</b> The up-ramp ends at AS65537, while the down-ramp ends at 65538. The two ramps do not connect.</p></li></ol><p>Because the "Up" path and "Down" path fail to connect, the system flags this as ASPA <b>Invalid</b>. ASPA is required to do this path validation, as without signed ASPA objects in RPKI, we cannot find which networks are authorized to advertise which prefixes to whom. By signing a list of provider networks for each AS, we know which networks should be able to propagate prefixes laterally or upstream.</p>
    <div>
      <h3>ASPA against forged-origin hijacks</h3>
      <a href="#aspa-against-forged-origin-hijacks">
        
      </a>
    </div>
    <p>ASPA can serve as an effective defense against <a href="https://www.usenix.org/conference/nsdi24/presentation/holterbach"><u>forged-origin hijacks</u></a>, where an attacker bypasses Route Origin Validation (ROV) by pretending and advertising a BGP path to a real origin prefix. Although the origin AS remains correct, the relationship between the hijacker and the victim is fabricated.</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/6MNVRoNDzxlHDVGTP2nRPw/87485ad246baa734eef3192fd48012a8/4.png" />
          </figure><p>ASPA exposes this deception by allowing the victim network to cryptographically declare its actual authorized providers; because the hijacker is not on that authorized list, the path is rejected as invalid, effectively preventing the malicious redirection.</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/2KGtsKWBdlNFySIswRb5rD/d26b266c2dc942be9b9f6c6ec383843b/5.png" />
          </figure><p>ASPA cannot fully protect against forged-origin hijacks, however. There is still at least one case where not even ASPA validation can fully prevent this type of attack on a network. An example of a forged-origin hijack that ASPA cannot account for is when a provider forges a path advertisement <i>to their customer.</i></p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/3DPYEXwSUPmWUWvsyxFJHX/b059dae85cf764fdcd5a5257f5ebc373/6.png" />
          </figure><p>Essentially, a provider could “fake” a peering link with another AS to attract traffic from a customer with a short AS_PATH length, even when no such peering link exists. ASPA does not prevent this path forgery by the provider, because ASPA only works off of provider information and knows nothing specific about peering relationships.</p><p>So while ASPA can be an effective means of rejecting forged-origin hijack routes, there are still some rare cases where it will be ineffective, and those are worth noting.</p>
    <div>
      <h2>Creating ASPA objects: just a few clicks away</h2>
      <a href="#creating-aspa-objects-just-a-few-clicks-away">
        
      </a>
    </div>
    <p>Creating an ASPA object for your network (or Autonomous System) is now a simple process in registries like <a href="https://labs.ripe.net/author/tim_bruijnzeels/aspa-in-the-rpki-dashboard-a-new-layer-of-routing-security/"><u>RIPE</u></a> and <a href="https://www.arin.net/announcements/20260120/"><u>ARIN</u></a>. All you need is your AS number and the AS numbers of the providers you purchase Internet transit service from. These are the authorized upstream networks you trust to announce your IP addresses to the wider Internet. In the opposite direction, these are also the networks you authorize to send you a full routing table, which acts as the complete map of how to reach the rest of the Internet.</p><p>We’d like to show you just how easy creating an ASPA object is with a quick example. </p><p>Say we need to create the ASPA object for AS203898, an AS we use for our Cloudflare London office Internet. At the time of writing we have three Internet providers for the office: AS8220, AS2860, and AS1273. This means we will create an ASPA object for AS203898 with those three provider members in a list.</p><p>First, we log into the RIPE <a href="https://dashboard.rpki.ripe.net/#overview"><u>RPKI dashboard</u></a> and navigate to the <a href="https://dashboard.rpki.ripe.net/#aspa"><u>ASPA</u></a> section:</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/3CCFItZpP8JbYCotDGfuM3/c7ad0041ceea4f48c37ff59f416f8242/7.png" />
          </figure><p>Then, we click on “Create ASPA” for the object we want to create an ASPA object for. From there, we just fill in the providers for that AS. </p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/6ABmfhRQQRbLhMat6Ug01K/3a9d73ad8ade315416c4cc6eb9073ada/8.png" />
          </figure><p>It’s as simple as that. After just a short period of waiting, we can query the global RPKI ecosystem and find our ASPA object for AS203898 with the providers we defined. </p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/xAKD1b704fg7SMxeg867P/59ce564867565331e2d531de15dc7e87/Screenshot_2026-02-27_at_11.09.55.png" />
          </figure><p>It’s a similar story with <a href="https://www.arin.net/"><u>ARIN</u></a>, the only other <a href="https://en.wikipedia.org/wiki/Regional_Internet_registry"><u>Regional Internet Registries (RIRs)</u></a> that currently supports the creation of ASPA objects. Log in to <a href="https://account.arin.net/public/login"><u>ARIN online,</u></a> then navigate to Routing Security, and click “Manage RPKI”.</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/PTCdyldcTc1Uc0iazZlg4/51ed97a8ef0d095b947f7ab2bf4b1fd3/9.png" />
          </figure><p>From there, you’ll be able to click on “Create ASPA”. In this example, we will create an object for another one of our ASNs, AS400095.</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/4bLLHAeU4Eaz6RPgSPDzn9/a482b6c7ed4ef78f7346ca80c9a5ba46/10.png" />
          </figure><p>And that’s it – now we have created our ASPA object for AS40095 with provider AS0.</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/9yOZPpMH4olQXu2DNpJPO/cca432fd82e61c6492987b7ccedbdc57/11.png" />
          </figure>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/6emDXxMbU0BSKk5BVXUdJ6/132b9a7279b93e3ae0329dec83a9bfac/Screenshot_2026-02-27_at_11.11.31.png" />
          </figure><p>The “AS0” provider entry is special when used, and means the AS owner attests there are <b>no</b> valid upstream providers for their network. By definition this means every transit-free Tier-1 network should eventually sign an ASPA with only “AS0” in their object, if they truly only have peer and customer relationships.</p>
    <div>
      <h2>New ASPA features in Cloudflare Radar </h2>
      <a href="#new-aspa-features-in-cloudflare-radar">
        
      </a>
    </div>
    <p>We have added a new ASPA deployment monitoring feature to <a href="https://radar.cloudflare.com/"><u>Cloudflare Radar</u></a>. The new ASPA deployment view allows users to examine the growth of ASPA adoption over time, with the ability to visualize trends across the five <a href="https://en.wikipedia.org/wiki/Regional_Internet_registry"><u>Regional Internet Registries</u></a> (RIRs) based on AS registration. </p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/7FoW86CVloqBqZO7wcOiq8/f5f2973db227b8184127f76fdad64dc4/12.png" />
          </figure><p>We have also integrated ASPA data directly into the country/region and ASN routing pages. Users can now track how different locations are progressing in securing their infrastructure, based on the associated ASPA records from the customer ASNs registered locally.</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/1tBhOpIxc6tNOJXWPPAk4U/7c334c9ddb089eb823ce23eb49ddbdc3/13.png" />
          </figure><p>There are also new features when you zoom into a particular Autonomous System (AS), for example <a href="https://radar.cloudflare.com/routing/AS203898#connectivity"><u>AS203898</u></a>.</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/7nTBPD0PgzPOS0eOxpG3rW/65d9b312be2e81e4c59164c98b7b6276/14.png" />
          </figure><p>We can see whether a network’s observed BGP upstream providers are ASPA authorized, their full list of providers in their ASPA object, and the timeline of ASPA changes that involve their AS.</p>
    <div>
      <h2>The road to better routing security</h2>
      <a href="#the-road-to-better-routing-security">
        
      </a>
    </div>
    <p>With ASPA finally becoming a reality, we have our cryptographic upgrade for Internet path validation. However, those who have been around since the start of RPKI for route origin validation know <a href="https://manrs.org/2023/05/estimating-the-timeline-for-aspa-deployment/"><u>this will be a long road</u></a> to actually providing significant value on the Internet. Changes are needed to RPKI Relaying Party (RP) packages, signer implementations, RTR (RPKI-to-Router protocol) software, and BGP implementations to actually use ASPA objects and validate paths with them.</p><p>In addition to ASPA adoption, operators should also configure BGP roles as described within <a href="https://datatracker.ietf.org/doc/rfc9234/"><u>RFC9234</u></a>. The BGP roles configured on BGP sessions will help future ASPA implementations on routers <a href="https://datatracker.ietf.org/doc/html/draft-ietf-sidrops-aspa-verification-24#section-6.3"><u>decide which algorithm to apply</u></a>: <i>upstream</i> or <i>downstream</i>. In other words, BGP roles give us the power as operators to directly tie our intended BGP relationships with another AS to sessions with those neighbors. Check with your routing vendors and make sure they support <a href="https://datatracker.ietf.org/doc/rfc9234/"><u>RFC9234 BGP roles and OTC</u></a> (Only-to-Customer) attribute implementation.</p><p>To get the most out of ASPA, we encourage everyone to create their ASPA objects for their AS<i>. </i>Creating and maintaining these ASPA objects requires careful attention. In the future, as networks use these records to actively block invalid paths, omitting a legitimate provider could cause traffic to be dropped. However, managing this risk is no different from how networks already handle Route Origin Authorizations (ROAs) today. ASPA is the necessary cryptographic upgrade for Internet path validation, and we’re happy it’s here!</p> ]]></content:encoded>
            <category><![CDATA[Security Week]]></category>
            <category><![CDATA[BGP]]></category>
            <category><![CDATA[RPKI]]></category>
            <category><![CDATA[Routing]]></category>
            <category><![CDATA[Routing Security]]></category>
            <category><![CDATA[Radar]]></category>
            <guid isPermaLink="false">5NwDf8fspgoSx9Pgcx1xLy</guid>
            <dc:creator>Mingwei Zhang</dc:creator>
            <dc:creator>Bryton Herdes</dc:creator>
        </item>
        <item>
            <title><![CDATA[A closer look at a BGP anomaly in Venezuela]]></title>
            <link>https://blog.cloudflare.com/bgp-route-leak-venezuela/</link>
            <pubDate>Tue, 06 Jan 2026 08:00:00 GMT</pubDate>
            <description><![CDATA[ There has been speculation about the cause of a BGP anomaly observed in Venezuela on January 2. We take a look at BGP route leaks, and dive into what the data suggests caused the anomaly in question. ]]></description>
            <content:encoded><![CDATA[ <p>As news unfolds surrounding the U.S. capture and arrest of Venezuelan leader Nicolás Maduro, a <a href="https://loworbitsecurity.com/radar/radar16/?cf_target_id=8EBD08FC8E3F122A23413E8273CF4AF3"><u>cybersecurity newsletter</u></a> examined <a href="https://radar.cloudflare.com/"><u>Cloudflare Radar</u></a> data and took note of a routing leak in Venezuela on January 2.</p><p>We dug into the data. Since the beginning of December there have been eleven route leak events, impacting multiple prefixes, where AS8048 is the leaker. Although it is impossible to determine definitively what happened on the day of the event, this pattern of route leaks suggests that the CANTV (AS8048) network, a popular Internet Service Provider (ISP) in Venezuela, has insufficient routing export and import policies. In other words, the BGP anomalies observed by the researcher could be tied to poor technical practices by the ISP rather than malfeasance.</p><p>In this post, we’ll briefly discuss Border Gateway Protocol (BGP) and BGP route leaks, and then dig into the anomaly observed and what may have happened to cause it. </p>
    <div>
      <h3>Background: BGP route leaks</h3>
      <a href="#background-bgp-route-leaks">
        
      </a>
    </div>
    <p>First, let’s revisit what a <a href="https://blog.cloudflare.com/route-leak-detection-with-cloudflare-radar/"><u>BGP route leak</u></a> is. BGP route leaks cause behavior similar to taking the wrong exit off of a highway. While you may still make it to your destination, the path may be slower and come with delays you wouldn’t otherwise have traveling on a more direct route.</p><p>Route leaks were given a formal definition in <a href="https://datatracker.ietf.org/doc/html/rfc7908"><u>RFC7908</u></a> as “the propagation of routing announcement(s) beyond their intended scope.” Intended scope is defined using <a href="https://en.wikipedia.org/wiki/Pairwise"><u>pairwise</u></a> business relationships between networks. The relationships between networks, which in BGP we represent using <a href="https://en.wikipedia.org/wiki/Autonomous_system_(Internet)"><u>Autonomous Systems (ASes)</u></a>, can be one of the following: </p><ul><li><p>customer-provider: A customer pays a provider network to connect them and their own downstream customers to the rest of the Internet</p></li><li><p>peer-peer: Two networks decide to exchange traffic between one another, to each others’ customers, settlement-free (without payment)</p></li></ul><p>In a customer-provider relationship, the provider will announce <i>all routes to the customer</i>. The customer, on the other hand, will advertise <i>only the routes</i> from their own customers and originating from their network directly.</p><p>In a peer-peer relationship, each peer will advertise to one another <i>only their own routes and the routes of their downstream customers</i>. </p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/16jXNbH5R5Q4evGm4oRY5p/08d0474000111923f37a7e53b809b5c2/BLOG-3107_2.png" />
          </figure><p>These advertisements help direct traffic in expected ways: from customers upstream to provider networks, potentially across a single peering link, and then potentially back down to customers on the far end of the path from their providers. </p><p>A valid path would look like the following that abides by the <a href="https://ieeexplore.ieee.org/document/6363987"><u>valley-free routing</u></a> rule: </p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/3qKtxTWTrGcpMm8u3nAjYT/dd19181418076c0e12b6035154639e75/BLOG-3107_3.png" />
          </figure><p>A <b>route leak</b> is a violation of valley-free routing where an AS takes routes from a provider or peer and redistributes them to another provider or peer. For example, a BGP path should never go through a “valley” where traffic goes up to a provider, and back down to a customer, and then up to a provider again. There are different types of route leaks defined in RFC7908, but a simple one is the Type 1: Hairpin route leak between two provider networks by a customer. </p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/4XdHugzuQAVhucuninsUfB/43912258386debc0500e3ceb7c8abab2/BLOG-3107_4.png" />
          </figure><p>In the figure above, AS64505 takes routes from one of its providers and redistributes them to their other provider. This is unexpected, since we know providers should not use their customer as an intermediate IP transit network. AS64505 would become overwhelmed with traffic, as a smaller network with a smaller set of backbone and network links than its providers. This can become very impactful quickly. </p>
    <div>
      <h3>Route leak by AS8048 (CANTV)</h3>
      <a href="#route-leak-by-as8048-cantv">
        
      </a>
    </div>
    <p>Now that we have reminded ourselves what a route leak is in BGP, let’s examine what was hypothesized  in <a href="https://loworbitsecurity.com/radar/radar16/?cf_history_state=%7B%22guid%22%3A%22C255D9FF78CD46CDA4F76812EA68C350%22%2C%22historyId%22%3A106%2C%22targetId%22%3A%2251107FD345D9B86C319316904C23F966%22%7D"><u>the newsletter post</u></a>. The post called attention to a <a href="https://radar.cloudflare.com/routing/as8048#bgp-route-leaks"><u>few route leak anomalies</u></a> on Cloudflare Radar involving AS8048. On the <a href="https://radar.cloudflare.com/routing/anomalies/leak-462460"><u>Radar page</u></a> for this leak, we see this information:</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/7d1gGdOtSEvPxciyfNswhw/619965acdc1a7a4d3eafbd99b0ccb9f3/BLOG-3107_5.png" />
          </figure><p>We see the leaker AS, which is AS8048 — CANTV, Venezuela’s state-run telephone and Internet Service Provider. We observe that routes were taken from one of their providers AS6762 (Sparkle, an Italian telecom company) and then redistributed to AS52320 (V.tal GlobeNet, a Colombian network service provider). This is definitely a route leak. </p><p>The newsletter suggests “BGP shenanigans” and posits that such a leak could be exploited to collect intelligence useful to government entities. </p><p>While we can’t say with certainty what caused this route leak, our data suggests that its likely cause was more mundane. That’s in part because BGP route leaks happen all of the time, and they have always been part of the Internet — most often for reasons that aren’t malicious.</p><p>To understand more, let’s look closer at the impacted prefixes and networks. The prefixes involved in the leak were all originated by AS21980 (Dayco Telecom, a Venezuelan company):</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/42cmmWdskKqGw7bByd3tGs/c3841ffa205b241798593b91194d25b1/BLOG-3107_6.png" />
          </figure><p>The prefixes are also all members of the same 200.74.224.0/20 <a href="https://www.cloudflare.com/learning/network-layer/what-is-a-subnet/"><u>subnet</u></a>, as noted by the newsletter author. Much more intriguing than this, though, is the relationship between the originating network AS21980 and the leaking network AS8048: AS8048 is a <i>provider</i> of AS21980. </p><p>The customer-provider relationship between AS8048 and AS21980 is visible in both <a href="https://radar.cloudflare.com/routing/as21980#connectivity"><u>Cloudflare Radar</u></a> and <a href="https://bgp.tools/as/21980#upstreams"><u>bgp.tools</u></a> AS relationship interference data. We can also get a confidence score of the AS relationship using the monocle tool from <a href="https://bgpkit.com/"><u>BGPKIT</u></a>, as you see here: </p><p><code>➜  ~ monocle as2rel 8048 21980
Explanation:
- connected: % of 1813 peers that see this AS relationship
- peer: % where the relationship is peer-to-peer
- as1_upstream: % where ASN1 is the upstream (provider)
- as2_upstream: % where ASN2 is the upstream (provider)</code></p><p><code>Data source: https://data.bgpkit.com/as2rel/as2rel-latest.json.bz2</code></p><p><code>╭──────┬───────┬───────────┬──────┬──────────────┬──────────────╮
│ asn1 │ asn2  │ connected │ peer │ as1_upstream │ as2_upstream │
├──────┼───────┼───────────┼──────┼──────────────┼──────────────┤
│ 8048 │ 21980 │    9.9%   │ 0.6% │     9.4%     │ 0.0%         │
╰──────┴───────┴───────────┴──────┴──────────────┴──────────────╯</code></p><p>While only 9.9% of route collectors see these two ASes as adjacent, almost all of the paths containing them reflect AS8048 as an upstream provider for AS21980, meaning confidence is high in the provider-customer relationship between the two.</p><p>Many of the leaked routes were also heavily prepended with AS8048, meaning it would have been <a href="https://blog.cloudflare.com/prepends-considered-harmful/"><u>potentially</u></a> <i>less</i> attractive for routing when received by other networks. <b>Prepending</b> is the padding of an AS more than one time in an outbound advertisement by a customer or peer, to attempt to switch traffic away from a particular circuit to another. For example, many of the paths during the leak by AS8048 looked like this: “52320,8048,8048,8048,8048,8048,8048,8048,8048,8048,23520,1299,269832,21980”. </p><p>You can see that AS8048 has sent their AS multiple times in an advertisement to AS52320, because by means of BGP loop prevention the path would never actually travel in and out of AS8048 multiple times in a row. A non-prepended path would look like this: “52320,8048,23520,1299,269832,21980”. </p><p>If AS8048 was intentionally trying to become a <a href="https://en.wikipedia.org/wiki/Man-in-the-middle_attack"><u>man-in-the-middle (MITM)</u></a> for traffic, why would they make the BGP advertisement less attractive instead of <i>more </i>attractive? Also, why leak prefixes to try and MITM traffic when you’re <i>already</i> a provider for the downstream AS anyway? That wouldn’t make much sense. </p><p>The leaks from AS8048 also surfaced in multiple separate announcements, each around an hour apart on January 2, 2026 between 15:30 and 17:45 UTC, suggesting they may have been having network issues that surfaced in a routing policy issue or a convergence-based mishap. </p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/4LrT6YC3j7V9p6ab0OYEmA/cebeba76857f7976d8dd4912371c1c43/BLOG-3107_7.png" />
          </figure><p>It is also noteworthy that these leak events begin over twelve hours prior to the <a href="https://www.nytimes.com/2026/01/03/world/americas/venezuela-maduro-capture-trump.html"><u>U.S. military strikes in Venezuela</u></a>. Leaks that impact South American networks <a href="https://radar.cloudflare.com/routing/br#routing-anomalies"><u>are common</u></a>, and we have no reason to believe, based on timing or the other factors I have discussed, that the leak is related to the capture of Maduro several hours later.</p><p>In fact, looking back the past two months, we can see plenty of leaks by AS8048 that are just like this one, meaning this is not a new BGP anomaly:</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/2Md8BdHtP1GIDkyNT8PTbD/3915068dc6f47046665410b665e29853/BLOG-3107_8.png" />
          </figure><p>You can see above in the history of Cloudflare Radar’s route leak alerting pipeline that AS8048 is no stranger to Type 1 hairpin route leaks. Since the beginning of December alone there have been <b>eleven route leak events</b> where AS8048 is the leaker.</p><p>From this we can draw a more innocent possible explanation about the route leak: AS8048 may have configured too loose of export policies facing at least one of their providers, AS52320. And because of that, redistributed routes belong to their customer even when the direct customer BGP routes were missing. If their export policy toward AS52320 only matched on <a href="https://blog.cloudflare.com/monitoring-as-sets-and-why-they-matter/"><u>IRR-generated</u></a> prefix list and not a <i>customer</i> BGP <a href="https://datatracker.ietf.org/doc/html/rfc1997"><u>community</u></a> tag, for example, it would make sense why an indirect path toward AS6762 was leaked back upstream by AS8048. </p><p>These types of policy errors are something <a href="https://datatracker.ietf.org/doc/rfc9234/"><u>RFC9234</u></a> and the Only-to-Customer (OTC) attribute would help with considerably, by coupling BGP more tightly to customer-provider and peer-peer roles, when supported <a href="https://blog.apnic.net/2025/09/05/preventing-route-leaks-made-simple-bgp-roleplay-with-junos-rfc-9234/"><u>by all routing vendors</u></a>. I will save the more technical details on RFC9234 for a follow-up blog post.</p>
    <div>
      <h3>The difference between origin and path validation</h3>
      <a href="#the-difference-between-origin-and-path-validation">
        
      </a>
    </div>
    <p>The newsletter also calls out as “notable” that Sparkle (AS6762) does not implement <a href="https://rpki.cloudflare.com/"><u>RPKI (Resource Public Key Infrastructure)</u></a> Route Origin Validation (ROV). While it is true that AS6762 appears to have an <a href="https://stats.labs.apnic.net/rpki/AS6762?a=6762&amp;c=IT&amp;ll=1&amp;ss=0&amp;mm=1&amp;vv=1&amp;w=7&amp;v=0"><u>incomplete deployment</u></a> of ROV and is flagged as “unsafe” on <a href="http://isbgpsafeyet.com"><u>isbgpsafeyet.com</u></a> <a href="https://isbgpsafeyet.com/#faq"><u>because of it</u></a>, origin validation would not have prevented this BGP anomaly in Venezuela. </p><p>It is important to separate BGP anomalies into two categories: route misoriginations, and path-based anomalies. Knowing the difference between the two helps to understand the solution for each. Route misoriginations, often called BGP hijacks, are meant to be fixed by RPKI Route Origin Validation (ROV) by making sure the originator of a prefix is who rightfully owns it. In the case of the BGP anomaly described in this post, the origin AS was correct as AS21980 and <b>only</b> the path was anomalous. This means ROV wouldn’t help here.</p><p>Knowing that, we need path-based validation. This is what <a href="https://datatracker.ietf.org/doc/draft-ietf-sidrops-aspa-verification/"><u>Autonomous System Provider Authorization (ASPA)</u></a>, an upcoming draft standard in the IETF, is going to provide. The idea is similar to RPKI Route Origin Authorizations (ROAs) and ROV: create an ASPA object that defines a list of authorized providers (upstreams) for our AS, and everyone will use this to invalidate route leaks on the Internet at various vantage points. Using a concrete example, AS6762 is a <a href="https://en.wikipedia.org/wiki/Tier_1_network"><u>Tier-1</u></a> transit-free network, and they would use the special reserved “AS0” member in their ASPA signed object to communicate to the world that they have no upstream providers, only lateral peers and customers. Then, AS52320, the other provider of AS8048, would see routes from their customer with “6762” in the path and reject them by performing an ASPA verification process.</p><p>ASPA is based on RPKI and is exactly what would help prevent route leaks similar to the one we observed in Venezuela.</p>
    <div>
      <h3>A safer BGP, built together </h3>
      <a href="#a-safer-bgp-built-together">
        
      </a>
    </div>
    <p>We felt it was important to offer an alternative explanation for the BGP route leak by AS8048 in Venezuela that was observed on Cloudflare Radar. It is helpful to understand that route leaks are an expected side effect of BGP historically being based entirely on trust and carefully-executed business relationship-driven intent. </p><p>While route leaks could be done with malicious intent, the data suggests this event may have been an accident caused by a lack of routing export and import policies that would prevent it. This is why to have a safer BGP and Internet we need to work together and drive adoption of RPKI-based ASPA, for which <a href="https://www.ripe.net/manage-ips-and-asns/resource-management/rpki/aspa/"><u>RIPE recently released object creation</u></a>, on the wide Internet. It will be a collaborative effort, just like RPKI has been for origin validation, but it will be worth it and prevent BGP incidents such as the one in Venezuela. </p><p>In addition to ASPA, we can all implement simpler mechanisms such as <a href="https://github.com/job/peerlock"><u>Peerlock</u></a> and <a href="https://archive.nanog.org/sites/default/files/Snijders_Everyday_Practical_Bgp.pdf"><u>Peerlock-lite</u></a> as operators, which sanity-checks received paths for obvious leaks. One especially promising initiative is the adoption of <a href="https://datatracker.ietf.org/doc/rfc9234/"><u>RFC9234</u></a>, which should be used in addition to ASPA for preventing route leaks with the establishing of BGP roles and a new Only-To-Customer (OTC) attribute. If you haven’t already asked your routing vendors for an implementation of RFC9234 to be on their roadmap: <i>please</i> <i>do</i>. You can help make a big difference.</p><p><i>Update: Sparkle (AS6762) finished RPKI ROV deployment and </i><a href="https://github.com/cloudflare/isbgpsafeyet.com/pull/829"><i><u>was marked safe</u></i></a><i> on February 3, 2026.</i></p> ]]></content:encoded>
            <category><![CDATA[BGP]]></category>
            <category><![CDATA[RPKI]]></category>
            <category><![CDATA[Routing]]></category>
            <category><![CDATA[Routing Security]]></category>
            <category><![CDATA[Network Services]]></category>
            <guid isPermaLink="false">4WOdNrtTGvlrQDV7Apw8R1</guid>
            <dc:creator>Bryton Herdes</dc:creator>
        </item>
        <item>
            <title><![CDATA[BGP zombies and excessive path hunting]]></title>
            <link>https://blog.cloudflare.com/going-bgp-zombie-hunting/</link>
            <pubDate>Fri, 31 Oct 2025 15:30:00 GMT</pubDate>
            <description><![CDATA[ A BGP “zombie” is essentially a route that has become stuck in the Default-Free Zone (DFZ) of the Internet, potentially due to a missed or lost prefix withdrawal. We’ll walk through some situations where BGP zombies are more likely to rise from the dead and wreak havoc.
 ]]></description>
            <content:encoded><![CDATA[ <p>Here at Cloudflare, we’ve been celebrating Halloween with some zombie hunting of our own. The zombies we’d like to remove are those that disrupt the core framework responsible for how the Internet routes traffic: <a href="http://cloudflare.com/learning/security/glossary/what-is-bgp/"><u>BGP (Border Gateway Protocol)</u></a>.</p><p>A <a href="https://dl.acm.org/doi/10.1145/3472305.3472315"><u>BGP zombie</u></a> is a silly name for a route that has become stuck in the Internet’s <a href="https://en.wikipedia.org/wiki/Default-free_zone"><u>Default-Free Zone</u></a>, aka the DFZ: the collection of all internet routers that do not require a default route, potentially due to a missed or lost prefix withdrawal.</p><p>The underlying root cause of a zombie could be multiple things, spanning from buggy software in routers or just general route processing slowness. It’s when a BGP prefix is meant to be gone from the Internet, but for one reason or another it becomes a member of the undead and hangs around for some period of time.</p><p>The longer these zombies linger, the more they create operational impact and become a real headache for network operators. Zombies can lead packets astray, either by trapping them inside of route loops or by causing them to take an excessively scenic route. Today, we’d like to celebrate Halloween by covering how BGP zombies form and how we can lessen the likelihood that they wreak havoc on Internet traffic.</p>
    <div>
      <h2>Path hunting</h2>
      <a href="#path-hunting">
        
      </a>
    </div>
    <p>To understand the slowness that can often lead to BGP zombies, we need to talk about path hunting. <a href="https://www.noction.com/blog/bgp-path-hunting"><u>Path hunting</u></a> occurs when routers running BGP exhaustively search for the best path to a prefix as determined by <a href="https://en.wikipedia.org/wiki/Longest_prefix_match"><u>Longest Prefix Matching</u></a> (LPM) and BGP routing attributes like path length and local preference. This becomes relevant in our observations of exactly how routes become stuck, for how long they become stuck, and how visible they are on the Internet.</p><p>For example, path hunting happens when a more-specific BGP prefix is withdrawn from the global routing table, and networks need to fallback to a less-specific BGP advertisement. In this example, we use 2001:db8::/48 for the more-specific BGP announcement and 2001:db8::/32 for the less-specific prefix. When the /48 is withdrawn by the originating <a href="https://www.cloudflare.com/learning/network-layer/what-is-an-autonomous-system/"><u>Autonomous System</u></a> (AS), BGP routers have to recognize that route as missing and begin routing traffic to IPs such as 2001:db8::1 via the 2001:db8::/32 route, which still remains while the prefix 2001:db8::/48 is gone. </p><p>Let’s see what this could look like in action with a few diagrams. </p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/7xRNAHChJUyiMbtBZyLOlF/973d10be053b7b7f088721389c34c10e/BLOG-3059_2.png" />
          </figure><p><sub><i>Diagram 1: Active 2001:db8::/48 route</i></sub></p><p>In this initial state, 2001:db8::/48 is used actively for traffic forwarding, which all flows through AS13335 on the way to AS64511. In this case, AS64511 would be a BYOIP customer of Cloudflare. AS64511 also announces a <i>backup</i> route to another Internet Service Provider (ISP), AS64510, but this route is not active even in AS64510’s routing table for forwarding to 2001:db8::1 because 2001:db8::/48 is a longer prefix match when compared to 2001:db8::/32.</p><p>Things get more interesting when AS64510 signals for 2001:db8::/48 to be withdrawn by Cloudflare (AS13335), perhaps because a DDoS attack is over and the customer opts to use Cloudflare only when they are actively under attack.</p><p>When the customer signals to Cloudflare (via BGP Control or API call) to withdraw the 2001:db8::/48 announcement, all BGP routers have to <a href="https://en.wikipedia.org/wiki/Convergence_(routing)"><u>converge</u></a> upon this update, which involves path hunting. AS13335 sends a BGP withdrawal message for 2001:db8::/48 to its directly-connected BGP neighbors. While the news of withdrawal may travel quickly from AS13335 to the other networks, news may travel more quickly to some of the neighbors than others. This means that until everyone has received and processed the withdrawal, networks may try routing through one another to reach the 2001:db8::/48 prefix – even after AS13335 has withdrawn it. </p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/7h3Vba4T7tm6XPB2pIyQex/f5f7c27148bed4dd72959b3820d045ac/BLOG-3059_3.png" />
          </figure><p><sub><i>Diagram 2: 2001:db8::/48 route withdrawn via AS13335</i></sub></p><p>Imagine AS64501 is a little slower than the rest – perhaps due to using older hardware, hardware being overloaded, a software bug, specific configuration settings, poor luck, or some other factor – and still has not processed the withdrawal of the /48. This in itself could be a BGP zombie, since the route is stuck for a small period. Our pings toward 2001:db8::1 are never able to actually reach AS64511, because AS13335 knows the /48 is meant to be withdrawn, but some routers carrying a full table have not yet converged upon that result.</p><p>The length of time spent path hunting is amplified by something called the Minimum Route Advertisement Interval (MRAI). The MRAI specifies the minimum amount of time between BGP advertisement messages from a BGP router, meaning it introduces a purposeful number of seconds of delay between each BGP advertisement update. <a href="https://datatracker.ietf.org/doc/html/rfc4271"><u>RFC4271</u></a> recommends an MRAI value of 30-seconds for eBGP updates, and while this can cut down on the chattiness of BGP, or even potential oscillation of updates, it also makes path hunting take longer. </p><p>At the next cycle of path hunting, even AS64501, which was previously still pointing toward a nonexistent /48 route from AS13335, should find the /32 advertisement is all that is left toward 2001:db8::1. Once it has done so, the traffic flow will become the following: </p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/5sCGMS95R8y32WTjnUigfN/1e5c9a7551c572a08596985edac5c17b/BLOG-3059_4.png" />
          </figure><p><sub><i>Diagram 3: Routing fallback to 2001:db8::/32 and 2001:db8::/48 is gone from DFZ</i></sub></p><p>This would mean BGP path hunting is over, and the Internet has realized that the 2001:db8::/32 is the best route available toward 2001:db8::1, and that 2001:db8::/48 is really gone. While in this example we’ve purposely made path hunting only last two cycles, in reality it can be far more, especially with how highly connected AS13335 is to thousands of peer networks and <a href="https://en.wikipedia.org/wiki/Tier_1_network"><u>Tier-1</u></a>’s globally. </p><p>Now that we’ve discussed BGP path hunting and how it works, you can probably already see how a BGP zombie outbreak can begin and how routing tables can become stuck for a lengthy period of time. Excessive BGP path hunting for a previously-known more-specific prefix can be an early indicator that a zombie could follow.</p>
    <div>
      <h2>Spawning a zombie</h2>
      <a href="#spawning-a-zombie">
        
      </a>
    </div>
    <p>Zombies have captured our attention more recently as they were noticed by some of our customers leveraging <a href="https://developers.cloudflare.com/byoip/"><u>Bring-Your-Own-IP (BYOIP)</u></a> on-demand advertisement for <a href="https://www.cloudflare.com/en-gb/network-services/products/magic-transit/"><u>Magic Transit</u></a>. BYOIP may be configured in two modes: "always-on", in which a prefix is continuously announced, or "on-demand", where a prefix is announced only when a customer chooses to. For some on-demand customers, announcement and withdrawal cycles <i>may</i> be a more frequent occurrence, which can lead to an increase in BGP zombies.</p><p>With that in mind and also knowing how path hunting works, let’s spawn our own zombie onto the Internet. To do so, we’ll take a spare block of IPv4 and IPv6 and announce them like so:</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/20shWBMhqLR3tBMh50v7Uy/bf40e90c2f6a506a5bcfc9bafd1e31d2/BLOG-3059_5.png" />
          </figure><p>Once the routes are announced and stable, we’ll then proceed to withdraw the more specific routes advertised via Cloudflare globally. With a few quick clicks, we’ve successfully re-animated the dead.</p><p><i>Variant A: Ghoulish Gateways</i></p><p>One place zombies commonly occur is between upstream ISPs. When one router in a given ISP’s network is a little slower to update, routes can become stuck. </p><p>Take, for example, the following loop we observed between two of our upstream partners:</p>
            <pre><code>7. be2431.ccr31.sjc04.atlas.cogentco.com
8. tisparkle.sjc04.atlas.cogentco.com
9. 213.144.177.184
10. 213.144.177.184
11. 89.221.32.227
12. (waiting for reply)
13. be2749.rcr71.goa01.atlas.cogentco.com
14. be3219.ccr31.mrs02.atlas.cogentco.com
15. be2066.agr21.mrs02.atlas.cogentco.com
16. telecomitalia.mrs02.atlas.cogentco.com
17. 213.144.177.186
18. 89.221.32.227</code></pre>
            <p></p><p>Or this loop - observed on the same withdrawal test - between two different providers:  </p>
            <pre><code>15. if-bundle-12-2.qcore2.pvu-paris.as6453.net
16. if-bundle-56-2.qcore1.fr0-frankfurt.as6453.net
17. if-bundle-15-2.qhar1.fr0-frankfurt.as6453.net
18. 195.219.223.11
19. 213.144.177.186
20. 195.22.196.137
21. 213.144.177.186
22. 195.22.196.137</code></pre>
            <p><i></i></p><p><i>Variant B: Undead LAN (Local Area Network)</i></p><p>Simultaneously, zombies can occur entirely within a given network. When a route is withdrawn from Cloudflare’s network, each device in our network must individually begin the process of withdrawing the route. While this is generally a smooth process, things can still become stuck.</p><p>Take, for instance, a situation where one router inside of our network has not yet fully processed the withdrawal. Connectivity partners will continue routing traffic towards that router (as they have not yet received the withdrawal) while no host remains behind the router which is capable of actually processing the traffic. The result is an internal-only looping path:</p>
            <pre><code>10. 192.0.2.112
11. 192.0.2.113
12. 192.0.2.112
13. 192.0.2.113
14. 192.0.2.112
15. 192.0.2.113
16. 192.0.2.112
17. 192.0.2.113
18. 192.0.2.112
19. 192.0.2.113
</code></pre>
            <p></p><p>Unlike most fictionally-depicted hoards of the walking dead, our highly-visible zombie has a limited lifetime in most major networks – in this instance, only around around 6 minutes, after which most had re-converged around the less-specific as the best path. Sadly, this is on the shorter side – in some cases, we have seen long-lived zombies cause reachability issues for more than 10 minutes. It’s safe to say this is longer than most network operators would expect BGP convergence to take in a normal situation. </p><p>But, you may ask – is this the excessive path hunting we talked about earlier, or a BGP zombie? Really, it depends on the expectation and tolerance around <a href="https://dl.acm.org/doi/10.1145/3472305.3472315"><u>how long BGP convergence</u></a> should take to process the prefix withdrawal. In any case, even over 30 minutes after our withdrawal of our more-specific prefix, we are able to see zombie routes in the route-views public collectors easily:</p>
            <pre><code>~ % monocle search --start-ts 2025-10-28T12:40:13Z --end-ts 2025-10-28T13:00:13Z --prefix 198.18.0.0/24
A|1761656125.550447|206.82.105.116|54309|198.18.0.0/24|54309 13335 395747|IGP|206.82.104.31|0|0|54309:111|false|||route-views.ny

</code></pre>
            <p></p><p>You might argue that six to eleven minutes (or more) is a reasonable time for worst-case BGP convergence in the Tier-1 network layer, though that itself seems like a stretch. Even setting that aside, our data shows that very real BGP zombies exist in the global routing table, and they will negatively impact traffic. Curiously, we observed the path hunting delay is worse on IPv4, with the longest observed IPv6 impact in major (Tier-1) networks being just over 4 minutes. One could speculate this is in part due to the <a href="https://bgp.potaroo.net/index-bgp.html"><u>much higher number</u></a> of IPv4 prefixes in the Internet global routing table than the IPv6 global table, and how BGP speakers handle them separately.</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/4WOQBZb7MV2a5PA84xm6Be/28e252e5212781ae2d477150692605db/25x_10fps_a.gif" />
          </figure><p><sub><i>Source: RIPEstat’s BGPlay</i></sub></p><p>Part of the delay appears to originate from how interconnected AS13335 is; being heavily peered with a large portion of the Internet increases the likelihood of a route becoming stuck in a given location. Given that, perhaps a zombie would be shorter-lived if we operated in the opposite direction: announcing a less-specific persistently to 13335 and announcing more specifics via our local ISP during normal operation. Since the withdrawal will come from what is likely a less well-peered network, the time-to-convergence may be shorter:</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/4O7r3Nffpbus6ht3Eo6eiF/a7d577042e43c9b4da988cf9bd29f6fe/BLOG-3059_7.png" />
          </figure><p>Indeed, as predicted, we still get a stuck route, and it only lives for around 20 seconds in the Tier-1 network layer:</p>
            <pre><code>19. be12488.ccr42.ams03.atlas.cogentco.com
20. 38.88.214.142
21. be2020.ccr41.ams03.atlas.cogentco.com
22. 38.88.214.142
23. (waiting for reply)
24. 38.88.214.142
25. (waiting for reply)
26. 38.88.214.142
</code></pre>
            <p></p><p>Unfortunately, that 20 seconds is still an impactful 20 seconds - while better, it’s not where we want to be. The exact length of time will depend on the native ISP networks one is connected with, and it could certainly ease into the minutes worth of stuck routing. </p><p>In both cases, the initial time-to-announce yielded no loss, nor was a zombie created, as both paths remained valid for the entirety of their initial lifetime. Zombies were only created when a more specific prefix was fully withdrawn. A newly-announced route is not subject to path hunting in the same way a withdrawn more-specific route is. As they say, good (new) news travels fast.</p>
    <div>
      <h2>Lessening the zombie outbreak</h2>
      <a href="#lessening-the-zombie-outbreak">
        
      </a>
    </div>
    <p>Our findings lead us to believe that the withdrawal of a more-specific prefix may lead to zombies running rampant for longer periods of time. Because of this, we are exploring some improvements that make the consequences of BGP zombie routing less impactful for our customers relying on our on-demand BGP functionality.</p><p>For the traffic that <b>does</b> reach Cloudflare with stuck routes, we will introduce some BGP traffic forwarding improvements internally that allow for a more graceful withdrawal of traffic, even if routes are erroneously pointing toward us. In many ways, this will closely resemble the BGP <a href="https://www.rfc-editor.org/rfc/rfc1997.html"><u>well-known no-export</u></a> community’s functionality from our servers running BGP. This means even if we receive traffic from external parties due to stuck routing, we will still have the opportunity to deliver traffic to our far-end customers over a tunneled connection or via a <a href="https://www.cloudflare.com/network-services/products/network-interconnect/"><u>Cloudflare Network Interconnect</u></a> (CNI). We look forward to reporting back the positive impact after making this improvement for a more graceful draining of traffic by default. </p><p>For the traffic that <b>does not</b> reach Cloudflare’s edge, and instead loops between network providers, we need to use a different approach. Since we know more-specific to less-specific prefix routing fallback is more prone to BGP zombie outbreak, we are encouraging customers to instead use a multi-step draining process when they want traffic drained from the Cloudflare edge for an on-demand prefix without introducing route loops or blackhole events. The draining process when removing traffic for a BYOIP prefix from Cloudflare should look like this: </p><ol><li><p>The customer is already announcing an example prefix from Cloudflare, ex. 198.18.0.0/24</p></li><li><p>The customer begins <i>natively </i>announcing the prefix 198.18.0.0/24 (i.e. the same-length as the prefix they are advertising via Cloudflare) from their network to the Internet Service Providers that they wish to fail over traffic to.</p></li><li><p>After a few minutes, the customer signals BGP withdrawal from Cloudflare for the 198.18.0.0/24 prefix.</p></li></ol><p>The result is a clean cut over: impactful zombies are avoided because the same-length prefix (198.18.0.0/24) remains in the global routing table. Excessive path hunting is avoided because instead of routers needing to aggressively seek out a missing more-specific prefix match, they can fallback to the same-length announcement that persists in the routing table from the natively-originated path to the customer’s network.</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/KmRvPsGUOp5PsNXcKCE1F/78f2c29c8c278d158972114df875ad0c/25x_10fps_b.gif" />
          </figure><p><sub><i>Source: RIPEstat’s BGPlay</i></sub></p>
    <div>
      <h2>What next?</h2>
      <a href="#what-next">
        
      </a>
    </div>
    <p>We are going to continue to refine our methods of measuring BGP zombies, so you can look forward to more insights in the future. There is also <a href="https://www.thousandeyes.com/bgp-stuck-route-observatory/"><u>work from others</u></a> in the <a href="https://blog.benjojo.co.uk/post/bgp-stuck-routes-tcp-zero-window"><u>community</u></a> around zombie measurement that is interesting and producing useful data. In terms of combatting the software bugs around BGP zombie creation, routing vendors should implement <a href="https://datatracker.ietf.org/doc/html/rfc9687"><u>RFC9687</u></a>, the BGP SendHoldTimer. The general idea is that a local router can detect via the SendHoldTimer if the far-end router stops processing BGP messages unexpectedly, which lowers the possibility of zombies becoming stuck for long periods of time. </p><p>In addition, it’s worth keeping in mind our observations made in this post about more-specific prefix announcements and excessive path hunting. If as a network operator you rely on more-specific BGP prefix announcements for failover, or for traffic engineering, you need to be aware that routes could become stuck for a longer period of time before full BGP convergence occurs.</p><p>If you’re interested in problems like BGP zombies, consider <a href="https://www.cloudflare.com/en-gb/careers/jobs/?location=default"><u>coming to work</u></a> at Cloudflare or applying for an <a href="https://www.cloudflare.com/en-gb/careers/early-talent/"><u>internship</u></a>. Together we can help build a better Internet!  </p> ]]></content:encoded>
            <category><![CDATA[BGP]]></category>
            <category><![CDATA[Routing]]></category>
            <category><![CDATA[Network]]></category>
            <category><![CDATA[BYOIP]]></category>
            <guid isPermaLink="false">6Qk6krBb9GkFrf67N6NhyW</guid>
            <dc:creator>Bryton Herdes</dc:creator>
            <dc:creator>June Slater</dc:creator>
            <dc:creator>Mingwei Zhang</dc:creator>
        </item>
        <item>
            <title><![CDATA[Making the Internet observable: the evolution of Cloudflare Radar]]></title>
            <link>https://blog.cloudflare.com/evolution-of-cloudflare-radar/</link>
            <pubDate>Mon, 27 Oct 2025 12:00:00 GMT</pubDate>
            <description><![CDATA[ Cloudflare Radar has evolved significantly since its 2020 launch, offering deeper insights into Internet security, routing, and traffic with new tools and data that help anyone understand what's happening online. ]]></description>
            <content:encoded><![CDATA[ <p>The Internet is constantly changing in ways that are difficult to see. How do we measure its health, spot new threats, and track the adoption of new technologies? When we <a href="https://blog.cloudflare.com/introducing-cloudflare-radar/"><u>launched Cloudflare Radar in 2020</u></a>, our goal was to illuminate the Internet's patterns, helping anyone understand what was happening from a security, performance, and usage perspective, based on aggregated data from Cloudflare services. From the start, Internet measurement, transparency, and resilience has been at the core of our mission.</p><p>The launch blog post noted, “<i>There are three key components that we’re launching today: Radar </i><a href="https://blog.cloudflare.com/introducing-cloudflare-radar/#radar-internet-insights"><i><u>Internet Insights</u></i></a><i>, Radar </i><a href="https://blog.cloudflare.com/introducing-cloudflare-radar/#radar-domain-insights"><i><u>Domain Insights</u></i></a><i> and Radar </i><a href="https://blog.cloudflare.com/introducing-cloudflare-radar/#radar-ip-insights"><i><u>IP Insights</u></i></a><i>.</i>” These components have remained at the core of Radar, and they have been continuously expanded and complemented by other data sets and capabilities to support that mission. By shining a brighter light on Internet security, routing, traffic disruptions, protocol adoption, DNS, and now AI, Cloudflare Radar has become an increasingly comprehensive source of information and insights. And despite our expanding scope, we’ve focused on maintaining Radar’s “easy access” by evolving our information architecture, making our search capabilities more powerful, and building everything on top of a powerful, publicly-accessible API.</p><p>Now more than ever, Internet observability matters. New protocols and use cases compete with new security threats. Connectivity is threatened not only by errant construction equipment, but also by governments practicing targeted content blocking. Cloudflare Radar is uniquely positioned to provide actionable visibility into these trends, threats, and events with local, network, and global level insights, spanning multiple data sets. Below, we explore some highlights of Radar’s evolution over the five years since its launch, looking at how Cloudflare Radar is building one of the industry’s most comprehensive views of what is happening on the Internet.</p>
    <div>
      <h2>Making Internet security more transparent</h2>
      <a href="#making-internet-security-more-transparent">
        
      </a>
    </div>
    <p>The <a href="https://research.cloudflare.com/"><u>Cloudflare Research</u></a> team takes a practical <a href="https://research.cloudflare.com/about/approach/"><u>approach</u></a> to research, tackling projects that have the potential to make a big impact. A number of these projects have been in the security space, and for three of them, we’ve collaborated to bring associated data sets to Radar, highlighting the impact of these projects.</p><p>The 2025 <a href="https://blog.cloudflare.com/new-regional-internet-traffic-and-certificate-transparency-insights-on-radar/#introducing-certificate-transparency-insights-on-radar"><u>launch</u></a> of the <a href="https://radar.cloudflare.com/certificate-transparency"><u>Certificate Transparency (CT) section on Radar</u></a> was the culmination of several months of collaborative work to expand visibility into key metrics for the Certificate Transparency ecosystem, enabling us to deprecate the original <a href="https://blog.cloudflare.com/a-tour-through-merkle-town-cloudflares-ct-ecosystem-dashboard/"><u>Merkle Town CT dashboard</u></a>, which was launched in 2018. Digital certificates are the foundation of trust on the modern Internet, and Certificate Authorities (CAs) serve as trusted gatekeepers, issuing those certificates, with CT logs providing a public, auditable record of every certificate issued, making it possible to detect fraudulent or mis-issued certificates. The information available in the new CT section allows users to explore information about these certificates and CAs, as well as about the CT logs that capture information about every issued certificate.</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/7peWlbK1j0Da36jqjlD6rV/4fd7ef53247992078bbc89bd34f18fa9/image3.png" />
          </figure><p>In 2024, members of Cloudflare’s Research team collaborated with outside researchers to publish a paper titled “<a href="https://research.cloudflare.com/publications/SundaraRaman2023/"><u>Global, Passive Detection of Connection Tampering</u></a>”. Among the findings presented in the paper, it noted that globally, about 20% of all connections to Cloudflare close unexpectedly before any useful data exchange occurs. This unexpected closure is consistent with connection tampering by a third party, which may occur, for instance, when repressive governments seek to block access to websites or applications. Working with the Research team, we added visibility into <a href="https://blog.cloudflare.com/tcp-resets-timeouts/"><u>TCP resets and timeouts</u></a> to the <a href="https://radar.cloudflare.com/security/network-layer#tcp-resets-and-timeouts"><u>Network Layer Security page</u></a> on Radar. This graph, such as the example below for Turkmenistan, provides a perspective on potential connection tampering activity globally, and at a country level. Changes and trends visible in this graph can be used to corroborate reports of content blocking and other local restrictions on Internet connectivity.</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/lxyCbxlW0mUHP9cU0n3Dp/a27081a3926ac4b0917fef1870197fce/image6.png" />
          </figure><p>The research team has been working on post-quantum encryption <a href="https://blog.cloudflare.com/tag/post-quantum/page/2/"><u>since 2017</u></a>, racing improvements in quantum computing to help ensure that today’s encrypted data and communications are resistant to being decrypted in the future. They have led the drive to incorporate post-quantum encryption across Cloudflare’s infrastructure and services, and in 2023 <a href="https://blog.cloudflare.com/post-quantum-crypto-should-be-free/"><u>we announced that it would be included in our delivery services</u></a>, available to everyone and free of charge, forever. However, to take full advantage, support is needed on the client side as well, so to track that, we worked together to add a <a href="https://radar.cloudflare.com/adoption-and-usage#post-quantum-encryption-adoption"><u>graph</u></a> to Radar’s <a href="https://radar.cloudflare.com/adoption-and-usage"><u>Adoption &amp; Usage</u></a> page that tracks the post-quantum encrypted share of HTTPS request traffic. <a href="https://radar.cloudflare.com/adoption-and-usage?dateStart=2024-01-01&amp;dateEnd=2024-01-28#post-quantum-encryption-adoption"><u>Starting 2024 at under 3%</u></a>, it has <a href="https://radar.cloudflare.com/adoption-and-usage?dateStart=2025-10-10&amp;dateEnd=2025-10-16#post-quantum-encryption-adoption"><u>grown to just over 47%</u></a>, thanks to major browsers and code libraries <a href="https://developers.cloudflare.com/ssl/post-quantum-cryptography/pqc-support/"><u>activating post-quantum support by default</u></a>.</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/3l2ceulOBO9S3Yytv7wIUr/c24b02ee132b7ced328993e2557cf765/image11.png" />
          </figure>
    <div>
      <h2>Measuring AI bot &amp; crawler activity</h2>
      <a href="#measuring-ai-bot-crawler-activity">
        
      </a>
    </div>
    <p>The rapid proliferation and growth of AI platforms since the launch of OpenAI’s ChatGPT in November 2022 has upended multiple industries. This is especially true for content creators. Over the last several decades, they generally allowed their sites to be crawled in exchange for the traffic that the search engines would send back to them — traffic that could be monetized in various ways. However, two developments have changed this dynamic. First, AI platforms began aggressively crawling these sites to vacuum up content to use for training their models (with no compensation to content creators). Second, search engines have evolved into answer engines, drastically reducing the amount of traffic they send back to sites. This has led content owners to demand <a href="https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/"><u>solutions</u></a>.</p><p>Among these solutions is providing customers with increased visibility into how frequently AI crawlers are <a href="https://www.cloudflare.com/learning/ai/how-to-prevent-web-scraping/">scraping their content</a>, and Radar has built on that to provide aggregated perspectives on this activity. Radar’s <a href="https://radar.cloudflare.com/ai-insights"><u>AI Insights page</u></a> provides graphs based on crawling traffic, including <a href="https://radar.cloudflare.com/ai-insights?industrySet=Finance#http-traffic-by-bot"><u>traffic trends by bot</u></a> and <a href="https://radar.cloudflare.com/ai-insights?industrySet=Finance#crawl-purpose"><u>traffic trends by crawl purpose</u></a>, both of which can be broken out by industry set as well. Customers can compare the traffic trends we show on the dashboard with trends across their industry.</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/4wNMjFo5eR2gBV78u2ITuD/833d83029224095d22fa0ad96aff9356/image1.png" />
          </figure><p>One key insight is the crawl-to-refer ratio:  a measure of how many HTML pages a crawler consumes in comparison to the number of page visits that they refer back to the crawled site. A view into these ratios by platform, and how they change over time, gives content creators insight into just how significant the reciprocal traffic imbalances are, and the impact of the ongoing transition of search engines into answer engines.</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/7fGlbFfPnuhaizCNZ5Wlr5/4e75c7fbb317428bffa5ea915d2ca428/image5.png" />
          </figure><p>Over the three decades, the humble <a href="https://www.robotstxt.org/robotstxt.html"><u>robots.txt file</u></a> has served as something of a gatekeeper for websites, letting crawlers know <a href="https://www.cloudflare.com/learning/ai/how-to-block-ai-crawlers/">if they are allowed to access content</a> on the site, and if so, which content. Well-behaved crawlers read and parse the file, and adjust their crawling activity accordingly. Based on the robots.txt files found across Radar’s top 10,000 domains, Radar’s AI Insights page shows <a href="https://radar.cloudflare.com/ai-insights#ai-user-agents-found-in-robotstxt"><u>how many of these sites explicitly allow or disallow these AI crawlers to access content</u></a>, and how complete that access/restriction is. With the ability to filter the data by domain category, this graph can provide site owners with visibility into how their peers may be dealing with these AI crawlers.</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/5r1iT7cCKr1OeCx3XlsFVq/78d101224e54673cfd84e513c73f6527/image8.png" />
          </figure>
    <div>
      <h2>Improving Internet resilience with routing visibility</h2>
      <a href="#improving-internet-resilience-with-routing-visibility">
        
      </a>
    </div>
    <p><a href="https://www.cloudflare.com/learning/network-layer/what-is-routing/"><u>Routing</u></a> is the process of selecting a path across one or more networks, and in the context of the Internet, routing selects the paths for <a href="https://www.cloudflare.com/learning/ddos/glossary/internet-protocol/"><u>Internet Protocol (IP)</u></a> packets to travel from their origin to their destination. It is absolutely critical to the functioning of the Internet, but lots of things can go wrong, and when they do, they can take a whole network offline. (And depending on the network, a <a href="https://blog.cloudflare.com/how-verizon-and-a-bgp-optimizer-knocked-large-parts-of-the-internet-offline-today/"><u>larger blast radius</u></a> of sites, applications, and other service providers may be impacted.</p><p>Routing visibility provides insights into the health of a network, and its relationship to other networks. These insights can help identify or troubleshoot problems when they occur. Among the more significant things that can go wrong are route leaks and origin hijacks. <a href="https://blog.cloudflare.com/route-leak-detection-with-cloudflare-radar/#about-bgp-and-route-leaks"><u>Route leaks</u></a> occur when a routing announcement propagates beyond its intended scope — that is, when the announcement reaches networks that it shouldn’t. An <a href="https://blog.cloudflare.com/bgp-hijack-detection/#what-is-bgp-origin-hijacking"><u>origin hijack</u></a> occurs when an attacker creates fake announcements for a targeted prefix, falsely identifying an <a href="https://developers.cloudflare.com/radar/glossary/#autonomous-systems"><u>autonomous systems (AS)</u></a> under their control as the origin of the prefix — in other words, the attacker claims that their network is responsible for a given set of IP addresses, which would cause traffic to those addresses to be routed to them.</p><p>In 2022 and 2023 respectively, we added <a href="https://blog.cloudflare.com/route-leak-detection-with-cloudflare-radar/"><u>route leak</u></a> and <a href="https://blog.cloudflare.com/bgp-hijack-detection/"><u>origin hijack</u></a> detection to <a href="https://radar.cloudflare.com/routing#routing-anomalies"><u>Radar</u></a>, providing network operators and other interested groups (such as researchers) with information to help identify which networks may be party to such events, whether as a leaker/hijacker, or a victim. And perhaps more importantly, in 2023 we also <a href="https://blog.cloudflare.com/traffic-anomalies-notifications-radar/#notifications-overview"><u>launched notifications</u></a> for route leaks and origin hijacks, automatically notifying subscribers via email or webhook when such an event is detected, enabling them to take immediate action.</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/31Q0SVrOitlfKiw4jkT0Po/298ad31e36807c3ebc89aa1adfb149f8/image7.png" />
          </figure><p>In 2025, we further improved this visibility by adding two additional capabilities. The first was <a href="https://blog.cloudflare.com/bringing-connections-into-view-real-time-bgp-route-visibility-on-cloudflare/"><u>real-time BGP route visibility</u></a>, which illustrates how a given network prefix is connected to other networks — what is the route that packets take to get from that set of IP addresses to the large “tier 1” network providers? Network administrators can use this information when facing network outages, implementing new deployments, or investigating route leaks.</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/69wWKFXVxN91YMHydKj8P9/858e3d8d0b90fbf737bb3b0b195b4885/image4.png" />
          </figure><p>An <a href="https://www.apnic.net/manage-ip/using-whois/guide/as-set/"><u>AS-SET</u></a> is a grouping of related networks, historically used for multiple purposes such as grouping together a list of downstream customers of a particular network provider. Our recently announced <a href="https://blog.cloudflare.com/monitoring-as-sets-and-why-they-matter/"><u>AS-SET monitoring</u></a> enables network operators to monitor valid and invalid AS-SET memberships for their networks, which can help prevent misuse and issues like route leaks.</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/5sydRT2tCT7VDJ84S87z7t/1386688e7fb96b477dcf56fbaae090ca/image10.png" />
          </figure>
    <div>
      <h2>Not just pretty pictures</h2>
      <a href="#not-just-pretty-pictures">
        
      </a>
    </div>
    <p>While Radar has been historically focused on providing clear, informative visualizations, we have also launched capabilities that enable users to get at the underlying data more directly, enabling them to use it in a more programmatic fashion. The most important one is the <a href="https://developers.cloudflare.com/api/resources/radar/"><u>Radar API</u></a>, <a href="https://blog.cloudflare.com/radar2/#sharing-insights"><u>launched in 2022</u></a>. Requiring just an access token, users can get access to all the data shown on Radar, as well as some more advanced filters that provide more specific data, enabling them to incorporate Radar data into their own tools, websites, and applications. The example below shows a simple API call that returns the global distribution of human and bot traffic observed over the last seven days.</p>
            <pre><code>curl -X 'GET' \
'https://api.cloudflare.com/client/v4/radar/http/summary/bot_class?name=main&amp;dateRange=1d' \
-H 'accept: application/json' \
-H 'Authorization: Bearer $TOKEN'</code></pre>
            
            <pre><code>{
  "success": true,
  "errors": [],
  "result": {
    "main": {
      "human": "72.520636",
      "bot": "27.479364"
    },
    "meta": {
      "dateRange": [
        {
          "startTime": "2025-10-19T19:00:00Z",
          "endTime": "2025-10-20T19:00:00Z"
        }
      ],
      "confidenceInfo": {
        "level": null,
        "annotations": []
      },
      "normalization": "PERCENTAGE",
      "lastUpdated": "2025-10-20T19:45:00Z",
      "units": [
        {
          "name": "*",
          "value": "requests"
        }
      ]
    }
  }
}</code></pre>
            <p>The <a href="https://www.cloudflare.com/learning/ai/what-is-model-context-protocol-mcp/"><u>Model Context Protocol</u></a> is a standard way to make information available to <a href="https://www.cloudflare.com/learning/ai/what-is-large-language-model/"><u>large language models (LLMs)</u></a>. Somewhat similar to the way an application programming interface (API) works, MCP offers a documented, standardized way for a computer program to integrate services from an external source. It essentially allows <a href="https://www.cloudflare.com/learning/ai/what-is-artificial-intelligence/"><u>AI</u></a> programs to exceed their training, enabling them to incorporate new sources of information into their decision-making and content generation, and helps them connect to external tools. The <a href="https://github.com/cloudflare/mcp-server-cloudflare/tree/main/apps/radar#cloudflare-radar-mcp-server-"><u>Radar MCP server</u></a> allows MCP clients to gain access to Radar data and tools, enabling exploration using natural language queries.</p><p>Radar’s <a href="https://radar.cloudflare.com/scan"><u>URL Scanner</u></a> has proven to be one of its most popular tools, <a href="https://blog.cloudflare.com/building-urlscanner/"><u>scanning millions of sites</u></a> since launching in 2023. It allows users to safely determine whether a site may contain malicious content, as well as providing information on technologies used and insights into the site’s headers, cookies, and links. In addition to being available on Radar, it is also accessible through the API and MCP server.</p><p>Finally, Radar’s user interface has seen a number of improvements over the last several years, in service of improved usability and a better user experience. As new data sets and capabilities are launched, they are added to the search bar, allowing users to search not only for countries and ASNs, but also IP address prefixes, certificate authorities, bot names, IP addresses, and more. Initially launching with just a few default date ranges (such as last 24 hours, last 7 days, etc.), we’ve expanded the number of default options, as well as enabling the user to select custom date ranges of up to one year in length. And because the Internet is global, Radar should be too. In 2024, we <a href="https://blog.cloudflare.com/cloudflare-radar-localization-journey/"><u>launched internationalized versions of Radar</u></a>, marking availability of the site in 14 languages/dialects, including downloaded and embedded content.</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/5zVcB5Wy98ekCJY0wAsx8e/77857f7fe3519a508c3db50a19432e08/image9.png" />
          </figure><p>This is a sampling of the updates and enhancements that we have made to Radar over the last five years in support of Internet measurement, transparency, and resilience. These individual data sets and tools combine to provide one of the most comprehensive views of the Internet available. And we’re not close to being done. We’ll continue to bring additional visibility to the unseen ways that the Internet is changing by adding more tools, data sets, and visualizations, to help users answer more questions in areas including AI, performance, adoption and usage, and security.</p><p>Visit <a href="http://radar.cloudflare.com"><u>radar.cloudflare.com</u></a> to explore all the great data sets, capabilities, and tools for yourself, and to use the Radar <a href="https://developers.cloudflare.com/api/resources/radar/"><u>API</u></a> or <a href="https://github.com/cloudflare/mcp-server-cloudflare/tree/main/apps/radar#cloudflare-radar-mcp-server-"><u>MCP server</u></a> to incorporate Radar data into your own tools, sites, and applications. Keep an eye on the <a href="https://developers.cloudflare.com/changelog/?product=radar"><u>Radar changelog feed</u></a>, <a href="https://developers.cloudflare.com/radar/release-notes/"><u>Radar release notes</u></a>, and the <a href="https://blog.cloudflare.com/tag/cloudflare-radar/"><u>Cloudflare blog</u></a> for news about the latest changes and launches, and don’t hesitate to <a><u>reach out to us</u></a> with feedback, suggestions, and feature requests.</p> ]]></content:encoded>
            <category><![CDATA[Research]]></category>
            <category><![CDATA[Radar]]></category>
            <category><![CDATA[AI]]></category>
            <category><![CDATA[Routing]]></category>
            <category><![CDATA[API]]></category>
            <guid isPermaLink="false">4hyomcz7ZJG76L799PaqhJ</guid>
            <dc:creator>David Belson</dc:creator>
        </item>
        <item>
            <title><![CDATA[How Cloudflare’s systems dynamically route traffic across the globe]]></title>
            <link>https://blog.cloudflare.com/meet-traffic-manager/</link>
            <pubDate>Mon, 25 Sep 2023 13:00:16 GMT</pubDate>
            <description><![CDATA[ Meet the smartest member of the Cloudflare network team that helps keep you online no matter what happens to the Internet underneath us ]]></description>
            <content:encoded><![CDATA[ <p></p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/4NgKY8lcObRRPp4XUwHWRH/fd6462ace767589fd26f2f52aa98252f/image14-1.png" />
            
            </figure><p>Picture this: you’re at an airport, and you’re going through an airport security checkpoint. There are a bunch of agents who are scanning your boarding pass and your passport and sending you through to your gate. All of a sudden, some of the agents go on break. Maybe there’s a leak in the ceiling above the checkpoint. Or perhaps a bunch of flights are leaving at 6pm, and a number of passengers turn up at once. Either way, this imbalance between localized supply and demand can cause huge lines and unhappy travelers — who just want to get through the line to get on their flight. How do airports handle this?</p><p>Some airports may not do anything and just let you suffer in a longer line. Some airports may offer fast-lanes through the checkpoints for a fee. But most airports will tell you to go to another security checkpoint a little farther away to ensure that you can get through to your gate as fast as possible. They may even have signs up telling you how long each line is, so you can make an easier decision when trying to get through.</p><p>At Cloudflare, we have the same problem. We are located in 300 cities around the world that are built to receive end-user traffic for all of our product suites. And in an ideal world, we always have enough computers and bandwidth to handle everyone at their closest possible location. But the world is not always ideal; sometimes we take a data center offline for maintenance, or a connection to a data center goes down, or some equipment fails, and so on. When that happens, we may not have enough attendants to serve every person going through security in every location. It’s not because we haven’t built enough kiosks, but something has happened in our data center that prevents us from serving everyone.</p><p>So, we built Traffic Manager: a tool that balances supply and demand across our entire global network. This blog is about Traffic Manager: how it came to be, how we built it, and what it does now.</p>
    <div>
      <h3>The world before Traffic Manager</h3>
      <a href="#the-world-before-traffic-manager">
        
      </a>
    </div>
    <p>The job now done by Traffic Manager used to be a manual process carried out by network engineers: our network would operate as normal until something happened that caused user traffic to be impacted at a particular data center.</p><p>When such events happened, user requests would start to fail with 499 or 500 errors because there weren’t enough machines to handle the request load of our users. This would trigger a page to our network engineers, who would then remove some Anycast routes for that data center. The end result: by no longer advertising those prefixes in the impacted data center, user traffic would divert to a different data center. This is how Anycast fundamentally works: user traffic is drawn to the closest data center advertising the prefix the user is trying to connect to, as determined by Border Gateway Protocol. For a primer on what Anycast is, check out <a href="https://www.cloudflare.com/learning/cdn/glossary/anycast-network/">this reference article</a>.</p><p>Depending on how bad the problem was, engineers would remove some or even all the routes in a data center. When the data center was again able to absorb all the traffic, the engineers would put the routes back and the traffic would return naturally to the data center.</p><p>As you might guess, this was a challenging task for our network engineers to do every single time any piece of hardware on our network had an issue. It didn’t scale.</p>
    <div>
      <h3>Never send a human to do a machine’s job</h3>
      <a href="#never-send-a-human-to-do-a-machines-job">
        
      </a>
    </div>
    <p>But doing it manually wasn’t just a burden on our Network Operations team. It also resulted in a sub-par experience for our customers; our engineers would need to take time to diagnose and re-route traffic. To solve both these problems, we wanted to build a service that would immediately and automatically detect if users were unable to reach a Cloudflare data center, and withdraw routes from the data center until users were no longer seeing issues. Once the service received notifications that the impacted data center could absorb the traffic, it could put the routes back and reconnect that data center. This service is called Traffic Manager, because its job (as you might guess) is to manage traffic coming into the Cloudflare network.</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/38NInVKI4VizcCukXlF7o4/77758610378a6721ecf30127b6246d0f/image19.png" />
            
            </figure>
    <div>
      <h3>Accounting for second order consequences</h3>
      <a href="#accounting-for-second-order-consequences">
        
      </a>
    </div>
    <p>When a network engineer removes a route from a router, they can make the best guess at where the user requests will move to, and try to ensure that the failover data center has enough resources to handle the requests — if it doesn’t, they can adjust the routes there accordingly prior to removing the route in the initial data center. To be able to automate this process, we needed to move from a world of intuition to a world of data — accurately predicting where traffic would go when a route was removed, and feeding this information to Traffic Manager, so it could ensure it doesn’t make the situation worse.</p>
    <div>
      <h3>Meet Traffic Predictor</h3>
      <a href="#meet-traffic-predictor">
        
      </a>
    </div>
    <p>Although we can adjust which data centers advertise a route, we are unable to influence what proportion of traffic each data center receives. Each time we add a new data center, or a new peering session, the distribution of traffic changes, and as we are in over 300 cities and 12,500 peering sessions, it has become quite difficult for a human to keep track of, or predict the way traffic will move around our network. Traffic manager needed a buddy: Traffic Predictor.</p><p>In order to do its job, Traffic Predictor carries out an ongoing series of real world tests to see where traffic actually moves. Traffic Predictor relies on a testing system that simulates removing a data center from service and measuring where traffic would go if that data center wasn’t serving traffic. To help understand how this system works, let’s simulate the removal of a subset of a data center in Christchurch, New Zealand:</p><ul><li><p>First, Traffic Predictor gets a list of all the IP addresses that normally connect to Christchurch. Traffic Predictor will send a ping request to hundreds of thousands of IPs that have recently made a request there.</p></li><li><p>Traffic Predictor records if the IP responds, and whether the response returns to Christchurch using a special Anycast IP range specifically configured for Traffic Predictor.</p></li><li><p>Once Traffic Predictor has a list of IPs that respond to Christchurch, it removes that route containing that special range from Christchurch, waits a few minutes for the Internet routing table to be updated, and runs the test again.</p></li><li><p>Instead of being routed to Christchurch, the responses are instead going to data centers around Christchurch. Traffic Predictor then uses the knowledge of responses for each data center, and records the results as the failover for Christchurch.</p></li></ul><p>This allows us to simulate Christchurch going offline without actually taking Christchurch offline!</p><p>But Traffic Predictor doesn’t just do this for any one data center. To add additional layers of resiliency, Traffic Predictor even calculates a second layer of indirection: for each data center failure scenario, Traffic Predictor also calculates failure scenarios and creates policies for when surrounding data centers fail.</p><p>Using our example from before, when Traffic Predictor tests Christchurch, it will run a series of tests that remove several surrounding data centers from service including Christchurch to calculate different failure scenarios. This ensures that even if something catastrophic happens which impacts multiple data centers in a region, we still have the ability to serve user traffic. If you think this data model is complicated, you’re right: it takes several days to calculate all of these failure paths and policies.</p><p>Here’s what those failure paths and failover scenarios look like for all of our data centers around the world when they’re visualized:</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/4NxD3OynAnVJFP9yfv9pa1/1b518728e831062aa5bcae1d8329ffe8/image17.png" />
            
            </figure><p>This can be a bit complicated for humans to parse, so let’s dig into that above scenario for Christchurch, New Zealand to make this a bit more clear. When we take a look at failover paths specifically for Christchurch, we see they look like this:</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/6JGMDbR9jjZHFTpQMEkfZI/29dbc0bfeb4fa4147156d331831d8ebc/image20.png" />
            
            </figure><p>In this scenario we predict that 99.8% of Christchurch’s traffic would shift to Auckland, which is able to absorb all Christchurch traffic in the event of a catastrophic outage.</p><p>Traffic Predictor allows us to not only see where traffic will move to if something should happen, but it allows us to preconfigure Traffic Manager policies to move requests out of failover data centers to prevent a thundering herd scenario: where sudden influx of requests can cause failures in a second data center if the first one has issues. With Traffic Predictor, Traffic Manager doesn’t just move traffic out of one data center when that one fails, but it also proactively moves traffic out of other data centers to ensure a seamless continuation of service.</p>
    <div>
      <h3>From a sledgehammer to a scalpel</h3>
      <a href="#from-a-sledgehammer-to-a-scalpel">
        
      </a>
    </div>
    <p>With Traffic Predictor, Traffic Manager can dynamically advertise and withdraw prefixes while ensuring that every datacenter can handle all the traffic. But withdrawing prefixes as a means of traffic management can be a bit heavy-handed at times. One of the reasons for this is that the only way we had to add or remove traffic to a data center was through advertising routes from our Internet-facing routers. Each one of our routes has thousands of IP addresses, so removing only one still represents a large portion of traffic.</p><p>Specifically, Internet applications will advertise prefixes to the Internet from a /24 subnet at an absolute minimum, but many will advertise prefixes larger than that. This is generally done to prevent things like route leaks or route hijacks: many providers will actually filter out routes that are more specific than a /24 (for more information on that, check out this blog on <a href="/route-leak-detection-with-cloudflare-radar/">how we detect route leaks</a>). If we assume that Cloudflare maps protected properties to IP addresses at a 1:1 ratio, then each /24 subnet would be able to service 256 customers, which is the number of IP addresses in a /24 subnet. If every IP address sent one request per second, we’d have to move 4 /24 subnets out of a data center if we needed to move 1,000 requests per second (RPS).</p><p>But in reality, Cloudflare maps a single IP address to hundreds of thousands of protected properties. So for Cloudflare, a /24 might take 3,000 requests per second, but if we needed to move 1,000 RPS out, we would have no choice but to move a single /24 out. And that’s just assuming we advertise at a /24 level. If we used /20s to advertise, the amount we can withdraw gets less granular: at a 1:1 website to IP address mapping, that’s 4,096 requests per second for each prefix, and even more if the website to IP address mapping is many to one.</p><p>While withdrawing prefix advertisements improved the customer experience for those users who would have seen a 499 or 500 error — there may have been a significant portion of users who wouldn’t have been impacted by an issue who still were moved away from the data center they should have gone to, probably slowing them down, even if only a little bit. This concept of moving more traffic out than is necessary is called “stranding capacity”: the data center is theoretically able to service more users in a region but cannot because of how Traffic Manager was built.</p><p>We wanted to improve Traffic Manager so that it only moved the absolute minimum of users out of a data center that was seeing a problem and not strand any more capacity. To do so, we needed to shift percentages of prefixes, so we could be extra fine-grained and only move the things that absolutely need to be moved. To solve this, we built an extension of our <a href="/unimog-cloudflares-edge-load-balancer/">Layer 4 load balancer Unimog,</a> which we call Plurimog.</p><p>A quick refresher on Unimog and layer 4 load balancing: every single one of our machines contains a service that determines whether that machine can take a user request. If the machine can take a user request then it sends the request to our HTTP stack which processes the request before returning it to the user. If the machine can’t take the request, the machine sends the request to another machine in the data center that can. The machines can do this because they are constantly talking to each other to understand whether they can serve requests for users.</p><p>Plurimog does the same thing, but instead of talking between machines, Plurimog talks in between data centers and points of presence. If a request goes into Philadelphia and Philadelphia is unable to take the request, Plurimog will forward to another data center that can take the request, like Ashburn, where the request is decrypted and processed. Because Plurimog operates at layer 4, it can send individual TCP or UDP requests to other places which allows it to be very fine-grained: it can send percentages of traffic to other data centers very easily, meaning that we only need to send away enough traffic to ensure that everyone can be served as fast as possible. Check out how that works in our Frankfurt data center, as we are able to shift progressively more and more traffic away to handle issues in our data centers. This chart shows the number of actions taken on free traffic that cause it to be sent out of Frankfurt over time:</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/2NqLJgn8hYCuHKXRaJfX2P/c666ab9c6c8aba3071a3528b274f997b/pasted-image-0-1.png" />
            
            </figure><p>But even within a data center, we can route traffic around to prevent traffic from leaving the datacenter at all. Our large data centers, called Multi-Colo Points of Presence (MCPs) contain logical sections of compute within a data center that are distinct from one another. These MCP data centers are enabled with another version of Unimog called Duomog, which allows for traffic to be shifted between logical sections of compute automatically. This makes MCP data centers fault-tolerant without sacrificing performance for our customers, and allows Traffic Manager to work within a data center as well as between data centers.</p><p>When evaluating portions of requests to move, Traffic Manager does the following:</p><ul><li><p>Traffic Manager identifies the proportion of requests that need to be removed from a data center or subsection of a data center so that all requests can be served.</p></li><li><p>Traffic Manager then calculates the aggregated space metrics for each target to see how many requests each failover data center can take.</p></li><li><p>Traffic Manager then identifies how much traffic in each plan we need to move, and moves either a proportion of the plan, or all of the plan through Plurimog/Duomog, until we've moved enough traffic. We move Free customers first, and if there are no more Free customers in a data center, we'll move Pro, and then Business customers if needed.</p></li></ul><p>For example, let’s look at Ashburn, Virginia: one of our MCPs. Ashburn has nine different subsections of capacity that can each take traffic. On 8/28, one of those subsections, IAD02, had an issue that reduced the amount of traffic it could handle.</p><p>During this time period, Duomog sent more traffic from IAD02 to other subsections within Ashburn, ensuring that Ashburn was always online, and that performance was not impacted during this issue. Then, once IAD02 was able to take traffic again, Duomog shifted traffic back automatically. You can see these actions visualized in the time series graph below, which tracks the percentage of traffic moved over time between subsections of capacity within IAD02 (shown in green):</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/2tHxLx4GG2vfqVOfscJlCr/c4864e24dc18b7ad1866c716862d79ee/pasted-image-0--1--1.png" />
            
            </figure>
    <div>
      <h3>How does Traffic Manager know how much to move?</h3>
      <a href="#how-does-traffic-manager-know-how-much-to-move">
        
      </a>
    </div>
    <p>Although we used requests per second in the example above, using requests per second as a metric isn’t accurate enough when actually moving traffic. The reason for this is that different customers have different resource costs to our service; a website served mainly from cache with the WAF deactivated is much cheaper CPU wise than a site with all WAF rules enabled and caching disabled. So we record the time that each request takes in the CPU. We can then aggregate the CPU time across each plan to find the CPU time usage per plan. We record the CPU time in ms, and take a per second value, resulting in a unit of milliseconds per second.</p><p>CPU time is an important metric because of the impact it can have on latency and customer performance. As an example, consider the time it takes for an eyeball request to make it entirely through the Cloudflare front line servers: we call this the cfcheck latency. If this number goes too high, then our customers will start to notice, and they will have a bad experience. When cfcheck latency gets high, it’s usually because CPU utilization is high. The graph below shows 95th percentile cfcheck latency plotted against CPU utilization across all the machines in the same data center, and you can see the strong correlation:</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/1ZNFbWlf35KUMAh9SchRHb/4e415f7d847d10bbe68585c9a7695266/Untitled.png" />
            
            </figure><p>So having Traffic Manager look at CPU time in a data center is a very good way to ensure that we’re giving customers the best experience and not causing problems.</p><p>After getting the CPU time per plan, we need to figure out how much of that CPU time to move to other data centers. To do this, we aggregate the CPU utilization across all servers to give a single CPU utilization across the data center. If a proportion of servers in the data center fail, due to network device failure, power failure, etc., then the requests that were hitting those servers are automatically routed elsewhere within the data center by Duomog. As the number of servers decrease, the overall CPU utilization of the data center increases. Traffic Manager has three thresholds for each data center; the maximum threshold, the target threshold, and the acceptable threshold:</p><ul><li><p>Maximum: the CPU level at which performance starts to degrade, where Traffic Manager will take action</p></li><li><p>Target: the level to which Traffic Manager will try to reduce the CPU utilization to restore optimal service to users</p></li><li><p>Acceptable: the level below which a data center can receive requests forwarded from another data center, or revert active moves</p></li></ul><p>When a data center goes above the maximum threshold, Traffic Manager takes the ratio of total CPU time across all plans to current CPU utilization, then applies that to the target CPU utilization to find the target CPU time. Doing it this way means we can compare a data center with 100 servers to a data center with 10 servers, without having to worry about the number of servers in each data center. This assumes that load increases linearly, which is close enough to true for the assumption to be valid for our purposes.</p><p>Target ratio equals current ratio:</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/2EGE8UPINB9MUPtpETprf5/0f82d6166774a63f8e252dea7f169490/Screenshot-2023-09-18-at-22.11.27.png" />
            
            </figure><p>Therefore:</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/uyhAhuKbaFhMpdxtJTBFq/89f9032c5e4380b895f5bf9542ced1d9/Screenshot-2023-09-18-at-22.12.17.png" />
            
            </figure><p>Subtracting the target CPU time from the current CPU time gives us the CPU time to move:</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/4SQDzRDMKby1OrhYxrLhBY/74142207095c5b9a4e869e2fe735ce6f/Screenshot-2023-09-18-at-22.13.14.png" />
            
            </figure><p>For example, if the current CPU utilization was at 90% across the data center, the target was 85%, and the CPU time across all plans was 18,000, we would have:</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/2A074OHvmPertIwkAIb0ev/e49abb26ac4b17cba2a96a4e5ccb7a8f/Screenshot-2023-09-18-at-22.14.02.png" />
            
            </figure><p>This would mean Traffic Manager would need to move 1,000 CPU time:</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/qKnOiyzymcphDes5ee2cS/d7cec01541398c1b3ae1b075622626c9/Screenshot-2023-09-25-at-10.44.41.png" />
            
            </figure><p>Now we know the total CPU time needed to move, we can go through the plans, until the required time to move has been met.</p>
    <div>
      <h3>What is the maximum threshold?</h3>
      <a href="#what-is-the-maximum-threshold">
        
      </a>
    </div>
    <p>A frequent problem that we faced was determining at which point Traffic Manager should start taking action in a data center - what metric should it watch, and what is an acceptable level?</p><p>As said before, different services have different requirements in terms of CPU utilization, and there are many cases of data centers that have very different utilization patterns.</p><p>To solve this problem, we turned to <a href="https://www.cloudflare.com/learning/ai/what-is-machine-learning/">machine learning</a>. We created a service that will automatically adjust the maximum thresholds for each data center according to customer-facing indicators. For our main service-level indicator (SLI), we decided to use the cfcheck latency metric we described earlier.</p><p>But we also need to define a service-level objective (SLO) in order for our machine learning application to be able to adjust the threshold. We set the SLO for 20ms. Comparing our SLO to our SLI, our 95th percentile cfcheck latency should never go above 20ms and if it does, we need to do something. The below graph shows 95th percentile cfcheck latency over time, and customers start to get unhappy when cfcheck latency goes into the red zone:</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/3A9IRcG1ztxPxIxF0ljSLj/9db6c767756a2e0e17b7c8b1d00089fa/Untitled--1-.png" />
            
            </figure><p>If customers have a bad experience when CPU gets too high, then the goal of Traffic Manager’s maximum thresholds are to ensure that customer performance isn’t impacted and to start redirecting traffic away before performance starts to degrade. At a scheduled interval the Traffic Manager service will fetch a number of metrics for each data center and apply a series of machine learning algorithms. After cleaning the data for outliers we apply a simple quadratic curve fit, and we are currently testing a linear regression algorithm.</p><p>After fitting the models we can use them to predict the CPU usage when the SLI is equal to our SLO, and then use it as our maximum threshold. If we plot the cpu values against the SLI we can see clearly why these methods work so well for our data centers, as you can see for Barcelona in the graphs below, which are plotted against curve fit and linear regression fit respectively.</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/695BSI9vd6OJ0jfhHwSj43/df037d425ddf50b3579696c770e6ff80/Untitled--2-.png" />
            
            </figure>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/2LDLfdY2PNOTJjA1FU6lQM/e4341455c659795ce9ee2e004d1eb5b0/Untitled--3-.png" />
            
            </figure><p>In these charts the vertical line is the SLO, and the intersection of this line with the fitted model represents the value that will be used as the maximum threshold. This model has proved to be very accurate, and we are able to significantly reduce the SLO breaches. Let’s take a look at when we started deploying this service in Lisbon:</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/vK8dNvcvRFb0L9WiKuaOI/9ed8d056f903d5ce09b62515c6ecb9c2/Untitled--4-.png" />
            
            </figure><p>Before the change, cfcheck latency was constantly spiking, but Traffic Manager wasn’t taking actions because the maximum threshold was static. But after July 29, we see that cfcheck latency has never hit the SLO because we are constantly measuring to make sure that customers are never impacted by CPU increases.</p>
    <div>
      <h3>Where to send the traffic?</h3>
      <a href="#where-to-send-the-traffic">
        
      </a>
    </div>
    <p>So now that we have a maximum threshold, we need to find the third CPU utilization threshold which isn’t used when calculating how much traffic to move - the acceptable threshold. When a data center is below this threshold, it has unused capacity which, as long as it isn’t forwarding traffic itself, is made available for other data centers to use when required. To work out how much each data center is able to receive, we use the same methodology as above, substituting target for acceptable:</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/1yHCxMMzytg6BnYdigXx1w/9f441adfb38cc474ab8580cdd1b5c72d/Screenshot-2023-09-18-at-22.16.48.png" />
            
            </figure><p>Therefore:</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/6A0qMzXEmTPhVl8KRIiTRO/7830d77baee83d779ff01ff3605f0f09/Screenshot-2023-09-18-at-22.17.36.png" />
            
            </figure><p>Subtracting the current CPU time from the acceptable CPU time gives us the amount of CPU time that a data center could accept:</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/7yQPGjUGnLWy9C9GGOzIRN/ad105de71360d03299039453b554059e/Screenshot-2023-09-18-at-22.18.16.png" />
            
            </figure><p>To find where to send traffic, Traffic Manager will find the available CPU time in all data centers, then it will order them by latency from the data center needing to move traffic. It moves through each of the data centers, using all available capacity based on the maximum thresholds before moving onto the next. When finding which plans to move, we move from the lowest priority plan to highest, but when finding where to send them, we move in the opposite direction.</p><p>To make this clearer let's use an example:</p><p>We need to move 1,000 CPU time from data center A, and we have the following usage per plan: Free: 500ms/s, Pro: 400ms/s, Business: 200ms/s, Enterprise: 1000ms/s.</p><p>We would move 100% of Free (500ms/s), 100% of Pro (400ms/s), 50% of Business (100ms/s), and 0% of Enterprise.</p><p>Nearby data centers have the following available CPU time: B: 300ms/s, C: 300ms/s, D: 1,000ms/s.</p><p>With latencies: A-B: 100ms, A-C: 110ms, A-D: 120ms.</p><p>Starting with the lowest latency and highest priority plan that requires action, we would be able to move all the Business CPU time to data center B and half of Pro. Next we would move onto data center C, and be able to move the rest of Pro, and 20% of Free. The rest of Free could then be forwarded to data center D. Resulting in the following action: Business: 50% → B, Pro: 50% → B, 50% → C, Free: 20% → C, 80% → D.</p>
    <div>
      <h3>Reverting actions</h3>
      <a href="#reverting-actions">
        
      </a>
    </div>
    <p>In the same way that Traffic Manager is constantly looking to keep data centers from going above the threshold, it is also looking to bring any forwarded traffic back into a data center that is actively forwarding traffic.</p><p>Above we saw how Traffic Manager works out how much traffic a data center is able to receive from another data center — it calls this the available CPU time. When there is an active move we use this available CPU time to bring back traffic to the data center — we always prioritize reverting an active move over accepting traffic from another data center.</p><p>When you put this all together, you get a system that is constantly measuring system and customer health metrics for every data center and spreading traffic around to make sure that each request can be served given the current state of our network. When we put all of these moves between data centers on a map, it looks something like this, a map of all Traffic Manager moves for a period of one hour. This map doesn’t show our full data center deployment, but it does show the data centers that are sending or receiving moved traffic during this period:</p><div>
  
</div>
<p></p><p>Data centers in red or yellow are under load and shifting traffic to other data centers until they become green, which means that all metrics are showing as healthy. The size of the circles represent how many requests are shifted from or to those data centers. Where the traffic is going is denoted by where the lines are moving. This is difficult to see at a world scale, so let’s zoom into the United States to see this in action for the same time period:</p><div>
  
</div>
<p></p><p>Here you can see Toronto, Detroit, New York, and Kansas City are unable to serve some requests due to hardware issues, so they will send those requests to Dallas, Chicago, and Ashburn until equilibrium is restored for users and data centers. Once data centers like Detroit are able to service all the requests they are receiving without needing to send traffic away, Detroit will gradually stop forwarding requests to Chicago until any issues in the data center are completely resolved, at which point it will no longer be forwarding anything. Throughout all of this, end users are online and are not impacted by any physical issues that may be happening in Detroit or any of the other locations sending traffic.</p>
    <div>
      <h3>Happy network, happy products</h3>
      <a href="#happy-network-happy-products">
        
      </a>
    </div>
    <p>Because Traffic Manager is plugged into the user experience, it is a fundamental component of the Cloudflare network: it keeps our products online and ensures that they’re as fast and reliable as they can be. It’s our real time load balancer, helping to keep our products fast by only shifting necessary traffic away from data centers that are having issues. Because less traffic gets moved, our products and services stay fast.</p><p>But Traffic Manager can also help keep our products online and reliable because they allow our products to predict where reliability issues may occur and preemptively move the products elsewhere. For example, Browser Isolation directly works with Traffic Manager to help ensure the uptime of the product. When you connect to a Cloudflare data center to create a hosted browser instance, Browser Isolation first asks Traffic Manager if the data center has enough capacity to run the instance locally, and if so, the instance is created right then and there. If there isn’t sufficient capacity available, Traffic Manager tells Browser Isolation which the closest data center with sufficient available capacity is, thereby helping Browser Isolation to provide the best possible experience for the user.</p>
    <div>
      <h3>Happy network, happy users</h3>
      <a href="#happy-network-happy-users">
        
      </a>
    </div>
    <p>At Cloudflare, we operate this huge network to service all of our different products and customer scenarios. We’ve built this network for resiliency: in addition to our MCP locations designed to reduce impact from a single failure, we are constantly shifting traffic around on our network in response to internal and external issues.</p><p>But that is our problem — not yours.</p><p>Similarly, when human beings had to fix those issues, it was customers and end users who would be impacted. To ensure that you’re always online, we’ve built a smart system that detects our hardware failures and preemptively balances traffic across our network to ensure it’s online and as fast as possible. This system works faster than any person — not only allowing our network engineers to sleep at night — but also providing a better, faster experience for all of our customers.</p><p>And finally: if these kinds of engineering challenges sound exciting to you, then please consider checking out the Traffic Engineering team's open position on Cloudflare’s <a href="https://cloudflare.com/careers/jobs">Careers</a> page!</p> ]]></content:encoded>
            <category><![CDATA[Birthday Week]]></category>
            <category><![CDATA[Network]]></category>
            <category><![CDATA[Traffic]]></category>
            <category><![CDATA[Routing]]></category>
            <category><![CDATA[Anycast]]></category>
            <category><![CDATA[Interconnection]]></category>
            <guid isPermaLink="false">lz1BR25h48nezzhD7C1Gr</guid>
            <dc:creator>David Tuber</dc:creator>
            <dc:creator>Luke Orden</dc:creator>
            <dc:creator>Gonçalo Grilo</dc:creator>
        </item>
        <item>
            <title><![CDATA[Routing information now on Cloudflare Radar]]></title>
            <link>https://blog.cloudflare.com/radar-routing/</link>
            <pubDate>Thu, 27 Jul 2023 13:00:30 GMT</pubDate>
            <description><![CDATA[ The Internet is a vast, sprawling collection of networks that connect to each other. The new Cloudflare Radar Routing page monitors routing changes and anomalies and presents statistics about how we see traffic getting routed on the Internet ]]></description>
            <content:encoded><![CDATA[ <p></p><p>Routing is one of the most critical operations of the Internet. Routing decides how and where the Internet traffic should flow from the source to the destination, and can be categorized into two major types: intra-domain routing and inter-domain routing. Intra-domain routing handles making decisions on how individual packets should be routed among the servers and routers within an organization/network. When traffic reaches the edge of a network, the inter-domain routing kicks in to decide what the next hop is and forward the traffic along to the corresponding networks. <a href="https://www.cloudflare.com/learning/security/glossary/what-is-bgp/">Border Gateway Protocol</a> (BGP) is the de facto inter-domain routing protocol used on the Internet.</p><p>Today, we are introducing another section on Cloudflare Radar: the <a href="https://radar.cloudflare.com/routing/">Routing</a> page, which focuses on monitoring the BGP messages exchanged to extract and present insights on the IP prefixes, individual networks, countries, and the Internet overall. The new routing data allows users to quickly examine routing status of the Internet, examine secure routing protocol deployment for a country, identify routing anomalies, validate IP block reachability and much more from globally distributed vantage points.</p><p>It’s a detailed view of how the Internet itself holds together.</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/7zxLtx3iFn9UlEBkUZWhjR/5e6b2a6f4bb10662875dbc83b89375c6/image6-4.png" />
            
            </figure>
    <div>
      <h2>Collecting routing statistics</h2>
      <a href="#collecting-routing-statistics">
        
      </a>
    </div>
    <p>The Internet consists of tens of thousands of interconnected organizations. Each organization manages its own internal networking infrastructure autonomously, and is referred to as an autonomous system (AS). ASes establish connectivity among each other and exchange routing information via BGP messages to form the current Internet.</p><p>When we open the Radar Routing page the “Routing Statistics” block provides a quick glance on the sizes and status of an autonomous system (AS), a country, or the Internet overall. The routing statistics component contains the following count information:</p><ul><li><p>The number of ASes on the Internet or registered from a given country;</p></li><li><p>The number of distinct prefixes and the routes toward them observed on the global routing table, worldwide, by country, or by AS;</p></li><li><p>The number of routes categorized by Resource Public Key Infrastructure (RPKI) validation results (valid, invalid, or unknown).</p></li></ul><p>We also show the breakdown of these numbers for IPv4 and IPv6 separately, so users may have a better understanding of such information with respect to different IP protocols.</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/4Y2m16cdOzvg2PrSIXE8iy/194cb2be2903da4723d9e4826023714e/image5-3.png" />
            
            </figure><p>For a given network, we also show the BGP announcements volume chart for the past week as well as other basic information like network name, registration country, estimated user count, and sibling networks.</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/1i5Z8LS0bHW3QZPSJJhUzP/57957224e6101e54d95fe0e45da34039/image7.png" />
            
            </figure>
    <div>
      <h2>Identifying routing anomalies</h2>
      <a href="#identifying-routing-anomalies">
        
      </a>
    </div>
    <p>BGP as a routing protocol suffers from a number of <a href="/is-bgp-safe-yet-rpki-routing-security-initiative/">security weaknesses</a>. In the new Routing page we consolidate the <a href="/route-leak-detection-with-cloudflare-radar/">BGP route leaks</a> and BGP hijacks detection results in one single place, showing the relevant detected events for any given network or globally.</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/1pro2exMMo1OOpmQNAB6E2/b4be157c3df4165e20d9c0df7e3e1552/image1-11.png" />
            
            </figure><p>The BGP Route Leaks table shows the <a href="/route-leak-detection-with-cloudflare-radar/">detected BGP route leak events</a>. Each entry in the table contains the information about the related ASes of the leak event, start and end time, as well as other numeric statistics that reflect the scale and impact of the event. The BGP Origin Hijacks table shows the detected potential <a href="https://www.cloudflare.com/learning/security/glossary/bgp-hijacking">BGP origin hijacks</a>. Apart from the relevant ASes, time, and impact information, we also show the key evidence that we collected for each event to provide additional context on why and how likely one event being a BGP hijack.</p><p>With this release, we introduce another anomaly detection: RPKI Invalid <a href="https://www.cs.colostate.edu/~massey/pubs/conf/massey_imw01.pdf">Multiple Origin AS (MOAS)</a> is one type of routing conflict where multiple networks (ASes) originate the same IP prefixes at the same time, which goes against <a href="https://datatracker.ietf.org/doc/html/rfc1930#section-7">the best practice recommendation</a>. Our system examines the most recent global routing tables and identifies MOASes on the routing tables. With the help of <a href="https://en.wikipedia.org/wiki/Resource_Public_Key_Infrastructure">Resource Public Key Infrastructure (RPKI)</a>, we can further identify MOAS events that have origins that were proven RPKI invalid, which are less likely to be <a href="https://www.usenix.org/system/files/sec22-qin.pdf">legitimate cases</a>. Users and operators can quickly identify such anomalies relevant to the networks of interest and take actions accordingly.</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/35iYJXb43qkHHrwPmjaDfd/a6f7a8dd556fa19922e9c5a6a3e17dd4/image8.png" />
            
            </figure><p>The Routing page will be the permanent home for all things BGP and routing data in the future; we will gradually introduce more anomaly detections and improve our pipeline to provide more security insights.</p>
    <div>
      <h2>Examining routing assets and connectivity</h2>
      <a href="#examining-routing-assets-and-connectivity">
        
      </a>
    </div>
    <p>Apart from examining the overall routing statistics and anomalies, we also gather information on the routing assets (IP prefixes for a network and networks for a country) and networks’ connectivity.</p><p>Tens of thousands autonomous systems (ASes) connect to each other to form the current Internet. The ASes differ in size and operate in different geolocations. Generally, larger networks are more well-connected and considered “upstream” and smaller networks are less connected and considered “downstream” on the Internet. Below is an example connectivity diagram showing how two smaller networks may connect to each other. AS1 announces its IP prefixes to its upstream providers and propagates upwards until it reaches the large networks AS3 and <code>AS4</code>, and then the route propagates downstream to smaller networks until it reaches AS6.</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/6T5ihQQzQjHUqEx25twDLa/a26decef0f209c733e2ca2c039a61d94/image4-5.png" />
            
            </figure><p>In the routing page, we examine what IP prefixes any given AS originates, as well as the interconnections among ASes. We show the full list of IP prefixes originated for any given AS, including the breakdown lists by RPKI validation status. We also show the detected connectivity among other ASes categorized into upstream, downstream and peering connections. Users can easily search for any ASes upstream, downstream, or peers.</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/4pT2PBWcRIhpIRT4HVovXZ/f8486a73af2e7d9058105467de16746f/image3-4.png" />
            
            </figure><p>For a given country, we show the full list of networks registered in the country, sorted by the number of IP prefixes originated from the corresponding networks. This allows users to quickly glance and find networks from any given country. The table is also searchable by network name or AS number.</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/1URRi90NGshwcyBWIu58cP/3b9cef1364f2341a2a9bff13ef39489f/image2-14.png" />
            
            </figure>
    <div>
      <h2>Routing data API access</h2>
      <a href="#routing-data-api-access">
        
      </a>
    </div>
    <p>Like all the other data, the Cloudflare Radar Routing data is powered by our developer API. The data API is freely available under <a href="https://creativecommons.org/licenses/by-nc/4.0/">Creative Commons Attribution-NonCommercial 4.0 (CC BY-NC 4.0)</a> license. In the following table, we list all of our data APIs available at launch. As we improve the routing section, we will introduce more APIs in the future.</p>
<table>
<thead>
  <tr>
    <th><span>API </span></th>
    <th><span>Type</span></th>
  </tr>
</thead>
<tbody>
  <tr>
    <td><a href="https://developers.cloudflare.com/api/operations/radar-get-bgp-hijacks-events"><span>Get BGP origin hijack events</span></a></td>
    <td><span>Anomaly detection</span></td>
  </tr>
  <tr>
    <td><a href="https://developers.cloudflare.com/api/operations/radar-get-bgp-route-leak-events"><span>Get BGP route leak events</span></a></td>
    <td><span>Anomaly detection</span></td>
  </tr>
  <tr>
    <td><a href="https://developers.cloudflare.com/api/operations/radar-get-bgp-pfx2as-moas"><span>Get MOASes</span></a></td>
    <td><span>Anomaly detection</span></td>
  </tr>
  <tr>
    <td><a href="https://developers.cloudflare.com/api/operations/radar-get-bgp-routes-stats"><span>Get BGP routing table stats</span></a></td>
    <td><span>Routing information</span></td>
  </tr>
  <tr>
    <td><a href="https://developers.cloudflare.com/api/operations/radar-get-bgp-pfx2as"><span>Get prefix-to-origin mapping</span></a></td>
    <td><span>Routing information</span></td>
  </tr>
  <tr>
    <td><a href="https://developers.cloudflare.com/api/operations/radar-get-bgp-top-asns-by-prefixes"><span>Get all ASes registered in a country</span></a></td>
    <td><span>Routing information</span></td>
  </tr>
  <tr>
    <td><a href="https://developers.cloudflare.com/api/operations/radar-get-asns-rel"><span>Get AS-level relationship</span></a></td>
    <td><span>Routing information</span></td>
  </tr>
</tbody>
</table>
    <div>
      <h3>Example 1: lookup origin AS for a given prefix with cURL</h3>
      <a href="#example-1-lookup-origin-as-for-a-given-prefix-with-curl">
        
      </a>
    </div>
    <p>The Cloudflare Radar <a href="https://developers.cloudflare.com/api/operations/radar-get-bgp-pfx2as">prefix-to-origin mapping API</a> returns the matching prefix-origin pairs observed on the global routing tables, allowing users to quickly examine the networks that originate a given prefix or listing all the prefixes a network originates.</p><p>In this example, we ask of “which network(s) originated the prefix 1.1.1.0/24?” using the following cURL command:</p>
            <pre><code>curl --request GET \
  --url "https://api.cloudflare.com/client/v4/radar/bgp/routes/pfx2as?prefix=1.1.1.0/24" \
  --header 'Content-Type: application/json' \
--header "Authorization: Bearer YOUR_TOKEN"</code></pre>
            <p>The returned JSON result shows that Cloudflare (AS13335) originates the prefix <code>1.1.1.0/24</code> and it is a RPKI valid origin. It also returns the meta information such as the UTC timestamp of the query as well as when the dataset is last updated (<code>data_time</code> field).</p>
            <pre><code>{
  "success": true,
  "errors": [],
  "result": {
    "prefix_origins": [
      {
        "origin": 13335,
        "peer_count": 82,
        "prefix": "1.1.1.0/24",
        "rpki_validation": "Valid"
      }
    ],
    "meta": {
      "data_time": "2023-07-24T16:00:00",
      "query_time": "2023-07-24T18:04:55",
      "total_peers": 82
    }
  }
}</code></pre>
            
    <div>
      <h3>Example 2: integrate Radar API into command-line tool</h3>
      <a href="#example-2-integrate-radar-api-into-command-line-tool">
        
      </a>
    </div>
    <p><a href="https://github.com/bgpkit/monocle">BGPKIT monocle</a> is an open-source command-line application that provides multiple utility functions like searching BGP messages on public archives, network lookup by name, RPKI validation status for a given IP prefix, etc.</p><p>By <a href="https://github.com/bgpkit/monocle/pull/50">integrating Cloudflare Radar APIs into monocle</a>, users can now quickly lookup routing statistics or prefix-to-origin mapping by running <code>monocle radar stats [QUERY]</code> and <code>monocle radar pfx2as</code> commands.</p>
            <pre><code>➜  monocle radar stats   
┌─────────────┬─────────┬──────────┬─────────────────┬───────────────┬─────────────────┐
│ scope       │ origins │ prefixes │ rpki_valid      │ rpki_invalid  │ rpki_unknown    │
├─────────────┼─────────┼──────────┼─────────────────┼───────────────┼─────────────────┤
│ global      │ 81769   │ 1204488  │ 551831 (45.38%) │ 15652 (1.29%) │ 648462 (53.33%) │
├─────────────┼─────────┼──────────┼─────────────────┼───────────────┼─────────────────┤
│ global ipv4 │ 74990   │ 1001973  │ 448170 (44.35%) │ 11879 (1.18%) │ 550540 (54.48%) │
├─────────────┼─────────┼──────────┼─────────────────┼───────────────┼─────────────────┤
│ global ipv6 │ 31971   │ 202515   │ 103661 (50.48%) │ 3773 (1.84%)  │ 97922 (47.68%)  │
└─────────────┴─────────┴──────────┴─────────────────┴───────────────┴─────────────────┘

➜  monocle radar pfx2as 1.1.1.0/24
┌────────────┬─────────┬───────┬───────────────┐
│ prefix     │ origin  │ rpki  │ visibility    │
├────────────┼─────────┼───────┼───────────────┤
│ 1.1.1.0/24 │ as13335 │ valid │ high (98.78%) │
└────────────┴─────────┴───────┴───────────────┘</code></pre>
            <p>Visit <a href="https://radar.cloudflare.com/">Cloudflare Radar</a> for additional insights around (Internet disruptions, routing issues, Internet traffic trends, attacks, Internet quality, etc.). Follow us on social media at <a href="https://twitter.com/CloudflareRadar">@CloudflareRadar</a> (Twitter), <a href="https://noc.social/@cloudflareradar">https://noc.social/@cloudflareradar</a> (Mastodon), and <a href="https://bsky.app/profile/radar.cloudflare.com">radar.cloudflare.com</a> (Bluesky), or contact us via <a>e-mail</a>.</p> ]]></content:encoded>
            <category><![CDATA[Radar]]></category>
            <category><![CDATA[Routing]]></category>
            <guid isPermaLink="false">3UOr7t88zpznMhvSj4fx4N</guid>
            <dc:creator>Mingwei Zhang</dc:creator>
        </item>
        <item>
            <title><![CDATA[Why BGP communities are better than AS-path prepends]]></title>
            <link>https://blog.cloudflare.com/prepends-considered-harmful/</link>
            <pubDate>Thu, 24 Nov 2022 17:31:47 GMT</pubDate>
            <description><![CDATA[ Routing on the Internet follows a few basic principles. Unfortunately not everything on the Internet is created equal, and prepending can do more harm than good. In this blog post we’ll talk about the problems that prepending aims to solve, and some alternative solutions ]]></description>
            <content:encoded><![CDATA[ <p><i></i></p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/2E41RLSZSaS34QDzUqX1Rd/7ebce0374cd5f67f333eaf954e6f1445/image7-9.png" />
            
            </figure><p>The Internet, in its purest form, is a loosely connected graph of independent networks (also called <a href="https://www.cloudflare.com/en-gb/learning/network-layer/what-is-an-autonomous-system/">Autonomous Systems</a> (AS for short)). These networks use a signaling protocol called <a href="https://www.cloudflare.com/en-gb/learning/security/glossary/what-is-bgp/">BGP</a> (Border Gateway Protocol) to inform their neighbors (also known as peers) about the reachability of IP prefixes (a group of IP addresses) in and through their network. Part of this exchange contains useful metadata about the IP prefix that are used to inform network routing decisions. One example of the metadata is the full AS-path, which consists of the different autonomous systems an IP packet needs to pass through to reach its destination.</p><p>As we all want our packets to get to their destination as fast as possible, selecting the shortest AS-path for a given prefix is a good idea. This is where something called prepending comes into play.</p>
    <div>
      <h2>Routing on the Internet, a primer</h2>
      <a href="#routing-on-the-internet-a-primer">
        
      </a>
    </div>
    <p>Let's briefly talk about how the Internet works at its most fundamental level, before we dive into some nitty-gritty details.</p><p>The Internet is, at its core, a massively interconnected network of thousands of networks. Each network owns two things that are critical:</p><p>1. An Autonomous System Number (ASN): a 32-bit integer that uniquely identifies a network. For example, one of the Cloudflare ASNs (we have multiple) is 13335.</p><p>2. IP prefixes: An IP prefix is a range of IP addresses, bundled together in powers of two: In the IPv4 space, two addresses form a /31 prefix, four form a /30, and so on, all the way up to /0, which is shorthand for “all IPv4 prefixes''. The same applies for IPv6  but instead of aggregating 32 bits at most, you can aggregate up to 128 bits. The figure below shows this relationship between IP prefixes, in reverse -- a /24 contains two /25s that contains two /26s, and so on.</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/5XUwaT0EJLzfHUwkpm7aN2/47a248cc3292ae8a970423c8f3de9f5b/image9-6.png" />
            
            </figure><p>To communicate on the Internet, you must be able to reach your destination, and that’s where routing protocols come into play. They enable each node on the Internet to know where to send your message (and for the receiver to send a message back).</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/6RzDynXtAzTFAAeSRNF7Qz/f3297d7747eb351a77dfb3da7461944a/image5-21.png" />
            
            </figure><p>As mentioned earlier, these destinations are identified by IP addresses, and contiguous ranges of IP addresses are expressed as IP prefixes. We use IP prefixes for routing as an efficiency optimization: Keeping track of where to go for four billion (2<sup>32</sup>) IP addresses in IPv4 would be incredibly complex, and require a lot of resources. Sticking to prefixes reduces that number down to about one million instead.</p><p>Now recall that Autonomous Systems are independently operated and controlled. In the Internet’s network of networks, how do I tell Source A in some other network that there is an available path to get to Destination B in (or through) my network? In comes BGP! BGP is the Border Gateway Protocol, and it is used to signal reachability information. Signal messages generated by the source ASN are referred to as ‘announcements’ because they declare to the Internet that IP addresses in the prefix are online and reachable.</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/cudmPAIkxVZr5tOuSqQdm/f096de61cdebaa7c9427343be70cb0ff/image4-33.png" />
            
            </figure><p>Have a look at the figure above. Source A should now know how to get to Destination B through 2 different networks!</p><p>This is what an actual BGP message would look like:</p>
            <pre><code>BGP Message
    Type: UPDATE Message
    Path Attributes:
        Path Attribute - Origin: IGP
        Path Attribute - AS_PATH: 64500 64496
        Path Attribute - NEXT_HOP: 198.51.100.1
        Path Attribute - COMMUNITIES: 64500:13335
        Path Attribute - Multi Exit Discriminator (MED): 100
    Network Layer Reachability Information (NLRI):
        192.0.2.0/24</code></pre>
            <p>As you can see, BGP messages contain more than just the IP prefix (the NLRI bit) and the path, but also a bunch of other metadata that provides additional information about the path. Other fields include communities (more on that later), as well as MED, or origin code. MED is a suggestion to other directly connected networks on which path should be taken if multiple options are available, and the lowest value wins. The origin code can be one of three values: IGP, EGP or Incomplete. IGP will be set if you originate the prefix through BGP, EGP is no longer used (it’s an ancient routing protocol), and Incomplete is set when you distribute a prefix into BGP from another routing protocol (like IS-IS or OSPF).</p><p>Now that source A knows how to get to Destination B through two different networks, let's talk about traffic engineering!</p>
    <div>
      <h2>Traffic engineering</h2>
      <a href="#traffic-engineering">
        
      </a>
    </div>
    <p>Traffic engineering is a critical part of the day to day management of any network. Just like in the physical world, detours can be put in place by operators to optimize the traffic flows into (inbound) and out of (outbound) their network. Outbound traffic engineering is significantly easier than inbound traffic engineering because operators can choose from neighboring networks, even prioritize some traffic over others. In contrast, inbound traffic engineering requires influencing a network that is operated by someone else entirely. The autonomy and self-governance of a network is paramount, so operators use available tools to inform or shape inbound packet flows from other networks. The understanding and use of those tools is complex, and can be a challenge.</p><p>The available set of traffic engineering tools, both in- and outbound, rely on manipulating attributes (metadata) of a given route. As we’re talking about traffic engineering between independent networks, we’ll be manipulating the attributes of an EBGP-learned route. BGP can be split into two categories:</p><ol><li><p>EBGP: BGP communication between two different ASNs</p></li><li><p>IBGP: BGP communication within the same ASN.</p></li></ol><p>While the protocol is the same, certain attributes can be exchanged on an IBGP session that aren’t exchanged on an EBGP session. One of those is local-preference. More on that in a moment.</p>
    <div>
      <h3>BGP best path selection</h3>
      <a href="#bgp-best-path-selection">
        
      </a>
    </div>
    <p>When a network is connected to multiple other networks and service providers, it can receive path information to the same IP prefix from many of those networks, each with slightly different attributes. It is then up to the receiving network of that information to use a BGP best path selection algorithm to pick the “best” prefix (and route), and use this to forward IP traffic. I’ve put “best” in quotation marks, as best is a subjective requirement. “Best” is frequently the shortest, but what can be best for my network might not be the best outcome for another network.</p><p>BGP will consider multiple prefix attributes when filtering through the received options. However, rather than combine all those attributes into a single selection criteria, BGP best path selection uses the attributes in tiers -- at any tier, if the available attributes are sufficient to choose the best path, then the algorithm terminates with that choice.</p><p>The BGP best path selection algorithm is extensive, containing 15 discrete steps to select the best available path for a given prefix. Given the numerous steps, it’s in the interest of the network to decide the best path as early as possible. The first four steps are most used and influential, and are depicted in the figure below as sieves.</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/28iOxj73xffT9imICxgTTI/504e3ea0c7767d8acae1ee0ca0b64c4d/image2-55.png" />
            
            </figure><p>Picking the shortest path possible is usually a good idea, which is why “AS-path length” is a step executed early on in the algorithm. However, looking at the figure above, “AS-path length” appears second, despite being the attribute to find the shortest path. So let’s talk about the first step: local preference.</p><p><b>Local preference</b>Local preference is an operator favorite because it allows them to handpick a route+path combination of their choice. It’s the first attribute in the algorithm because it is unique for any given route+neighbor+AS-path combination.</p><p>A network sets the local preference on import of a route (having learned about the route from a neighbor network). Being a non-transitive property, meaning that it’s an attribute that is never sent in an EBGP message to other networks. This intrinsically means, for example, that the operator of AS 64496 can’t set the local preference of routes to their own (or transiting) IP prefixes inside neighboring AS 64511. The inability to do so is partially why inbound traffic engineering through EBGP is so difficult.</p><p><b>Prepending artificially increases AS-path length</b>Since no network is able to directly set the local preference for a prefix inside another network, the first opportunity to influence other networks’ choices is modifying the AS-path. If the next hops are valid, and the local preference for all the different paths for a given route are the same, modifying the AS-path is an obvious option to change the path traffic will take towards your network. In a BGP message, prepending looks like this:</p><p>BEFORE:</p>
            <pre><code>BGP Message
    Type: UPDATE Message
    Path Attributes:
        Path Attribute - Origin: IGP
        Path Attribute - AS_PATH: 64500 64496
        Path Attribute - NEXT_HOP: 198.51.100.1
        Path Attribute - COMMUNITIES: 64500:13335
        Path Attribute - Multi Exit Discriminator (MED): 100
    Network Layer Reachability Information (NLRI):
        192.0.2.0/24</code></pre>
            <p>AFTER:</p>
            <pre><code>BGP Message
    Type: UPDATE Message
    Path Attributes:
        Path Attribute - Origin: IGP
        Path Attribute - AS_PATH: 64500 64496 64496
        Path Attribute - NEXT_HOP: 198.51.100.1
        Path Attribute - COMMUNITIES: 64500:13335
        Path Attribute - Multi Exit Discriminator (MED): 100
    Network Layer Reachability Information (NLRI):
        192.0.2.0/24</code></pre>
            <p>Specifically, operators can do AS-path prepending. When doing AS-path prepending, an operator adds additional autonomous systems to the path (usually the operator uses their own AS, but that’s not enforced in the protocol). This way, an AS-path can go from a length of 1 to a length of 255. As the length has now increased dramatically, that specific path for the route will not be chosen. By changing the AS-path advertised to different peers, an operator can control the traffic flows coming into their network.</p><p>Unfortunately, prepending has a catch: To be the deciding factor, all the other attributes need to be equal. This is rarely true, especially in large networks that are able to choose from many possible routes to a destination.</p>
    <div>
      <h2>Business Policy Engine</h2>
      <a href="#business-policy-engine">
        
      </a>
    </div>
    <p>BGP is colloquially also referred to as a Business Policy Engine: it does <b>not</b> select the best path from a performance point of view; instead, and more often than not, it will select the best path from a <i>business</i> point of view. The business criteria could be anything from investment (port) efficiency to increased revenue, and more. This may sound strange but, believe it or not, this is what BGP is designed to do! The power (and complexity) of BGP is that it enables a network operator to make choices according to the operator’s needs, contracts, and policies, many of which cannot be reflected by conventional notions of engineering performance.</p>
    <div>
      <h3>Different local preferences</h3>
      <a href="#different-local-preferences">
        
      </a>
    </div>
    <p>A lot of networks (including Cloudflare) assign a local preference depending on the type of connection used to send us the routes. A higher value is a higher preference. For example, routes learned from transit network connections will get a lower local preference of 100 because they are the most costly to use; backbone-learned routes will be 150, Internet exchange (IX) routes get 200, and lastly private interconnect (PNI) routes get 250. This means that for egress (outbound) traffic, the Cloudflare network, by default, will prefer a PNI-learned route, even if a shorter AS-path is available through an IX or transit neighbor.</p><p>Part of the reason a PNI is preferred over an IX is reliability, because there is no third-party switching platform involved that is out of our control, which is important because we operate on the assumption that all hardware can and will eventually break. Another part of the reason is for port efficiency reasons. Here, efficiency is defined by cost per megabit transferred on each port. Roughly speaking, the cost is calculated by:</p><p><code>((cost_of_switch / port_count) + transceiver_cost)</code></p><p>which is combined with the cross-connect cost (might be monthly recurring (MRC), or a one-time fee). PNI is preferable because it helps to optimize value by reducing the overall cost per megabit transferred, because the unit price decreases with higher utilization of the port.</p><p>This reasoning is similar for a lot of other networks, and is very prevalent in transit networks. BGP is at least as much about cost and business policy, as it is about performance.</p>
    <div>
      <h3>Transit local preference</h3>
      <a href="#transit-local-preference">
        
      </a>
    </div>
    <p>For simplicity, when referring to transits, I mean the <a href="https://en.wikipedia.org/wiki/Tier_1_network">traditional tier-1 transit networks</a>. Due to the nature of these networks, they have two distinct sets of network peers:</p><p>1. Customers (like Cloudflare)2. Settlement-free peers (like other tier-1 networks)</p><p>In normal circumstances, transit customers will get a higher local preference assigned than the local preference used for their settlement-free peers. This means that, no matter how much you prepend a prefix, if traffic enters that transit network, traffic will <b>always</b> land on your interconnection with that transit network, it will not be offloaded to another peer.</p><p>A prepend can still be used if you want to switch/offload traffic from a single link with one transit if you have multiple distinguished links with them, or if the source of traffic is multihomed behind multiple transits (and they don’t have their own local preference playbook preferring one transit over another). But inbound traffic engineering traffic away from one transit port to another through AS-path prepending has significant diminishing returns: once you’re past three prepends, it’s unlikely to change much, if anything, at that point.</p><p><b>Example</b></p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/8QW8idtVhFb2jtY64uTLf/696a61224544698eb35a8e7ae20f8a22/image8-6.png" />
            
            </figure><p>In the above scenario, no matter the adjustment Cloudflare makes in its AS-path towards AS 64496, the traffic will keep flowing through the Transit B &lt;&gt; Cloudflare interconnection, even though the path Origin A → Transit B → Transit A → Cloudflare is shorter from an AS-path point of view.</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/2uNwYLR8BdzCHVTzISwgtk/581ea233f8058f998bf00dfaa44ca3e3/image6-12.png" />
            
            </figure><p>In this scenario, not a lot has changed, but Origin A is now multi-homed behind the two transit providers. In this case, the AS-path prepending was effective, as the paths seen on the Origin A side are both the prepended and non-prepended path. As long as Origin A is not doing any egress traffic engineering, and is treating both transit networks equally, then the path chosen will be Origin A → Transit A → Cloudflare.</p>
    <div>
      <h3>Community-based traffic engineering</h3>
      <a href="#community-based-traffic-engineering">
        
      </a>
    </div>
    <p>So we have now identified a pretty critical problem within the Internet ecosystem for operators: with the tools mentioned above, it’s not always (some might even say outright impossible) possible to accurately dictate paths traffic can ingress your own network, reducing the control an autonomous system has over its own network. Fortunately, there is a solution for this problem: community-based local preference.</p><p>Some transit providers allow their customers to influence the local preference in the transit network through the use of BGP communities. BGP communities are an optional transitive attribute for a route advertisement. The communities can be informative (“I learned this prefix in Rome”), but they can also be used to trigger actions on the receiving side. For example, Cogent publishes the following action communities:</p><table>
<thead>
  <tr>
    <th>Community</th>
    <th>Local preference</th>
  </tr>
</thead>
<tbody>
  <tr>
    <td>174:10</td>
    <td>10</td>
  </tr>
  <tr>
    <td>174:70</td>
    <td>70</td>
  </tr>
  <tr>
    <td>174:120</td>
    <td>120</td>
  </tr>
  <tr>
    <td>174:125</td>
    <td>125</td>
  </tr>
  <tr>
    <td>174:135</td>
    <td>135</td>
  </tr>
  <tr>
    <td>174:140</td>
    <td>140</td>
  </tr>
</tbody>
</table><p>When you know that Cogent uses the following default local preferences in their network:</p><p>Peers → Local preference 100Customers → Local preference 130</p><p>It’s easy to see how we could use the communities provided to change the route used. It’s important to note though that, as we can’t set the local preference of a route to 100 (or 130), AS-path prepending remains largely irrelevant, as the local preference won’t ever be the same.</p><p>Take for example the following configuration:</p>
            <pre><code>term ADV-SITELOCAL {
    from {
        prefix-list SITE-LOCAL;
        route-type internal;
    }
    then {
        as-path-prepend "13335 13335";
        accept;
    }
}</code></pre>
            
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/2QJPzju5z4aHCOTxSEAO9y/21cf93066b2a67a48f530af5934c58d3/image1-71.png" />
            
            </figure><p>We’re prepending the Cloudflare ASN two times, resulting in a total AS-path of three, yet we were still seeing a lot (too much) traffic coming in on our Cogent link. At that point, an engineer could add another prepend, but for a well-connected network as Cloudflare, if two prepends didn’t do much, or three, then four or five isn’t going to do much either. Instead, we can leverage the Cogent communities documented above to change the routing within Cogent:</p>
            <pre><code>term ADV-SITELOCAL {
    from {
        prefix-list SITE-LOCAL;
        route-type internal;
    }
    then {
        community add COGENT_LPREF70;
        accept;
    }
}</code></pre>
            <p>The above configuration changes the traffic flow to this:</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/1hJxj0OColxsHeP06JAfBb/8fa01b2f92b44de8b129b761dccc9acf/image3-47.png" />
            
            </figure><p>Which is exactly what we wanted!</p>
    <div>
      <h2>Conclusion</h2>
      <a href="#conclusion">
        
      </a>
    </div>
    <p>AS-path prepending is still useful, and has its use as part of the toolchain for operators to do traffic engineering, but should be used sparingly. <a href="https://ripe79.ripe.net/presentations/64-prepending_madory2.pdf">Excessive prepending opens a network up to wider spread route hijacks</a>, which should be avoided at all costs. As such, using community-based ingress traffic engineering is highly preferred (and recommended). In cases where communities aren’t available (or not available to steer customer traffic), prepends can be applied, but I encourage operators to actively monitor their effects, and roll them back if ineffective.</p><p>As a side-note, P Marcos et al. have published an interesting paper on AS-path prepending, and go into some trends seen in relation to prepending, I highly recommend giving it a read: <a href="https://www.caida.org/catalog/papers/2020_aspath_prepending/aspath_prepending.pdf">https://www.caida.org/catalog/papers/2020_aspath_prepending/aspath_prepending.pdf</a></p> ]]></content:encoded>
            <category><![CDATA[Routing]]></category>
            <category><![CDATA[BGP]]></category>
            <category><![CDATA[Network]]></category>
            <guid isPermaLink="false">7xqrew8H7IM1awzPwN3MOw</guid>
            <dc:creator>Tom Strickx</dc:creator>
        </item>
    </channel>
</rss>