
<rss version="2.0" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:media="http://search.yahoo.com/mrss/">
    <channel>
        <title><![CDATA[ The Cloudflare Blog ]]></title>
        <description><![CDATA[ Get the latest news on how products at Cloudflare are built, technologies used, and join the teams helping to build a better Internet. ]]></description>
        <link>https://blog.cloudflare.com</link>
        <atom:link href="https://blog.cloudflare.com/" rel="self" type="application/rss+xml"/>
        <language>en-us</language>
        <image>
            <url>https://blog.cloudflare.com/favicon.png</url>
            <title>The Cloudflare Blog</title>
            <link>https://blog.cloudflare.com</link>
        </image>
        <lastBuildDate>Tue, 07 Apr 2026 23:47:53 GMT</lastBuildDate>
        <item>
            <title><![CDATA[Enhanced security and simplified controls with automated botnet protection, cipher suite selection, and URL Scanner updates]]></title>
            <link>https://blog.cloudflare.com/enhanced-security-and-simplified-controls-with-automated-botnet-protection/</link>
            <pubDate>Mon, 17 Mar 2025 13:00:00 GMT</pubDate>
            <description><![CDATA[ Enhanced security, simplified control! This Security Week, Cloudflare unveils automated botnet protection, flexible cipher suites, and an upgraded URL Scanner. ]]></description>
            <content:encoded><![CDATA[ <p>At Cloudflare, we are constantly innovating and launching new features and capabilities across our product portfolio. Today, we're releasing a number of new features aimed at improving the security tools available to our customers.</p><p><b>Automated security level: </b>Cloudflare’s Security Level setting has been improved and no longer requires manual configuration. By integrating botnet data along with other request rate signals, all customers are protected from confirmed known malicious botnet traffic without any action required.</p><p><b>Cipher suite selection:</b> You now have greater control over encryption settings via the Cloudflare dashboard, including specific cipher suite selection based on our client or compliance requirements.</p><p><b>Improved URL scanner:</b> New features include bulk scanning, similarity search, location picker and more.</p><p>These updates are designed to give you more power and flexibility when managing online security, from proactive threat detection to granular control over encryption settings.</p>
    <div>
      <h3>Automating Security Level to provide stronger protection for all</h3>
      <a href="#automating-security-level-to-provide-stronger-protection-for-all">
        
      </a>
    </div>
    <p>Cloudflare’s <a href="https://developers.cloudflare.com/waf/tools/security-level/"><u>Security Level feature</u></a> was designed to protect customer websites from malicious activity.</p><p>Available to all Cloudflare customers, including the free tier, it has always had very simple logic: if a connecting client IP address has shown malicious behavior across our network, issue a <a href="https://developers.cloudflare.com/waf/reference/cloudflare-challenges/"><u>managed challenge</u></a>. The system tracks malicious behavior by assigning a threat score to each IP address. The more bad behavior is observed, the higher the score. Cloudflare customers could configure <a href="https://developers.cloudflare.com/waf/tools/security-level/"><u>the threshold that would trigger the challenge</u></a>.</p><p>We are now announcing an update to how Security Level works, by combining the IP address threat signal with threshold and botnet data. The resulting detection improvements have allowed us to automate the configuration, no longer requiring customers to set a threshold.</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/1RFWQl2Da9xu9MdfbJCRhy/8750770351d124ecf8d2f2b274f2e3cc/image1.png" />
          </figure><p>The Security Level setting is now <b>Always protected</b> in the dashboard, and ip_threat_score fields in WAF Custom Rules will no longer be populated. No change is required by Cloudflare customers. The <a href="https://developers.cloudflare.com/fundamentals/reference/under-attack-mode/"><u>“I am under attack”</u></a> option remains unchanged.</p>
    <div>
      <h3>Stronger protection, by default, for all customers</h3>
      <a href="#stronger-protection-by-default-for-all-customers">
        
      </a>
    </div>
    <p>Although we always favor simplicity, privacy-related services, including our own WARP, have seen growing use. Meanwhile, <a href="https://en.wikipedia.org/wiki/Carrier-grade_NAT"><u>carrier-grade network address translation (CGNATs)</u></a> and outbound forward proxies have been widely used for many years.</p><p>These services often result in multiple users sharing the same IP address, which can lead to legitimate users being challenged unfairly since individual addresses don’t strictly correlate with unique client behavior. Moreover, threat actors have become increasingly adept at anonymizing and dynamically changing their IP addresses using tools like VPNs, proxies, and botnets, further diminishing the reliability of IP addresses as a standalone indicator of malicious activity. Recognising these limitations, it was time for us to revisit Security Level’s logic to reduce the number of false positives being observed.</p><p>In February 2024, we introduced a new security system that automatically combines the real-time DDoS score with a traffic threshold and a botnet tracking system. The real-time DDoS score is part of our autonomous DDoS detection system, which analyzes traffic patterns to identify potential threats. This system superseded and replaced the existing Security Level logic, and is deployed on all customer traffic, including free plans. After thorough monitoring and analysis over the past year, we have confirmed that these behavior-based mitigation systems provide more accurate results. Notably, we've observed a significant reduction in false positives, demonstrating the limitations of the previous IP address-only logic.</p>
    <div>
      <h4>Better botnet tracking</h4>
      <a href="#better-botnet-tracking">
        
      </a>
    </div>
    <p>Our new logic combines IP address signals with behavioral and threshold indicators to improve the accuracy of botnet detection. While IP addresses alone can be unreliable due to potential false positives, we enhance their utility by integrating them with additional signals. We monitor surges in traffic from known "bad" IP addresses and further refine this data by examining specific properties such as path, accept, and host headers.</p><p>We also introduced a new botnet tracking system that continuously detects and tracks botnet activity across the Cloudflare network. From our unique vantage point as a <a href="https://w3techs.com/technologies/overview/proxy"><u>reverse proxy for nearly 20% of all websites</u></a>, we maintain a dynamic database of IP addresses associated with botnet activity. This database is continuously updated, enabling us to automatically respond to emerging threats without manual intervention. This effect is visible in the <a href="https://radar.cloudflare.com/security-and-attacks?dateStart=2024-02-01&amp;dateEnd=2024-03-31#mitigated-traffic-sources"><u>Cloudflare Radar chart</u></a> below, as we saw sharp growth in DDoS mitigations in February 2024 as the botnet tracking system was implemented.</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/3yOP8zoC5ZLVi4WHnXI0jH/ef3fd6ad10e8357b6b4f1bfb90e6d6b6/image4.png" />
          </figure>
    <div>
      <h4>What it means for our customers and their users</h4>
      <a href="#what-it-means-for-our-customers-and-their-users">
        
      </a>
    </div>
    <p>Customers now get better protection while having to manage fewer configurations, and they can rest assured that their online presence remains fully protected. These security measures are integrated and enabled by default across all of our plans, ensuring protection without the need for manual configuration or rule management.
This improvement is particularly beneficial for users accessing sites through proxy services or CGNATs, as these setups can sometimes trigger unnecessary security checks, potentially disrupting access to websites.</p>
    <div>
      <h4>What’s next</h4>
      <a href="#whats-next">
        
      </a>
    </div>
    <p>Our team is looking at defining the next generation of threat scoring mechanisms. This initiative aims to provide our customers with more relevant and effective controls and tools to combat today's and tomorrow's potential security threats.</p><p>Effective March 17, 2025, we are removing the option to configure manual rules using the threat score parameter in the Cloudflare dashboard. The "I'm Under Attack" mode remains available, allowing users to issue managed challenges to all traffic when needed.</p><p>By the end of Q1 2026, we anticipate disabling all rules that rely on IP threat score. This means that using the threat score parameter in the Rulesets API and via Terraform won’t be available after the end of the transition period. However, we encourage customers to be proactive and edit or remove the rules containing the threat score parameter starting today.</p>
    <div>
      <h3>Cipher suite selection now available in the UI</h3>
      <a href="#cipher-suite-selection-now-available-in-the-ui">
        
      </a>
    </div>
    
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/2e5Q0ghzpkuTQrR335fzIa/156b9531735fd9164768970fd08f5f85/image5.png" />
          </figure><p>Building upon our core security features, we're also giving you more control over your encryption: cipher suite selection is now available in the Cloudflare dashboard! </p><p>When a client initiates a visit to a Cloudflare-protected website, a <a href="https://www.cloudflare.com/learning/ssl/what-happens-in-a-tls-handshake/"><u>TLS handshake</u></a> occurs, where clients present a list of supported <a href="https://developers.cloudflare.com/ssl/edge-certificates/additional-options/cipher-suites/"><u>cipher suites</u></a> — cryptographic algorithms crucial for secure connections. While newer algorithms enhance security, balancing this with broad compatibility is key, as some customers prioritise reach by supporting older devices, even with less secure ciphers. To accommodate varied client needs, Cloudflare's default settings emphasise wide compatibility, allowing customers to tailor cipher suite selection based on their priorities: strong security, compliance (PCI DSS, FIPS 140-2), or legacy device support.</p><p>Previously, customizing cipher suites required multiple API calls, proving cumbersome for many users. Now, Cloudflare introduces Cipher Suite Selection to the dashboard. This feature introduces user-friendly selection flows like security recommendations, compliance presets, and custom selections.  </p>
    <div>
      <h4>Understanding cipher suites</h4>
      <a href="#understanding-cipher-suites">
        
      </a>
    </div>
    <p>Cipher suites are collections of cryptographic algorithms used for key exchange, authentication, encryption, and message integrity, essential for a TLS handshake. During the handshake’s initiation, the client sends a "client hello" message containing a list of supported cipher suites. The server responds with a "server hello" message, choosing a cipher suite from the client's list based on security and compatibility. This chosen cipher suite forms the basis of TLS termination and plays a crucial role in establishing a secure HTTPS connection. Here’s a quick overview of each component:</p><ul><li><p><b>Key exchange algorithm:</b> Secures the exchange of encryption keys between parties.</p></li><li><p><b>Authentication algorithm:</b> Verifies the identities of the communicating parties.</p></li><li><p><b>Encryption algorithm:</b> Ensures the confidentiality of the data.</p></li><li><p><b>Message integrity algorithm:</b> Confirms that the data remains unaltered during transmission.</p></li></ul><p><a href="https://www.geeksforgeeks.org/perfect-forward-secrecy/"><b><u>Perfect forward secrecy</u></b></a><b> </b>is an important feature of modern cipher suites. It ensures that each session's encryption keys are generated independently, which means that even if a server’s private key is compromised in the future, past communications remain secure.</p>
    <div>
      <h4>What we are offering </h4>
      <a href="#what-we-are-offering">
        
      </a>
    </div>
    <p>You can find cipher suite configuration under Edge Certificates in your zone’s SSL/TLS dashboard. There, you will be able to view your allow-listed set of cipher suites. </p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/6fT7BvPow3zvKTl1JYw7yX/8dcd8b797f671b05211defaaf4c4bb83/image5.png" />
          </figure><p>Additionally, you will be able to choose from three different user flows, depending on your specific use case, to seamlessly select your appropriate list. Those three user flows are: security recommendation selection, compliance selection, or custom selection. The goal of the user flows is to outfit customers with cipher suites that match their goals and priorities, whether those are maximum compatibility or best possible security.</p><p>1. Security recommendations </p><p>To streamline the process, we have turned our <a href="https://developers.cloudflare.com/ssl/reference/cipher-suites/recommendations/"><u>cipher suites recommendations</u></a> into selectable options. This is in an effort to expose our customers to cipher suites in a tangible way and enable them to choose between different security configurations and compatibility. Here is what they mean:</p><ul><li><p><b>Modern:</b> Provides the highest level of security and performance with support for Perfect Forward Secrecy and <a href="https://www.ietf.org/archive/id/draft-irtf-cfrg-aead-properties-03.html"><u>Authenticated Encryption (AEAD)</u></a>. Ideal for customers who prioritize top-notch security and performance, such as financial institutions, healthcare providers, or government agencies. This selection requires TLS 1.3 to be enabled and the minimum TLS version set to 1.2.</p></li><li><p><b>Compatible:</b> Balances security and compatibility by offering forward-secret cipher suites that are broadly compatible with older systems. Suitable for most customers who need a good balance between security and reach. This selection also requires TLS 1.3 to be enabled and the minimum TLS version set to 1.2.</p></li><li><p><b>Legacy:</b> Optimizes for the widest reach, supporting a wide range of legacy devices and systems. Best for customers who do not handle sensitive data and need to accommodate a variety of visitors. This option is ideal for blogs or organizations that rely on older systems.</p></li></ul><p>2. Compliance selection</p><p>Additionally, we have also turned our <a href="https://developers.cloudflare.com/ssl/reference/cipher-suites/compliance-status/"><u>compliance recommendations</u></a> into selectable options to make it easier for our customers to meet their PCI DSS or FIPS-140-2 requirements.</p><ul><li><p><a href="https://www.pcisecuritystandards.org/standards/pci-dss/"><b><u>PCI DSS Compliance:</u></b></a> Ensures that your cipher suite selection aligns with PCI DSS standards for protecting cardholder data. This option will enforce a requirement to set a minimum TLS version of 1.2, and TLS 1.3 to be enabled, to maintain compliance.</p><ul><li><p>Since the list of supported cipher suites require TLS 1.3 to be enabled and a minimum TLS version of 1.2 in order to be compliant, we will disable compliance selection until the zone settings are updated to meet those requirements. This effort is to ensure that our customers are truly compliant and have the proper zone settings to be so. </p></li></ul></li><li><p><a href="https://csrc.nist.gov/pubs/fips/140-2/upd2/final"><b><u>FIPS 140-2 Compliance</u></b><u>:</u></a> Tailored for customers needing to meet federal security standards for cryptographic modules. Ensures that your encryption practices comply with FIPS 140-2 requirements.</p></li></ul><p>3. Custom selection </p><p>For customers needing precise control, the custom selection flow allows individual cipher suite selection, excluding TLS 1.3 suites which are automatically enabled with TLS 1.3. To prevent disruptions, guardrails ensure compatibility by validating that the minimum TLS version aligns with the selected cipher suites and that the <a href="https://www.cloudflare.com/application-services/products/ssl/">SSL/TLS certificate</a> is compatible (e.g., RSA certificates require RSA cipher suites).</p>
    <div>
      <h3>API </h3>
      <a href="#api">
        
      </a>
    </div>
    <p>The <a href="https://developers.cloudflare.com/ssl/edge-certificates/additional-options/cipher-suites/"><u>API</u></a> will still be available to our customers. This aims to support an existing framework, especially to customers who are already API reliant. Additionally, Cloudflare preserves the specified cipher suites in the order they are set via the API and that control of ordering will remain unique to our API offering. </p><p>With your Advanced Certificate Manager or Cloudflare for SaaS subscription, head to Edge Certificates in your zone’s SSL dashboard and give it a try today!</p>
    <div>
      <h3>Smarter scanning, safer Internet with the new version of URL Scanner</h3>
      <a href="#smarter-scanning-safer-internet-with-the-new-version-of-url-scanner">
        
      </a>
    </div>
    
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/5eFwJMzk3JuwYNKcSk4kiH/63e4a8713be583d83df737cf6f59281d/image10.png" />
          </figure><p>Cloudflare's URL Scanner is a tool designed to detect and analyze potential security threats like phishing and malware by scanning and evaluating websites, providing detailed insights into their safety and technology usage. We've leveraged our own <a href="https://developers.cloudflare.com/radar/investigate/url-scanner/"><u>URL Scanner</u></a> to enhance our internal <u>Trust &amp; Safety efforts</u>, automating the detection and mitigation of some forms of abuse on our platform. This has not only strengthened our own security posture, but has also directly influenced the development of the new features we're announcing today. </p><p>Phishing attacks are on the rise across the Internet, and we saw a major opportunity to be "customer zero" for our URL Scanner to address abuse on our own network. By working closely with our Trust &amp; Safety team to understand how the URL Scanner could better identify potential phishing attempts, we've improved the speed and accuracy of our response to abuse reports, making the Internet safer for everyone. Today, we're excited to share the new API version and the latest updates to URL Scanner, which include the ability to scan from specific geographic locations, bulk scanning, search by Indicators of Compromise (IOCs), improved UI and information display, comprehensive IOC listings, advanced sorting options, and more. These features are the result of our own experiences in leveraging URL Scanner to safeguard our platform and our customers, and we're confident that they will prove useful to our security analysts and threat intelligence users.</p>
    <div>
      <h4>Scan up to 100 URLs at once by using bulk submissions</h4>
      <a href="#scan-up-to-100-urls-at-once-by-using-bulk-submissions">
        
      </a>
    </div>
    <p>Cloudflare Enterprise customers can now conduct routine scans of their web assets to identify emerging vulnerabilities, ensuring that potential threats are addressed proactively, by using the <a href="https://developers.cloudflare.com/api/resources/url_scanner/subresources/scans/methods/bulk_create/"><u>Bulk Scanning API endpoint</u></a>. Another use case for the bulk scanning functionality is developers leveraging bulk scanning to verify that all URLs your team is accessing are secure and free from potential exploits before launching new websites or updates.</p><p>Scanning of multiple URLs addresses the specific needs of our users engaged in threat hunting. Many of them maintain extensive lists of URLs that require swift investigation to identify potential threats. Currently, they face the task of submitting these URLs one by one, which not only slows down their workflow but also increases the manual effort involved in their security processes. With the introduction of bulk submission capabilities, users can now submit up to 100 URLs at a time for scanning. </p>
    <div>
      <h4>How we built the bulk scanning feature</h4>
      <a href="#how-we-built-the-bulk-scanning-feature">
        
      </a>
    </div>
    <p>Let’s look at a regular workflow:</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/6l8aN6xhN4HEfw4ZMi1MT8/5eb62472b42f75487c55b17b3415b584/image6.png" />
          </figure><p>In this workflow, when the user submits a new scan, we create a <a href="https://developers.cloudflare.com/durable-objects/"><u>Durable Object</u></a> with the same ID as the scan, save the scan options, like the URL to scan, to the <a href="https://developers.cloudflare.com/durable-objects/api/storage-api/"><u>Durable Objects’s storage</u></a> and schedule an <a href="https://developers.cloudflare.com/durable-objects/api/storage-api/#setalarm"><u>alarm</u></a> for a few seconds later. This allows us to respond immediately to the user, signalling a successful submission. A few seconds later the alarm triggers, and we start the scan itself. </p><p>However, with bulk scanning, the process is slightly different:</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/2kXLJ5sSGBbM06H3Ftsrqi/a4440fd0efc7c0271580c6da6f08f814/image9.png" />
          </figure><p>In this case, there are no Durable Objects involved just yet; the system simply sends each URL in the bulk scan submission as a new message to the queue.</p><p>Notice that in both of these cases the scan is triggered asynchronously. In the first case, it starts when the Durable Objects alarm fires and, in the second case, when messages in the queue are consumed. While the durable object alarm will always fire in a few seconds, messages in the queue have no predetermined processing time, they may be processed seconds to minutes later, depending on how many messages are already in the queue and how fast the system processes them.</p><p>When users bulk scan, having the scan done at <i>some </i>point in time is more important than having it done <i>now</i>. When using the regular scan workflow, users are limited in the number of scans per minute they can submit. However, when using bulk scan this is not a concern, and users can simply send all URLs they want to process in a single HTTP request. This comes with the tradeoff that scans may take longer to complete, which is a perfect fit for <a href="https://developers.cloudflare.com/queues/"><u>Cloudflare Queues</u></a>. Having the ability to <a href="https://developers.cloudflare.com/queues/configuration/configure-queues/#consumer-worker-configuration"><u>configure</u></a> retries, max batch size, max batch timeouts, and max concurrency is something we’ve found very useful. As the scans are completed asynchronously, users can request the resulting scan reports <a href="https://developers.cloudflare.com/api/resources/url_scanner/subresources/scans/methods/get/"><u>via the API</u></a>.</p>
    <div>
      <h4>Discover related scans and better IOC search</h4>
      <a href="#discover-related-scans-and-better-ioc-search">
        
      </a>
    </div>
    <p>The <i>Related Scans</i> feature allows <a href="https://developers.cloudflare.com/api/resources/url_scanner/subresources/scans/methods/list/"><u>API</u></a>, <a href="http://dash.cloudflare.com"><u>Cloudflare dashboard</u></a> and <a href="http://radar.cloudflare.com"><u>Radar</u></a> users alike to view related scans directly within the URL Scanner Report. This helps users analyze and understand the context of a scanned URL by providing insights into similar URLs based on various attributes. Filter and search through URL Scanner reports to retrieve information on related scans, including those with identical favicons, similar HTML structures, and matching IP addresses.</p><p>The <i>Related Scans</i> tab presents a table with key headers corresponding to four distinct filters. Each entry includes the scanned URL and a direct link to view the detailed scan report, allowing for quick access to further information. </p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/6yRzKVd0M9sNF1uGOWA1vb/212008b5296ad6df23088571f0602930/image3.png" />
          </figure><p>We've introduced the ability to search by indicators of compromise (IOCs), such as IP addresses and hashes, directly within the user interface. Additionally, we've added advanced filtering options by various criteria, including screenshots, hashes, favicons, and HTML body content. This allows for more efficient organization and prioritization of URLs based on specific needs. While attackers often make minor modifications to the HTML structure of phishing pages to evade detection, our advanced filtering options enable users to search for URLs with similar HTML content. This means that even if the visual appearance of a phishing page changes slightly, we can still identify connections to known phishing campaigns by comparing the underlying HTML structure. This proactive approach helps users identify and block these threats effectively.</p><p>Another use case for the advanced filtering options is the search by hash; a user who has identified a malicious JavaScript file through a previous investigation can now search using the file's hash. By clicking on an HTTP transaction, you'll find a direct link to the relevant hash, immediately allowing you to pivot your investigation. The real benefit comes from identifying other potentially malicious sites that have that same hash. This means that if you know a given script is bad, you can quickly uncover other compromised websites delivering the same malware.</p>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/3rWKgTrGLW297cVFbH9hSY/4555697b668d90f3df4d740bd91d3116/image7.png" />
          </figure><p>The user interface has also undergone significant improvements to enhance the overall experience. Other key updates include:</p><ul><li><p>Page title and favicon surfaced, providing immediate visual context</p></li><li><p>Detailed summaries are now available</p></li><li><p>Redirect chains allow users to understand the navigation path of a URL</p></li><li><p>The ability to scan files from URLs that trigger an automatic file download</p></li></ul>
          <figure>
          <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/5O55W8CLMrlPANpzkPAUY0/35748cb200feb79de6251c79d2be87f9/image2.png" />
          </figure>
    <div>
      <h4>Download HAR files</h4>
      <a href="#download-har-files">
        
      </a>
    </div>
    <p>With the latest updates to our URL Scanner, users can now download both the <a href="https://en.wikipedia.org/wiki/HAR_(file_format)"><u>HAR (HTTP Archive) file</u></a> and the JSON report from their scans. The <a href="https://blog.cloudflare.com/introducing-har-sanitizer-secure-har-sharing/"><u>HAR file</u></a> provides a detailed record of all interactions between the web browser and the scanned website, capturing crucial data such as request and response headers, timings, and status codes. This format is widely recognized in the industry and can be easily analyzed using various tools, making it invaluable for developers and security analysts alike.</p><p>For instance, a threat intelligence analyst investigating a suspicious URL can download the HAR file to examine the network requests made during the scan. By analyzing this data, they can identify potential malicious behavior, such as unexpected redirects and correlate these findings with other threat intelligence sources. Meanwhile, the JSON report offers a structured overview of the scan results, including security verdicts and associated IOCs, which can be integrated into broader security workflows or automated systems.</p>
    <div>
      <h4>New API version</h4>
      <a href="#new-api-version">
        
      </a>
    </div>
    <p>Finally, we’re announcing a <a href="https://developers.cloudflare.com/api/operations/urlscanner-create-scan-v2"><u>new version of our API</u></a>, allowing users to transition effortlessly to our service without needing to overhaul their existing workflows. Moving forward, any future features will be integrated into this updated API version, ensuring that users have access to the latest advancements in our URL scanning technology.</p><p>We understand that many organizations rely on automation and integrations with our previous API version. Therefore, we want to reassure our customers that there will be no immediate deprecation of the old API. Users can continue to use the existing API without disruption, giving them the flexibility to migrate at their own pace. We invite you to try the <a href="https://developers.cloudflare.com/api/operations/urlscanner-create-scan-v2"><u>new API</u></a> today and explore these new features to help with your web security efforts.</p>
    <div>
      <h3>Never miss an update</h3>
      <a href="#never-miss-an-update">
        
      </a>
    </div>
    <p>In summary, these updates to Security Level, cipher suite selection, and URL Scanner help us provide comprehensive, accessible, and proactive security solutions. Whether you're looking for automated protection, granular control over your encryption, or advanced threat detection capabilities, these new features are designed to empower you to build a safer and more secure online presence. We encourage you to explore these features in your Cloudflare dashboard and discover how they can benefit your specific needs.</p><p><i>We’ll continue to share roundup blog posts as we build and innovate. Follow along on the </i><a href="https://blog.cloudflare.com/"><i>Cloudflare Blog</i></a><i> for the latest news and updates. </i></p> ]]></content:encoded>
            <category><![CDATA[Security Week]]></category>
            <category><![CDATA[URL Scanner]]></category>
            <category><![CDATA[Threat Intelligence]]></category>
            <category><![CDATA[Security]]></category>
            <guid isPermaLink="false">5E0Ceo6CEHszKOpdxV3sl0</guid>
            <dc:creator>Alexandra Moraru</dc:creator>
            <dc:creator>Mia Malden</dc:creator>
            <dc:creator>Yomna Shousha</dc:creator>
            <dc:creator>Sofia Cardita</dc:creator>
        </item>
        <item>
            <title><![CDATA[Cloudflare’s URL Scanner, new features, and the story of how we built it]]></title>
            <link>https://blog.cloudflare.com/building-urlscanner/</link>
            <pubDate>Fri, 08 Mar 2024 14:00:09 GMT</pubDate>
            <description><![CDATA[ Discover the enhanced URL Scanner API: Now with direct access from the Security Center Investigate Portal, enjoy unlisted scans, multi-device screenshots, and seamless integration within the Cloudflare ecosystem ]]></description>
            <content:encoded><![CDATA[ <p></p><p>Today, we’re excited to talk about <a href="https://radar.cloudflare.com/scan">URL Scanner</a>, a tool that helps everyone from security teams to everyday users to detect and safeguard against malicious websites by scanning and analyzing them. URL Scanner has executed almost a million scans since its <a href="/radar-url-scanner-early-access/">launch</a> last March on <a href="https://radar.cloudflare.com/scan">Cloudflare Radar</a>, driving us to continuously innovate and enhance its capabilities. Since that time, we have introduced unlisted scans, detailed malicious verdicts, enriched search functionality, and now, integration with Security Center and an official API, all built upon the robust foundation of <a href="https://developers.cloudflare.com/workers/">Cloudflare Workers</a>, <a href="https://developers.cloudflare.com/workers/runtime-apis/durable-objects/">Durable Objects</a>, and the <a href="/browser-rendering-open-beta/">Browser Rendering API</a>.</p>
    <div>
      <h2>Integration with the Security Center in the Cloudflare Dashboard</h2>
      <a href="#integration-with-the-security-center-in-the-cloudflare-dashboard">
        
      </a>
    </div>
    <p>Security Center is the single place in the Cloudflare Dashboard to map your <a href="https://www.cloudflare.com/learning/security/what-is-an-attack-surface/">attack surface</a>, identify potential security risks, and mitigate risks with a few clicks. Its users can now access the URL scanner directly from the <a href="https://developers.cloudflare.com/security-center/investigate/">Investigate Portal</a>, enhancing their cybersecurity workflow. These scans will be unlisted by default, ensuring privacy while facilitating a deep dive into <a href="https://www.cloudflare.com/learning/security/glossary/website-security-checklist/">website security.</a> Users will be able to see their historic scans and access the related reports when they need to, and they will benefit from automatic screenshots for multiple screen sizes, enriching the context of each scan.</p><p>Customers with Cloudflare dashboard access will enjoy higher API limits and faster response times, crucial for agile security operations. Integration with internal workflows becomes seamless, allowing for sophisticated network and user protection strategies.</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/5xsq5tQ1cjM8cQwnCRUCmh/f6f3a929b008c706765db6a22e5abde7/image2-24.png" />
            
            </figure><p><i>Security Center in the Cloudflare Dashboard</i></p>
    <div>
      <h2>Unlocking the potential of the URL Scanner API</h2>
      <a href="#unlocking-the-potential-of-the-url-scanner-api">
        
      </a>
    </div>
    <p>The <a href="https://developers.cloudflare.com/radar/investigate/url-scanner/">URL Scanner API</a> is a powerful asset for developers, enabling custom scans to detect <a href="https://www.cloudflare.com/learning/access-management/phishing-attack/">phishing</a> or <a href="https://www.cloudflare.com/learning/ddos/glossary/malware/">malware</a> risks, analyze website technologies, and much more. With new features like custom HTTP headers and multi-device screenshots, developers gain a comprehensive toolkit for thorough website assessment.</p>
    <div>
      <h3>Submitting a scan request</h3>
      <a href="#submitting-a-scan-request">
        
      </a>
    </div>
    <p>Using the API, here’s the simplest way to <a href="https://developers.cloudflare.com/api/operations/urlscanner-create-scan">submit</a> a scan request:</p>
            <pre><code>curl --request POST \
	--url https://api.cloudflare.com/client/v4/accounts/&lt;accountId&gt;/urlscanner/scan \
	--header 'Content-Type: application/json' \
--header "Authorization: Bearer &lt;API_TOKEN&gt;" \
	--data '{
		"url": "https://www.cloudflare.com",
	}'</code></pre>
            <p>New features include the option to set custom HTTP headers, like <a href="https://developer.mozilla.org/en-US/docs/Glossary/User_agent">User-Agent</a> and <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Authorization">Authorization</a>, request multiple target device screenshots, like mobile and desktop, as well as set the visibility level to “unlisted”. This essentially marks the scan as private and was often requested by developers who wanted to keep their investigations confidential. Public scans, on the other hand, can be found by anyone through search and are useful to share results with the wider community. You can find more details in our <a href="https://developers.cloudflare.com/radar/investigate/url-scanner/">developer documentation</a>.</p>
    <div>
      <h3>Exploring the scan results</h3>
      <a href="#exploring-the-scan-results">
        
      </a>
    </div>
    
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/2NRMloVOilGBXsYy12xeeT/a43d66e1b6cd00bd5fdf2bd18ede1256/image5-16.png" />
            
            </figure><p><i>Scan results for</i> <a href="http://www.cloudflare.com"><i>www.cloudflare.com</i></a> <i>on Cloudflare Radar</i></p><p>Once a scan concludes, fetch the final <a href="https://developers.cloudflare.com/api/operations/urlscanner-get-scan">report</a> and the full <a href="https://developers.cloudflare.com/api/operations/urlscanner-get-scan-har">network log</a>. Recently added features include the `verdict` property, indicating the site’s malicious status, and the `securityViolations` section detailing <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP">CSP</a> or <a href="https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity">SRI</a> policy breaches — as a developer, you can also scan your own website and see our recommendations. Expect improvements on verdict accuracy over time, as this is an area we’re focusing on.</p>
    <div>
      <h3>Enhanced search functionality</h3>
      <a href="#enhanced-search-functionality">
        
      </a>
    </div>
    <p>Developers can now <a href="https://developers.cloudflare.com/api/operations/urlscanner-search-scans">search</a> scans by hostname, a specific URL or even <i>any</i> URL the page connected to during the scan. This allows, for example, to search for websites that use a JavaScript library named jquery.min.js (‘?path=jquery.min.js’). Future plans include additional features like searching by IP address, <a href="https://www.cloudflare.com/learning/network-layer/what-is-an-autonomous-system/">ASN</a>, and malicious website categorisation.</p><p>The URL Scanner can be used for a diverse range of applications. These include capturing a website's evolving state over time (such as tracking changes to the front page of an online newspaper), analyzing technologies employed by a website, preemptively assessing potential risks (as when scrutinizing shortened URLs), and supporting the investigation of persistent cybersecurity threats (such as identifying affected websites hosting a malicious JavaScript file).</p>
    <div>
      <h2>How we built the URL Scanner API</h2>
      <a href="#how-we-built-the-url-scanner-api">
        
      </a>
    </div>
    <p>In recounting the process of developing the URL Scanner, we aim to showcase the potential and versatility of Cloudflare Workers as a platform. This story is more than a technical journey, but a testament to the capabilities inherent in our platform's suite of APIs. By dogfooding our own technology, we not only demonstrate confidence in its robustness but also encourage developers to harness the same capabilities for building sophisticated applications. The URL Scanner exemplifies how Cloudflare Workers, Durable Objects, and the Browser Rendering API seamlessly integrate.</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/55HCrSeFuu3FUjcjjIXJyl/53c0ed5a74a3ca5052972fa191bd679b/image4-23.png" />
            
            </figure><p><i>High level overview of the Cloudflare URL Scanner technology stack</i></p><p>As seen above, Cloudflare’s runtime infrastructure is the foundation the system runs on. <a href="https://developers.cloudflare.com/workers/">Cloudflare Workers</a> serves the public API, <a href="https://developers.cloudflare.com/workers/runtime-apis/durable-objects/">Durable Objects</a> handles orchestration, <a href="https://developers.cloudflare.com/r2/">R2</a> acts as the primary storage solution, and <a href="https://developers.cloudflare.com/queues/">Queues</a> efficiently handles batch operations, all at the edge. However, what truly enables the URL Scanner’s capabilities is the <a href="https://developers.cloudflare.com/browser-rendering/">Browser Rendering API</a>. It’s what initially allowed us to release in such a short time frame, since we didn’t have to build and manage an entire fleet of Chrome browsers from scratch. We simply request a browser, and then using the well known <a href="https://pptr.dev/">Puppeteer</a> library, instruct it to fetch the webpage and process it in the way we want. This API is at the heart of the entire system.</p>
    <div>
      <h3>Scanning a website</h3>
      <a href="#scanning-a-website">
        
      </a>
    </div>
    <p>The entire process of scanning a website, can be split into 4 phases:</p><ol><li><p>Queue a scan</p></li><li><p>Browse to the website and compile initial report</p></li><li><p>Post-process: compile additional information and build final report</p></li><li><p>Store final report, ready for serving and searching</p></li></ol>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/5AUAfO5VBEUvbRlYwoJ5zL/4da6566b0dff56dca49e14fc500cc427/image1-28.png" />
            
            </figure><p>In short, we create a Durable Object, the Scanner, unique to each scan, which is responsible for orchestrating the scan from start to finish. Since we want to respond immediately to the user, we save the scan to the Durable Object’s transactional Key-Value storage, and schedule an alarm so we can perform the scan asynchronously a second later.  We then respond to the user, informing them that the scan request was accepted.</p><p>When the Scanner’s alarm triggers, we enter the second phase:</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/4qnzMmDQbPtWmyIClv6680/4ff2ee5c2a36b78a9d5e2d7f6d3a134a/image7-5.png" />
            
            </figure><p>There are 3 components at work in this phase, the Scanner, the Browser Pool and the Browser Controller, all <a href="https://developers.cloudflare.com/workers/runtime-apis/durable-objects/">Durable Objects</a>.</p><p>In the initial release, for each new scan we would launch a brand-new browser. However, This operation would take time and was inefficient, so after review, we decided to reuse browsers across multiple scans. This is why we introduced both the Browser Pool and the Browser Controller components. The Browser Pool keeps track of what browsers we have open, when they last pinged the browser pool (so it knows they’re alive), and whether they’re free to accept a new scan. The Browser Controller is responsible for keeping the browser instance alive, once it’s launched, and orchestrating (ahem, <a href="http://pptr.dev">puppeteering</a>) the entire browsing session. Here’s a simplified version of our Browser Controller code:</p>
            <pre><code>export class BrowserController implements DurableObject {
	//[..]
	private async handleNewScan(url: string) {
		if (!this.browser) {
			// Launch browser: 1st request to durable object
			this.browser = await puppeteer.launch(this.env.BROWSER)
			await this.state.storage.setAlarm(Date.now() + 5 * 1000)
		}
		// Open new page and navigate to url
		const page = await this.browser.newPage()
		await page.goto(url, { waitUntil: 'networkidle2', timeout: 5000, })

		// Capture DOM
		const dom = await page.content()

		// Clean up
		await page.close()

		return {
			dom: dom,
		}
	}

	async alarm() {
		if (!this.browser) {
			return
		}
		await this.browser.version() // stop websocket connection to Chrome from going idle
		
		// ping browser pool, let it know we're alive
		
		// Keep durable object alive
		await this.state.storage.setAlarm(Date.now() + 5 * 1000)
	}
}</code></pre>
            <p>Launching a browser (Step 6) and maintaining a connection to it is abstracted away from us thanks to the <a href="/browser-rendering-open-beta/">Browser Rendering API</a>. This API is responsible for all the infrastructure required to maintain a fleet of Chrome browsers, and led to a much quicker development and release of the URL Scanner. It also allowed us to use a well-known library, <a href="https://pptr.dev/">Puppeteer</a>, to communicate with Google Chrome via the <a href="https://chromedevtools.github.io/devtools-protocol/">DevTools</a> protocol.</p><p>The initial report is made up of the network log of all requests, captured in <a href="https://en.wikipedia.org/wiki/HAR_(file_format)">HAR</a> (HTTP Archive) format. HAR files, essentially JSON files, provide a detailed record of all interactions between a web browser and a website. As an established standard in the industry, HAR files can be easily <a href="https://developers.cloudflare.com/api/operations/urlscanner-get-scan-har">shared</a> and analyzed using specialized <a href="https://toolbox.googleapps.com/apps/har_analyzer/">tools</a>. In addition to this network log, we augment our dataset with an array of other metadata, including base64-encoded screenshots which provide a snapshot of the website at the moment of the scan.</p><p>Having this data, we transition to phase 3, where the Scanner Durable Object initiates a series of interactions with a few other Cloudflare APIs in order to collect additional information, like running a phishing scanner over the web page's Document Object Model (DOM), fetching <a href="https://www.cloudflare.com/learning/dns/dns-records/">DNS records</a>, and extracting information about <a href="https://developers.cloudflare.com/api/operations/domain-intelligence-get-domain-details">categories</a> and <a href="https://developers.cloudflare.com/api/operations/radar-get-ranking-domain-details">Radar rank</a> associated with the main hostname.</p><p>This process ensures that the final report is enriched with insights coming from different sources, making the URL Scanner more efficient in assessing websites. Once all the necessary information is collected, we compile the final report and store it as a JSON file within <a href="https://developers.cloudflare.com/r2">R2</a>, Cloudflare’s <a href="https://www.cloudflare.com/developer-platform/products/r2/">object storage solution</a>. To empower users with efficient scan searches, we use Postgres.</p><p>While the initial approach involved sending each completed scan promptly to the core API for immediate storage in Postgres, we realized that, as the rate of scans grew, a more efficient strategy would be to batch those operations, and for that, we use Worker Queues:</p>
            <figure>
            
            <img src="https://cf-assets.www.cloudflare.com/zkvhlag99gkb/1gyX0lHVmgVGhmYZeyipcM/93933f31d4dd0905fef9d7cc234a1528/image6-8.png" />
            
            </figure><p>This allows us to better manage the write load on Postgres. We wanted scans available as soon as possible to those who requested them, but it’s ok if they’re only available in search results at a slightly later point in time (seconds to minutes, depending on load).</p><p>In short, <a href="https://developers.cloudflare.com/workers/runtime-apis/durable-objects/">Durable Objects</a> together with the <a href="/browser-rendering-open-beta/">Browser Rendering API</a> power the entire scanning process. Once that’s finished, the Cloudflare Worker serving the API will simply fetch it from <a href="https://developers.cloudflare.com/r2/">R2</a> by ID. All together, Workers, Durable Objects, and R2 scale seamlessly and will allow us to grow as demand evolves.</p>
    <div>
      <h3>Last but not least</h3>
      <a href="#last-but-not-least">
        
      </a>
    </div>
    <p>While we've extensively covered the URL scanning workflow, we've yet to delve into the construction of the API worker itself. Developed with <a href="https://www.typescriptlang.org/">Typescript</a>, it uses <a href="https://github.com/cloudflare/itty-router-openapi">itty-router-openapi</a>, a Javascript router with <a href="https://spec.openapis.org/oas/v3.1.0">Open API 3</a> schema generation and validation, originally built for <a href="https://radar.cloudflare.com/">Radar</a>, but that’s been improving ever since with contributions from the community. Here’s a quick example of how to set up an endpoint, with input validation built in:</p>
            <pre><code>import { DateOnly, OpenAPIRoute, Path, Str, OpenAPIRouter } from '@cloudflare/itty-router-openapi'

import { z } from 'zod'
import { OpenAPIRoute, OpenAPIRouter, Uuid } from '@cloudflare/itty-router-openapi'

export class ScanMetadataCreate extends OpenAPIRoute {
  static schema = {
    tags: ['Scans'],
    summary: 'Create Scan metadata',
    requestBody: {
      scan_id: Uuid,
      url: z.string().url(),
      destination_ip: z.string().ip(),
      timestamp: z.string().datetime(),
      console_logs: [z.string()],
    },
  }

  async handle(
    request: Request,
    env: any,
    context: any,
    data: any,
  ) {
    // Retrieve validated scan
    const newScanMetadata = data.body

    // Insert the scan

    // Return scan as json
    return newScanMetadata
  }
}


const router = OpenAPIRouter()
router.post('/scan/metadata/', ScanMetadataCreate)

// 404 for everything else
router.all('*', () =&gt; new Response('Not Found.', { status: 404 }))

export default {
  fetch: router.handle,
}</code></pre>
            <p>In the example above, the ScanMetadataCreate endpoint will make sure to validate the incoming POST data to match the defined schema before calling the ‘async handle(request,env,context,data)’ function. This way you can be sure that if your code is called, the data argument will always be validated and formatted.</p><p>You can learn more about the project on its <a href="https://github.com/cloudflare/itty-router-openapi">GitHub page</a>.</p>
    <div>
      <h2>Future plans and new features</h2>
      <a href="#future-plans-and-new-features">
        
      </a>
    </div>
    <p>Looking ahead, we're committed to further elevating the URL Scanner's capabilities. Key upcoming features include geographic scans, where users can customize the location that the scan is done from, providing critical insights into regional security threats and content compliance; expanded scan details, including more comprehensive headers and security details; and continuous performance improvements and optimisations, so we can deliver faster scan results.</p><p>The evolution of the URL Scanner is a reflection of our commitment to Internet safety and innovation. Whether you're a developer, a security professional, or simply invested in the safety of the digital landscape, the URL Scanner API offers a comprehensive suite of tools to enhance your efforts. Explore the new features today, and join us in shaping a safer Internet for everyone.</p><p>Remember, while Security Center's new capabilities offer advanced tools for URL Scanning for Cloudflare’s existing customers, the URL Scanner remains accessible for basic scans to the public on <a href="https://radar.cloudflare.com/scan">Cloudflare Radar</a>, ensuring our technology benefits a broad audience.</p><p>If you’re considering a new career direction, check out <a href="https://cloudflare.com/careers">our open positions</a>. We’re looking for individuals who want to help make the Internet better; learn more about our mission <a href="https://www.cloudflare.com/learning/what-is-cloudflare/">here</a>.</p> ]]></content:encoded>
            <category><![CDATA[Security Week]]></category>
            <category><![CDATA[URL Scanner]]></category>
            <category><![CDATA[Security]]></category>
            <category><![CDATA[API]]></category>
            <guid isPermaLink="false">1JMMDhLvjentdgwo5df7bC</guid>
            <dc:creator>Sofia Cardita</dc:creator>
            <dc:creator>Alexandra Moraru</dc:creator>
        </item>
    </channel>
</rss>