MORE POSTS
August 29, 2025 2:00 PM
The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals
By mid-2025, training drives nearly 80% of AI crawling, while referrals to publishers (especially from Google) are falling and crawl-to-refer ratios show AI consumes far more than it sends back....
August 28, 2025 2:05 PM
A deeper look at AI crawlers: breaking down traffic by purpose and industry
We are extending AI-related insights on Cloudflare Radar with new industry-focused data and a breakdown of bot traffic by purpose, such as training or user action. ...
August 28, 2025 2:00 PM
The age of agents: cryptographically recognizing agent traffic
Cloudflare now lets websites and bot creators use Web Bot Auth to segment agents from verified bots, making it easier for customers to allow or disallow the many types of user and partner directed....
August 28, 2025 2:00 PM
The next step for content creators in working with AI bots: Introducing AI Crawl Control
Cloudflare launches AI Crawl Control (formerly AI Audit) and introduces easily customizable 402 HTTP responses....
August 18, 2025 2:00 PM
Announcing the Cloudflare Browser Developer Program
Announcing the Browser Developer Program: Cloudflare’s new collaborative program to help shape Cloudflare challenges that work seamlessly with your browser. Join us today!...
August 04, 2025 1:00 PM
Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives
Perplexity is repeatedly modifying their user agent and changing IPs and ASNs to hide their crawling activity, in direct conflict with explicit no-crawl preferences expressed by websites....
July 01, 2025 10:00 AM
The crawl before the fall… of referrals: understanding AI’s impact on content providers
Cloudflare Radar now shows how often a given AI model sends traffic to a site relative to how often it crawls that site. This helps site owners make decisions about which AI bots to allow or block.
...
July 01, 2025 10:00 AM
Control content use for AI training with Cloudflare’s managed robots.txt and blocking for monetized content
Cloudflare is making it easier for publishers and content creators of all sizes to prevent their content from being scraped for AI training by managing robots.txt on their behalf. ...
July 01, 2025 10:00 AM
Introducing pay per crawl: Enabling content owners to charge AI crawlers for access
Pay per crawl is a new feature to allow content creators to charge AI crawlers for access to their content. ...
July 01, 2025 10:00 AM
From Googlebot to GPTBot: who’s crawling your site in 2025
From May 2024 to May 2025, crawler traffic rose 18%, with GPTBot growing 305% and Googlebot 96%....
July 01, 2025 10:00 AM
Message Signatures are now part of our Verified Bots Program, simplifying bot authentication
Bots can start authenticating to Cloudflare using public key cryptography, preventing them from being spoofed and allowing origins to have confidence in their identity....
May 15, 2025 1:00 PM
Forget IPs: using cryptography to verify bot and agent traffic
Bots now browse like humans. We're proposing bots use cryptographic signatures so that website owners can verify their identity. Explanations and demonstration code can be found within the post....
March 19, 2025 1:00 PM
Improved Bot Management flexibility and visibility with new high-precision heuristics
By building and integrating a new heuristics framework into the Cloudflare Ruleset Engine, we now have a more flexible system to write rules and deploy new releases rapidly....
March 19, 2025 1:00 PM
Trapping misbehaving bots in an AI Labyrinth
How Cloudflare uses generative AI to slow down, confuse, and waste the resources of AI Crawlers and other bots that don’t respect “no crawl” directives....
March 18, 2025 1:00 PM
Extending Cloudflare Radar’s security insights with new DDoS, leaked credentials, and bots datasets
For Security Week 2025, we are adding several new DDoS-focused graphs, new insights into leaked credential trends, and a new Bots page to Cloudflare Radar. ...