2022-08-12
Cloudflare is uniquely positioned to help give crawlers hints about when they should recrawl, if new content has been added, or if content on a site has recently changed...
Continue reading »
2021-12-16
Cloudflare Is reducing the environmental impact of web searches with 20+ billions crawler hints delivered so far. This blog describes the technical solution of how we built the Crawler Hints system that makes all this possible. ...
2021-10-18
Crawler Hints now supports IndexNow, a new protocol that allows websites to notify search engines whenever content on their website content is created, updated, or deleted. ...
2021-07-28
Crawler Hints provide high quality data to search engine crawlers on when content has been changed on sites using Cloudflare, allowing them to precisely time their crawling, avoid wasteful crawls, and generally reduce resource consumption of customer origins....
