Robotcop: enforcing your robots.txt policies and stopping bots before they reach your website
2024-12-10
Today, the AI Audit dashboard gets an upgrade: you can now quickly see which AI services are honoring your robots.txt policies and then automatically enforce the policies against those that aren’t. ...
