Does Anthropic crawl data from the web, and how can site owners block the crawler? (original) (raw)

  1. All Collections
  2. Privacy and legal
  3. Does Anthropic crawl data from the web, and how can site owners block the crawler?

As per industry standard, Anthropic uses a variety of robots to gather data from the public web for model development, to search the web, and to retrieve web content at users’ direction. Anthropic uses different robots to enable website owner transparency and choice. Below is information on the three robots that Anthropic uses and how to set your site preferences to enable those you want to access your content and limit those you don’t.

As part of our mission to build safe and reliable frontier systems and advance the field of responsible AI development, we’re sharing the principles by which we collect data as well as instructions on how to opt out of our crawling going forward:

To limit crawling activity, we support the non-standard Crawl-delay extension to robots.txt. An example of this might be:

User-agent: ClaudeBot

Crawl-delay: 1

To block a Bot from your entire website, add this to the robots.txt file in your top-level directory. Please do this for every subdomain that you wish to opt out from. An example of this is:

User-agent: ClaudeBot

Disallow: /

Opting out of being crawled by Anthropic Bots requires modifying the robots.txt file in the manner above. Alternate methods like blocking IP address(es) from which Anthropic Bots operates may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file. If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic.

You can learn more about our data handling practices and commitments at our Help Center. If you have further questions, or believe that our Bots may be malfunctioning, please reach out to [email protected]. Please reach out from an email that includes the domain you are contacting us about, as it is otherwise difficult to verify reports.

Related Articles

Reporting, Blocking, and Removing Content from ClaudeHow to get supportReporting, Blocking, and Removing Content from ClaudeGet started with Claude in ChromeClaude in Chrome Permissions Guide