Table of Contents

Background

Many service providers wish to restrict certain Web activities including spambot registration, web crawlers and more. LLM-associated user agents are an increasing source of traffic, consuming network resources, driving up individual hosting costs, as well as storing content it finds forever.

Some of these bots honour the robots.txt directives, but it’s up to each individual domain admin to get that file in place, on each domain and sub-domain you oversee. For other bots that ignore or work around robots.txt, other measures need to be taken.

Community Resources

• https://kolektiva.social/@ophiocephalic/115011403604370053 – will share links and resources for self CDN, and DDOS protection
• https://eupolicy.social/about – a terms of service that prohibits machine learning or research
• https://lobste.rs/s/s9yq5a/block_ai_scrapers_with_anubis – discussion around using Anubis and other methods

Crawler Identification Resources

• Dark Visitors – A List of Known AI Agents on the Internet
  • Automated robots.txt file for WordPress
• udger crawler UA list
• Eight tips about consent for fediverse developers – background on consent in the Fediverse as it relates to web scraping
• https://mastodon.bentasker.co.uk/@scrapersnitch – automated OSINT account to notify of possible Fediverse scrapers
• https://momenticmarketing.com/blog/ai-search-crawlers-bots#complete-ai-crawler-list – routinely updated list of AI Search Web crawlers

Managed Blocks

Both Cloudflare and Fastly offer detection and blocking.

Notable Bots

Amazon

Amazonbot is Amazon’s web crawler used to improve services, such as enabling Alexa to answer questions for customers. Amazonbot respects standard robots.txt rules.

To disallow Amazonbot using robots.txt:

User-agent: AmazonbotDisallow: /

Anthropic AI

Anthropic is a U.S.-based AI company researching artificial intelligence as a public-benefit company to develop AI systems to “study their safety properties at the technological frontier” and use this research to deploy safe, reliable models for the public. Anthropic has developed a family of large language models (LLMs) named Claude as a competitor to OpenAI’s ChatGPT and Google’s Gemini.

To disallow ClaudeBot using robots.txt:

User-agent: ClaudeBot Disallow: /

AppleBot-Extended

With Applebot-Extended, web publishers can choose to opt out of their website content being used to train Apple’s foundation models powering generative AI features across Apple products, including Apple Intelligence, Services, and Developer Tools. Applebot-Extended does not crawl webpages. Webpages that disallow Applebot-Extended can still be included in search results. Applebot-Extended is only used to determine how to use the data crawled by the Applebot user agent.

To disallow AppleBot-Extended using robots.txt:

User-agent: AppleBot-ExtendedDisallow: /

Additional information, including information about Apple’s web crawler “AppleBot”, is available at https://support.apple.com/en-us/119829

CCBot

Common Crawl is a non-profit foundation founded with the goal of democratizing access to web information by producing and maintaining an open repository of web crawl data that is universally accessible and analyzable by anyone. The user agent is CCBot/2.0.

To disallow CCBot using robots.txt:

User-agent: CCBotDisallow: /

FacebookBot

FacebookBot crawls public web pages to improve language models for speech recognition technology.

To disallow FacebookBot using robots.txt:

User-agent: FacebookBotUser-agent: meta-externalagentDisallow: /

Google

Google-Extended is Google’s training data crawler.

To disallow Google-Extended using robots.txt:

User-agent: Google-ExtendedDisallow: /

To Disallow all Google bot traffic by IP address, see this json file

Microsoft

User-agent: BingBotDisallow: /

OpenAI

GPTBot is OpenAI’s web crawler and can be identified by the following user agent and string:

To disallow GPTBot using robots.txt:

User-agent: GPTBotDisallow: /