Guide
Robots.txt for AI Crawlers
AI crawling is no longer one category. A modern website policy separates training crawlers, AI search crawlers, user-triggered fetchers, and traditional search bots.
The simple decision model
Use three questions before publishing a robots.txt policy: do you want the content used for training, do you want visibility in AI search answers, and which private paths should never be crawled by any crawler?
| Bot type | Typical decision | Reason |
|---|---|---|
| Training crawlers | Allow or block by content strategy | Useful for contribution to model training, but many publishers block them when content value or licensing is a concern. |
| AI search crawlers | Often allow | Blocking can reduce the chance that your pages are surfaced or cited in AI search experiences. |
| User-triggered fetchers | Usually allow for public pages | These fetchers often respond to a user explicitly asking an AI tool to view a page. |
| Traditional search crawlers | Allow | Blocking Googlebot or Bingbot can harm normal search indexing. |
Common mistake
Do not use one broad rule that blocks every crawler unless you intend to remove search visibility too. A site can block GPTBot while allowing OAI-SearchBot, or block ClaudeBot while allowing Claude-SearchBot.