Reference guide
AI crawler user-agent directory
A robots.txt decision starts with user agents, but it should not end there. The same company may operate separate crawlers for model training, AI search, user-triggered fetching, indexing, or infrastructure testing.
| User agent | Common policy question | Typical review note |
|---|---|---|
| GPTBot | Should public content be available for model training? | Often reviewed separately from AI search visibility. |
| OAI-SearchBot | Should pages be discoverable in OpenAI search experiences? | Blocking may reduce the chance of surfacing or citation. |
| ChatGPT-User | Should a user-requested fetch be allowed for public pages? | Usually treated differently from autonomous training crawlers. |
| ClaudeBot | Should Anthropic training or indexing crawlers access public content? | Review licensing, publisher strategy, and source documentation. |
| Claude-SearchBot | Should content appear in Claude-related search or retrieval experiences? | Consider visibility separately from training restrictions. |
| Google-Extended | Should content be used for certain Google AI products? | Do not confuse this with blocking Googlebot for normal Search. |
| CCBot | Should Common Crawl access public content? | Some teams block it because downstream use can be broad. |
| Bytespider | Should the crawler access publisher or product content? | Useful to review if bandwidth, licensing, or content control is a concern. |
How to use this directory
Do not copy a user-agent list blindly. First decide whether the website needs broad visibility, strict content protection, balanced AI search participation, or internal clarity for a regulated workflow. Then test the final robots.txt file against the exact public URLs that matter.
Keep a dated note with each rule. A short note such as "Blocked GPTBot on 2026-07-30 because article archives are licensed separately, allowed OAI-SearchBot for public citation visibility" is more useful than a silent block list. It lets future owners understand the business reason behind the technical file.
Primary-source crawler directory
| Token | Documented role | Source |
|---|---|---|
| GPTBot | OpenAI model-development crawler. | OpenAI |
| OAI-SearchBot | OpenAI search crawler. | OpenAI |
| ChatGPT-User | User-requested OpenAI fetcher. | OpenAI |
| ClaudeBot | Anthropic model-development crawler. | Anthropic |
| Claude-SearchBot | Anthropic search crawler. | Anthropic |
| Google-Extended | A robots.txt control token. Google says it has no separate request user agent. | |
| CCBot | Common Crawl's web crawler. | Common Crawl |
Do not copy an old bot list and assume every name still has the same purpose. Open the provider page, verify the token, then check the live server response. Last source review: September 9, 2026.