OpenAI crawler decisions
GPTBot vs OAI-SearchBot vs ChatGPT-User
OpenAI documents multiple crawler and fetcher roles. Treating them as one bot can accidentally block the traffic you want or allow the training access you meant to restrict.
| User-agent token | Role | Typical balanced decision |
|---|---|---|
| GPTBot | Training crawler | Block when you do not want public pages used for future model improvement. |
| OAI-SearchBot | Search crawler | Allow when you want eligibility for OpenAI search discovery and citations. |
| ChatGPT-User | User-triggered fetcher | Allow for public pages so users can ask ChatGPT to retrieve a page. |
Copy-ready balanced pattern
User-agent: GPTBot Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: /
Review this with the generator before publishing. Add private paths under each allowed group if there are public URL patterns you still want crawlers to skip.