Examples

Robots.txt examples for AI crawlers

Examples are useful only when the trade-off is visible. The snippets below are starting points for discussion, not universal recommendations.

Balanced public site

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: *
Disallow: /admin/
Disallow: /account/

Sitemap: https://public-site.test/sitemap.xml

This pattern protects against a training crawler while keeping AI search and user-triggered public retrieval open. It is a common starting point for documentation, product pages, and public help centers.

Publisher with licensed archives

User-agent: GPTBot
Disallow: /archive/
Disallow: /premium/

User-agent: CCBot
Disallow: /

User-agent: Googlebot
Allow: /

User-agent: *
Disallow: /account/
Disallow: /subscribe/

A publisher may want normal search visibility but tighter control over training or broad crawl datasets. The important part is documenting why each archive path is treated differently from current public articles.

Storefront with private account paths

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /order-status/
Allow: /products/
Allow: /collections/

Robots.txt is not enough for private commerce pages. Checkout and account paths must also require authentication and server-side access control. The robots file simply reduces accidental crawling of obvious private paths.

Review checklist

Confirm the sitemap URL returns 200.
Check at least one public URL and one protected URL.
Do not block Googlebot unless you truly want search visibility reduced.
Keep a dated decision note for each crawler class.