Free browser-based tool

AI Crawler Robots.txt Generator

Create a modern robots.txt file that separates AI training crawlers from AI search bots and user-triggered fetchers. Nothing is uploaded to a server.

Policy settings

Policy preset

Balanced blocks common training crawlers while allowing AI search and user fetchers.

Website URL

Sitemap URL

Paths to protect from all crawlers

One path per line. These are added to every allowed bot group and the wildcard group.

Per-bot decisions

Generated file

Recommended first policy

Keep Googlebot open

Googlebot is still the core search crawler. Blocking it can remove pages from normal Google Search visibility.

Separate training from search

GPTBot and OAI-SearchBot have different purposes. Treating them separately lets you protect content while keeping AI-search discoverability.

Protect private paths elsewhere too

Robots.txt does not secure admin or customer pages. Use authentication and server access rules for anything private.

What the generated rules control

Training collection

GPTBot and ClaudeBot are documented as model-development crawlers. A publisher can block those tokens while making a separate decision about search or user-requested access.

AI search discovery

OAI-SearchBot and Claude-SearchBot support search features. Blocking a training crawler does not automatically express the same choice for an AI search crawler.

User-requested fetches

ChatGPT-User and Claude-User can fetch a page after a person asks an assistant to open it. Provider documentation may treat these requests differently from automatic crawling.

Tested behavior: the generator creates a separate user-agent group for each selected bot, adds the entered paths, preserves a Googlebot group, and includes the sitemap URL. Download the file, review it, then publish it at the root of the exact host it controls.

Source register

OpenAI crawler documentation for GPTBot, OAI-SearchBot, and ChatGPT-User.
Anthropic crawler documentation for ClaudeBot, Claude-SearchBot, and Claude-User.
Google crawler documentation for Google-Extended and Googlebot.
Common Crawl CCBot documentation.

Sources reviewed September 9, 2026. Bot names and provider behavior can change, so recheck the linked provider page before a major policy change.