Free browser-based tool
AI Crawler Robots.txt Generator
Create a modern robots.txt file that separates AI training crawlers from AI search bots and user-triggered fetchers. Nothing is uploaded to a server.
Policy settings
Balanced blocks common training crawlers while allowing AI search and user fetchers.
One path per line. These are added to every allowed bot group and the wildcard group.
Per-bot decisions
Generated file
Recommended first policy
Keep Googlebot open
Googlebot is still the core search crawler. Blocking it can remove pages from normal Google Search visibility.
Separate training from search
GPTBot and OAI-SearchBot have different purposes. Treating them separately lets you protect content while keeping AI-search discoverability.
Protect private paths elsewhere too
Robots.txt does not secure admin or customer pages. Use authentication and server access rules for anything private.