Audit existing files
Robots.txt AI Crawler Analyzer
Paste your robots.txt and test whether AI training bots, AI search crawlers, and user-triggered fetchers are allowed or blocked for important URL paths.
Input
AI crawler access result
This analyzer implements the common longest-match robots.txt rule pattern for quick planning. Use Google Search Console and server logs for production verification where available.
How to read the result
The analyzer parses the pasted groups and compares the most specific matching Allow and Disallow paths for each user-agent token. It is useful for spotting a blanket block, a missing search-bot rule, or a private path that was left open in the file.
| Result | Meaning | Next check |
|---|---|---|
| Allowed | No matching disallow rule won for this agent and path. | Check the server or firewall, which can still return 403 or a challenge. |
| Blocked | A matching disallow rule won in the pasted text. | Confirm that the crawler token is the one used by the provider. |
| No specific rule | The wildcard group or default behavior controls the request. | Decide whether an explicit rule would make the policy easier to maintain. |
Limits of a pasted-file test
This tool does not fetch the live file, inspect CDN rules, validate crawler IP addresses, or prove that a bot obeys robots.txt. Production review should include the live /robots.txt response and recent server logs. Google also recommends verifying claimed Googlebot traffic because user-agent text can be spoofed.
Provider references
- OpenAI crawler documentation for GPTBot, OAI-SearchBot, and ChatGPT-User.
- Anthropic crawler documentation for ClaudeBot, Claude-SearchBot, and Claude-User.
- Google crawler documentation for Google-Extended and Googlebot.
- Common Crawl CCBot documentation.