Tool directory
AI crawler policy tools
Use this page to choose the right workflow before editing a public crawler policy. Each tool runs in the browser and is designed to produce a concrete artifact that can be reviewed before publishing.
Create
Robots.txt Generator
Generate a policy that separates AI training crawlers, AI search crawlers, user-triggered fetchers, and traditional search bots. Best for new sites or teams replacing a generic all-bot rule.
AuditRobots.txt Analyzer
Paste an existing robots.txt file and check common AI crawler access outcomes against representative URL paths. Best before migration, review, or AdSense/Search Console cleanup.
DocumentLLMs.txt Generator
Create a curated Markdown map of public pages that AI agents and assistants can use to understand your site without crawling low-value pages.
Which tool should you start with?
| Situation | Start here | Why |
|---|---|---|
| You have no AI crawler policy yet | Robots.txt Generator | It gives you a copy-ready draft and shows the trade-off behind each bot group. |
| You inherited a robots.txt file | Robots.txt Analyzer | It helps detect broad blocks, duplicated groups, missing sitemap lines, and confusing access outcomes. |
| You want AI assistants to understand docs | LLMs.txt Generator | It creates a public map of valuable pages while keeping private or thin pages out of the summary. |
| You need an internal decision record | Generator plus Analyzer | Create the draft, then audit it and save both the assumptions and the result. |
These tools do not bypass crawler behavior, security controls, paywalls, or platform policy. They help you express preferences and create a reviewable policy artifact.