Scenario

AI Crawler Policy for Blogs

Independent writers need discoverability, but may not want every article used for model training. A balanced policy usually blocks training bots and allows search-style bots.

Suggested workflow

  1. Generate a balanced robots.txt policy.
  2. Keep Googlebot, OAI-SearchBot, Claude-SearchBot, and user fetchers open for public posts.
  3. Block admin, drafts, tag experiments, and account paths.
  4. Add an llms.txt file with your best evergreen posts and contact page.
Generate blog policyCreate llms.txt