Scenario
AI Crawler Policy for News Publishers
Publishers need a more deliberate policy because AI summaries can affect traffic, licensing, attribution, and newsroom economics.
Suggested workflow
- Separate training crawler decisions from AI-search citation decisions.
- Keep archives, author pages, and evergreen explainers in a crawlable structure if AI citation traffic matters.
- Use paywall, subscriber, and account controls outside robots.txt.
- Document licensing and contact information in public policy pages.
Newsroom policy worksheet
A publisher may want articles discoverable in search while keeping licensed archives or subscriber-only material out of model-development crawls. Start by separating public article URLs, live blogs, image archives, account pages, and paid content. A single site-wide block rarely expresses all of those choices.
- Keep Googlebot rules separate from AI-specific tokens.
- Decide whether OAI-SearchBot and Claude-SearchBot may reach public articles.
- Use authentication for subscriber content. Robots.txt is public and does not secure a paywall.
- Review logs after publishing to see which documented agents actually request the site.
Use the generator to create a draft, then have the newsroom and technical owner approve the same URL groups.