Scenario library

AI Crawler Policy Scenarios

Start with the website type closest to your business. Each scenario explains the crawler trade-off and points into the right tool.

Protect original writing while keeping public posts discoverable through search and AI answers.

Help agents find docs and examples without exposing private app or account paths.

Separate citation visibility from model-training access and licensing concerns.

Keep product discovery open while blocking account, cart, and checkout paths.