Scenario
AI Crawler Policy for Ecommerce Stores
Stores usually want product pages discoverable, but cart, checkout, account, search result filters, and internal campaign URLs should be handled carefully.
Suggested protected paths
/cart/ /checkout/ /account/ /login/ /admin/ /search?
Use robots.txt for crawl preference and noindex or canonical rules for search quality. Real customer data must remain behind authentication.
Ecommerce policy worksheet
Product pages, category pages, on-site search, cart routes, customer accounts, and checkout URLs have different jobs. Product discovery may benefit from search access, while carts and accounts should be protected by application security and excluded from crawling.
- Keep canonical product and category URLs crawlable if they are meant to be discovered.
- Block faceted or internal-search URLs when they create duplicate crawl paths.
- Exclude cart, checkout, account, and admin routes for all automated crawlers.
- Make a separate policy choice for AI training and AI search tokens.
Robots.txt does not protect prices, inventory feeds, or customer data. Use authentication and server controls for those resources.