Who this is for
An online store owner balancing discoverability, server load, dynamic filters, and private account paths.
Worked example
Ecommerce Product Pages Bot Policy Example
An ecommerce example for deciding which product, search, cart, and account paths should be open to crawlers.
Signals to check
- Product pages should be found in search.
- Filtered search URLs can create crawler waste.
- Cart, checkout, and account paths should not be crawled.
Best tool path
Step-by-step workflow
- Allow clean product and category URLs that help discovery.
- Block internal search, cart, checkout, and account routes.
- Consider crawl-delay or CDN rules separately from robots.txt when load is the real issue.
- Test several URLs with expected allow and disallow outcomes.
A useful crawler policy includes URL groups, intended bot categories, rule text, test examples, and a change note. The useful result is not just a score or file; it is a small record that explains what was checked, what changed, and what should be reviewed next.
Before you start
After the output
Known limitation
Robots.txt is a public crawl instruction. It is not authentication, paywall enforcement, or a substitute for server-side access control. Keep an editable copy and re-run the workflow when the destination requirement or policy changes.