Who this is for
A creator or small publisher using WordPress with public posts, tag pages, media attachments, and admin paths.
Worked example
WordPress Blog Robots.txt Example
A WordPress blog example for managing AI crawler access without accidentally blocking useful public posts.
Signals to check
- The public posts should remain accessible.
- Admin and login routes should not be indexed.
- Attachment and tag pages may create duplicate or low-value crawler paths.
Best tool path
Step-by-step workflow
- Keep public post and category routes available unless there is a specific licensing reason to block them.
- Block admin, login, and private workflow paths.
- Review tag, search, and attachment pages for duplication.
- Use the analyzer after publishing to confirm the final behavior.
A useful crawler policy includes URL groups, intended bot categories, rule text, test examples, and a change note. The useful result is not just a score or file; it is a small record that explains what was checked, what changed, and what should be reviewed next.
Before you start
After the output
Known limitation
Robots.txt is a public crawl instruction. It is not authentication, paywall enforcement, or a substitute for server-side access control. Keep an editable copy and re-run the workflow when the destination requirement or policy changes.