Scenario
AI crawler setup for WordPress blogs
A WordPress blog has useful public posts, but it also has admin paths, search result pages, tag archives, feed URLs, and duplicate pagination that may not be worth exposing to every crawler.
Useful public pages
Start by protecting the quality of what crawlers see. Include evergreen posts, category hubs, author pages when they are maintained, and cornerstone guides. Exclude thin tag pages, internal search results, login pages, preview URLs, and private member paths.
| Path | Typical decision | Reason |
|---|---|---|
| /wp-admin/ | Disallow and require auth | Administrative path, not public content. |
| /?s= | Disallow or noindex | Internal search pages can create low-value duplicates. |
| /category/main-topic/ | Usually allow | Helpful topic hub if curated. |
| /tag/random/ | Review carefully | Often thin or duplicate unless maintained. |
For AdSense-oriented sites, this also matters because thin archive pages can dilute perceived site value. A cleaner crawler policy and better internal links help crawlers find the strongest pages first.
WordPress setup notes
Publish robots.txt at the root of the live host, not inside the WordPress media library. A generated file may be overridden by a security plugin, SEO plugin, CDN rule, or managed host. Open the public URL after every change.
Keep /wp-admin/ restricted and avoid blocking assets that normal search rendering needs. If the blog uses a separate media or shop subdomain, review that host separately because each host has its own robots.txt.
- Generate the draft.
- Compare it with the current live file.
- Publish through the host or plugin that owns robots.txt.
- Open the public file and test representative paths in the analyzer.