1. Test the live URL
Open the production /robots.txt URL over HTTPS, not only a local file. Confirm the exact content is being served on the canonical domain and the www or non-www variant.
Testing workflow
How to test AI crawler access
A robots.txt file can look correct in source control and still behave differently online because of CDN caching, redirects, host aliases, or a missing sitemap line.
2. Test paths
Use representative paths: homepage, article or docs page, product page, admin path, account path, and one URL that should never be indexed.
3. Test after cache
If a CDN is present, purge or wait for cache expiry. Crawler decisions can lag if a stale file remains at the edge.
What to record
| Field | Example | Reason |
|---|---|---|
| Date | 2026-07-30 | Policies change; a date makes later reviews easier. |
| Domain tested | https://public-site.test/robots.txt | Prevents confusion between root, www, staging, and CDN aliases. |
| Policy intent | Block training, allow AI search | Connects the technical rule to a business reason. |
| Representative paths | /docs/, /products/, /account/ | Shows that the policy was tested beyond the homepage. |
Testing is also a content-quality signal for your own team. It turns a robots.txt change from a copied snippet into a maintainable decision. That matters when a founder, editor, SEO owner, or developer asks why a crawler was allowed or blocked months later.