Operations checklist
Crawler log review checklist
Robots.txt tells cooperative crawlers what you prefer. Server logs tell you what actually happened. Reviewing both is the practical way to maintain an AI crawler policy.
What to check in logs
- User-agent names that match AI crawlers or broad datasets.
- Repeated requests to expensive, private, or duplicate paths.
- Requests that ignore robots.txt expectations.
- HTTP status codes such as 200, 301, 403, 404, and 429.
- Whether robots.txt was fetched before deep crawling.
Review table
| Signal | What it may mean | Possible action |
|---|---|---|
| Many hits to /account/ | Private paths are visible or linked | Review access control and internal links. |
| 404-heavy crawl | Old URLs, bad sitemap, or generated paths | Fix sitemap and add redirects where useful. |
| No robots.txt fetch | Client may not be a cooperative crawler | Use server rules or WAF controls if needed. |
| High bandwidth from one bot | Operational cost issue | Rate limit, block, or narrow allowed paths. |
Do not publish log data publicly when it includes IP addresses, private paths, session tokens, or customer information. This checklist is intended for internal review and for deciding whether the public robots.txt file still matches the real traffic pattern.