Scenario
AI Crawler Policy for SaaS Docs
SaaS companies often want AI tools to retrieve docs, quick starts, API references, and changelogs. The risk is accidentally exposing app paths, dashboards, staging docs, or account URLs.
Suggested workflow
- Allow search and user-triggered fetch bots for public docs.
- Block app, account, billing, admin, and internal paths.
- Create an llms.txt file that links only to canonical docs and API references.
- Use authentication for real private content; robots.txt is not access control.
SaaS documentation policy worksheet
Public product docs are often useful in both normal search and AI answers. Private support tickets, customer workspaces, preview deployments, and internal API references are different. Write those URL groups down before choosing a bot preset.
A practical starting point is to keep public docs open to search crawlers, decide separately about training crawlers, and block private paths for every agent. Authentication must still protect the private paths. Add an llms.txt file only as a map to stable public docs, not as an access-control layer.
After publishing, test the root robots.txt file and a representative docs URL. Repeat the test for each subdomain because robots.txt is host-specific.