Who this is for
A library, research group, or archive that publishes summaries but restricts full documents under license.
Worked example
Research Library AI Crawler Boundary Example
A research-library example for distinguishing public abstracts, licensed PDFs, and metadata pages before writing crawler rules.
Signals to check
- Metadata pages may be public and useful.
- Full PDFs may be licensed or restricted.
- The team needs a record of which parts are public, restricted, or excluded.
Best tool path
Step-by-step workflow
- Map abstract pages, public metadata, downloadable files, and restricted content separately.
- Write crawler rules that match those content classes rather than the whole domain.
- Add an llms.txt map for official public pages if helpful.
- Keep a decision log because licensing and data-sharing rules can change.
A useful crawler policy includes URL groups, intended bot categories, rule text, test examples, and a change note. The useful result is not just a score or file; it is a small record that explains what was checked, what changed, and what should be reviewed next.
Before you start
After the output
Known limitation
Robots.txt is a public crawl instruction. It is not authentication, paywall enforcement, or a substitute for server-side access control. Keep an editable copy and re-run the workflow when the destination requirement or policy changes.