Scenario

AI crawler policy for university research sites

Research groups often want broad public discovery while protecting draft materials, unpublished datasets, student records, and lab administration pages.

Allow public research pages

Project descriptions, publications, accepted papers, lab profiles, and public documentation may benefit from normal search and AI search discovery.

Protect unpublished work

Draft papers, private datasets, grant materials, and internal review notes should not rely on robots.txt alone. Use authentication and access control.

Document the policy

Labs change maintainers often. A short decision note helps the next student or administrator understand why the crawler policy exists.

Recommended workflow

  1. List public sections: papers, projects, datasets, software, people, and news.
  2. List private sections: admin, course work, unpublished data, staging, and internal documents.
  3. Generate a robots.txt draft and test representative paths.
  4. Create an llms.txt file that highlights papers, datasets, source repositories, and citation pages.