AI crawler
A bot that fetches web content for an AI service. Training crawlers, search crawlers and user-initiated fetches serve different purposes and have different access controls.
Explanation
GPTBot collects content that may be used for model training. OAI-SearchBot supports ChatGPT search. Their robots.txt settings are independent: a site can allow search crawling while declining training use.
ChatGPT-User visits pages for certain user actions. It is not an automatic search crawler, and robots.txt rules may not apply to those visits.
Allowing a crawler does not guarantee a citation. Check CDN and firewall restrictions as well as robots.txt. Rendering capabilities also differ: Googlebot can render JavaScript, while other bots may not. Keeping essential information in the initial HTML reduces dependence on rendering.
Sources
Updated: Oct 1, 2026