OAI-SearchBot vs GPTBot: Allow ChatGPT Search & Block AI Training in robots.txt

Many website owners accidentally block ChatGPT Search citations when trying to opt out of AI model training. OpenAI operates three separate user-agent tokens—OAI-SearchBot, GPTBot, and ChatGPT-User—and each crawler obeys its own independent section in your /robots.txt file.

If you want your pages to appear as clickable source citations inside ChatGPT Search while preventing your content from being scraped to train future foundation models, you must allow OAI-SearchBot and disallow GPTBot. You can generate this exact configuration interactively in our free robots.txt generator and pair it with a curated /llms.txt index.

OpenAI Crawler Comparison: OAI-SearchBot vs. GPTBot vs. ChatGPT-User

According to OpenAI's official crawler documentation, each bot serves a distinct purpose:

User-Agent Token	Primary Role	Used for Model Training?	Effect of Blocking in robots.txt
OAI-SearchBot	Indexes web pages to surface clickable links and citations in ChatGPT Search results	No	Removes your site from ChatGPT Search source summaries and link cards
GPTBot	Crawls public web pages to train OpenAI generative AI foundation models	Yes	Opts your domain out of future OpenAI generative model training
ChatGPT-User	Fetches a specific live URL when a human user asks ChatGPT to visit or summarize a link	No	Prevents live user-initiated page fetches inside ChatGPT conversations

Copy-and-Paste robots.txt Configuration: Allow AI Search, Block AI Training

Place the following rules at the root of your domain (https://example.com/robots.txt) to allow standard search engines and AI citation engines while blocking major AI training crawlers (GPTBot, Google-Extended, ClaudeBot, and CCBot):

# 1. Allow standard search engines & AI search citation bots
User-agent: *
Disallow: /admin/
Disallow: /api/

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# 2. Block AI foundation model training crawlers
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

Want to customize disallowed folders or toggle between Allow All AI Bots, Allow AI Search Only (Block Training), and Block All AI Bots? Open the NitroSERP robots.txt generator and select the Block AI Training Only preset.

3 Common robots.txt Mistakes That Kill ChatGPT Visibility

1. Blocking User-agent: * and Forgetting Explicit Allow: / Rules

In the Robots Exclusion Protocol, a crawler matches the most specific user-agent group declared for it. If your file has User-agent: * with Disallow: /, any bot without its own explicit User-agent block will be blocked completely. Conversely, once you add a dedicated User-agent: GPTBot block, GPTBot will obey only the rules inside User-agent: GPTBot and ignore User-agent: *.

2. Confusing Google-Extended with Googlebot

Just as OpenAI separates OAI-SearchBot from GPTBot, Google separates Googlebot (which powers Google Search and AI Overviews) from Google-Extended (which controls whether your content trains Gemini models). Blocking Google-Extended does not remove your pages from Google Search, whereas blocking Googlebot de-indexes your entire site.

3. Expecting robots.txt Changes to Update Overnight

OpenAI notes that after you update your /robots.txt file, it can take roughly 24 hours for OpenAI's systems to adjust to the new directives. Always verify that your live https://yourdomain.com/robots.txt returns HTTP 200 OK with Content-Type: text/plain.

Next Steps: Help AI Search Engines Understand Your Best Pages

Allowing OAI-SearchBot lets ChatGPT crawl your pages, but it does not tell AI assistants which URLs on your domain are canonical. Complete your AI search setup with these three steps:

1. Generate your robots.txt file: Use the free robots.txt generator to output clean rules for Googlebot, OAI-SearchBot, and GPTBot.
2. Publish an /llms.txt index: Use our free llms.txt generator to create a Markdown map of your core pages and documentation at https://yourdomain.com/llms.txt.
3. Audit your live ChatGPT citations: Follow the 5-prompt testing workflow in our ChatGPT brand visibility checker guide and verify your XML sitemap in the XML sitemap generator.