Free Robots.txt Generator with AI Crawler Rules

Generate a robots.txt file with AI crawler rules

A robots.txt file tells search engine bots (Googlebot, Bingbot) and AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot) which paths on your server they may request. This free robots.txt generator adds one-click AI crawler policies on top of the usual Disallow, Allow, and Sitemap rules, so you can opt out of AI model training without disappearing from AI search answers.

AI crawler reference: training vs. search vs. user fetchers

Crawler (User-agent)	Company	Purpose	Blocked by "Block AI training" preset?
GPTBot	OpenAI	Collects content to train models	Yes
ClaudeBot	Anthropic	Collects content to train models	Yes
Google-Extended	Google	Control token for Gemini training (not a separate crawler; Search unaffected)	Yes
Applebot-Extended	Apple	Control token for Apple AI training	Yes
Meta-ExternalAgent	Meta	Collects content to train models	Yes
CCBot	Common Crawl	Open web dataset widely used for AI training	Yes
OAI-SearchBot	OpenAI	Indexes pages for ChatGPT Search citations	No (allowed)
Claude-SearchBot	Anthropic	Indexes pages for Claude search answers	No (allowed)
PerplexityBot	Perplexity	Indexes pages for Perplexity answers	No (allowed)
ChatGPT-User, Claude-User, Perplexity-User	Various	Fetch a page when a user asks about it; may not consult robots.txt	Only in "Block all AI"

Crawler names change; check each company's crawler documentation before relying on a rule. robots.txt is a request, not access control: use authentication or firewall rules for content that must stay private.

Example output: block AI training, keep AI search

User-agent: *
Disallow: /admin/
Disallow: /api/

# Opt out of AI model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: CCBot
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

Grouping several User-agent lines above one rule set is valid under RFC 9309.

Core robots.txt directives reference

Directive	Syntax Example	What It Controls	Crawler Support
User-agent	User-agent: * or User-agent: Googlebot	Specifies which crawler bot the following rules apply to	All compliant bots (RFC 9309)
Disallow	Disallow: /admin/	Instructs the bot not to request URLs starting with that path prefix	Googlebot, Bingbot, AI crawlers
Allow	Allow: /admin/public-help	Overrides a broader Disallow rule for a specific subpath	Googlebot, Bingbot
Sitemap	Sitemap: https://example.com/sitemap.xml	Points crawlers directly to your XML sitemap or sitemap index	Google, Bing, Yandex
Crawl-delay	Crawl-delay: 10	Requests a delay (in seconds) between crawler requests	Respected by Bingbot; ignored by Googlebot

Production robots.txt example (SEO + AI Search ready)

For most websites that want full Google and ChatGPT Search visibility while blocking private admin routes and internal search spam:

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /search?

Sitemap: https://www.example.com/sitemap.xml

robots.txt vs noindex vs Canonical tags

• Use robots.txt Disallow to save crawl budget on non-public directories (/api/, /admin/, infinite faceted filter URLs).
• Use <meta name="robots" content="noindex"> when you want a page completely removed from Google search results. (Important: do not block a noindex page in robots.txt, or Googlebot will never see the noindex tag!)
• Use rel="canonical" when duplicate or UTM-tagged URLs show the same content and you want ranking signals consolidated.

Related technical SEO tools & guides

Build your sitemap file with the free XML sitemap generator, create meta robots tags with the SEO meta tag generator, or consolidate duplicate URLs with the canonical URL generator. For full examples, read our robots.txt beginner's guide and how to allow ChatGPT search crawlers.