Robots.txt Guide: Examples, Rules, and Indexing Checks

Robots.txt tells compliant crawlers which paths they may request on a website. Put it at the root of the relevant host, such as https://www.example.com/robots.txt. It manages crawling, but it does not secure private information or guarantee that a URL will stay out of Google.

This guide uses NitroSERP's public robots.txt as a worked example, then explains how to choose rules and diagnose Search Console messages without accidentally blocking useful pages.

A real robots.txt example

NitroSERP's live robots.txt, checked on September 23, 2026, contains:

User-Agent: *
Allow: /
Disallow: /api/

Host: https://www.nitroserp.com
Sitemap: https://www.nitroserp.com/sitemap.xml

Here is what matters for Google:

Line	Meaning
User-agent: *	Rules for crawlers without a more specific matching group
Allow: /	Allows crawling of the root and paths unless a more specific rule applies
Disallow: /api/	Blocks requests to paths beginning with /api/
Sitemap:	Gives the absolute URL of the XML sitemap
Host:	Not a directive Google supports; it does not choose your canonical domain

For example, /tools/json-formatter and /blog/robots-txt-beginners-guide are allowed by these rules. /api/tools/dns is disallowed. Blocking an API route should be a deliberate choice: if a page needs that route to display its main content to Google, it can interfere with rendering.

Google generally uses the most specific matching path rule; when equally specific Allow and Disallow rules conflict, the less restrictive rule applies. Rules are case-sensitive for paths. Check Google's robots.txt specification when combining groups or patterns.

Allow and Disallow examples

Allow all public pages

User-agent: *
Disallow:

Sitemap: https://www.example.com/sitemap.xml

An empty Disallow value blocks nothing. Replace the example domain with your own. Most small public sites do not need a complicated list of exclusions.

Block a folder with one public exception

User-agent: *
Disallow: /downloads/
Allow: /downloads/public-guide.pdf

This allows the more specific PDF path while blocking other paths in that folder. It is only a crawling rule. Anyone with the URL can still request those files unless your server requires authentication.

Recognize an accidental whole-site block

User-agent: *
Disallow: /

This requests that compliant crawlers avoid the entire site. It is a common staging configuration that should not be copied to a public site intended for search. Use authentication for private staging environments.

Robots.txt versus noindex, canonicals, and redirects

Choose the control that matches your goal:

Goal	Appropriate control
Limit crawling of unnecessary paths	robots.txt
Keep an accessible page out of search	A noindex meta tag or response header, while allowing crawling
Protect private data	Authentication and server access controls
Identify the preferred duplicate URL	A canonical link
Move visitors and crawlers to a replacement URL	A permanent redirect

Google must fetch a page to read its noindex directive. Blocking the same page in robots.txt can prevent that directive from being seen. Google does not support placing noindex rules inside robots.txt. Read its noindex documentation before combining these controls.

Do not block a referral URL just because it appears under “Alternate page with proper canonical tag.” For example, a homepage URL containing ?ref=launches.uicomet.com can correctly canonicalize to the clean homepage. Google needs access to see that relationship.

How to create and check your file

1. List the public pages you want search engines to discover and any paths you intentionally want to exclude from crawling.
2. Create a plain-text file using the Robots.txt Generator. Review every path before publishing.
3. Serve it at /robots.txt on the correct host. A file on example.com does not automatically govern www.example.com.
4. Open the live URL and check that it returns the intended plain text, not a login form, error page, or HTML document.
5. Include your absolute sitemap URL. List preferred, indexable page URLs in the sitemap; omit redirects and tracking variants.
6. Inspect a representative page in Search Console using Test Live URL. Check crawl access and the rendered result, including important CSS and JavaScript resources.

Keep a copy of the previous file so you can restore it if a rule blocks important pages. Google's guide to creating robots.txt explains supported file placement and syntax.

Does robots.txt cause “Crawled—currently not indexed”?

That message means Google fetched the page but did not index it at that time. It is not the same as “Blocked by robots.txt,” and adding more Allow rules does not force indexing.

For a page you want indexed, check:

• The live URL returns 200 and has substantive, visible content.
• Neither the page nor its response headers contain an unintended noindex directive.
• The canonical points to the correct preferred URL.
• Relevant pages link to it using ordinary crawlable links.
• Its content serves a distinct purpose, with examples or functionality useful to readers.

In Search Console, compare the last crawl date and Google-selected canonical with the current page. A successful live test checks present accessibility; it does not confirm that the page is indexed. Google's Page indexing report documentation explains the distinction.

Common questions

Do I need robots.txt to appear in Google?

No. A small public site may not need crawl restrictions. A robots.txt file can communicate exclusions and point to a sitemap, but it is not a requirement for indexing. See Google's robots.txt introduction.

Should I block CSS and JavaScript?

Avoid blocking resources needed to display your pages. A crawler that cannot render the main content may see an incomplete page. For a Next.js site, blanket blocking of /_next/ can interfere with essential resources.

Does adding a sitemap guarantee indexing?

No. A sitemap helps discovery. Google still decides which pages to index. Use accurate modification dates when known, and avoid refreshing every URL's date solely because a deployment occurred. See our XML sitemap guide.

Do these rules also control AI search?

Different services use different crawler identifiers and policies. Google Search crawling and ChatGPT search discovery should be checked separately. Our AI search visibility guide explains the relevant checks.

Start with rules you can explain path by path. Use the Robots.txt Generator to draft them, inspect the live file, and verify access to important pages before treating the change as complete.