July 29, 2026 update: We compared RFC 9309, Google's current robots.txt specification and Search Console report, Google-Extended, and OpenAI's OAI-SearchBot and GPTBot, then added settings for AI search and training alongside conventional SEO.
The short answer: robots.txt is a public set of rules telling crawlers whether they may fetch URLs. It does not remove pages from search results or protect admin screens and confidential files. The safest approach is to distinguish the purpose and create a decision matrix for each origin and URL.
Use robots.txt for crawling and separate tools for search exclusion and access protection
Robots.txt can reduce crawling of unnecessary search-result pages and duplicate URLs. Use noindex to remove pages from search results and authentication and authorization to protect secrets. Before and after release, verify the file's HTTP response and representative URL decisions separately.
| What is robots.txt? | Public text telling compliant crawlers whether they may fetch URLs on the same protocol, host and port |
|---|---|
| Location | The lowercase file at each origin's root: /robots.txt |
| Core directives | User-agent, Allow, Disallow, with the additional record Sitemap |
| SEO considerations | A disallowed URL can remain in search results. Do not block URLs where noindex must be read. |
| Key Google-specific limits | Processes up to 500 KiB, usually caches for up to about 24 hours and follows at least five redirects |
| Recorded check date for main sources | July 25, 2026 |
What is robots.txt? Understand its role and limits first
Search engines fetch a URL before understanding a page. Robots.txt manages this entry point, so its role differs from noindex in a page's robots meta tag or HTTP header.
Robots.txt is most useful for large, low-value URL spaces such as site search, filters, session URLs and nearly infinite calendars. It does not solve URL-design problems alone: also review internal links, canonical URLs, sitemaps, parameters and server performance.
Robots.txt is a request, not a security boundary.
Use authentication, authorization, WAF, network restrictions and rate controls to stop malicious clients or unsupported crawlers.
Do not confuse crawl control, search exclusion and access protection
Before configuring rules, classify URLs by whether you want to reduce fetching, remove them from search results or stop public access. Writing Disallow without this distinction can prevent noindex from being read, block important pages or expose sensitive paths.
| Purpose | Main approach | Role of robots.txt | What to check |
|---|---|---|---|
| Reduce crawling of unnecessary URLs | Robots.txt, URL design and internal-link cleanup | Suitable | Whether important pages or rendering resources are affected |
| Remove a page from search results | Robots meta noindex, X-Robots-Tag, removal or 404/410 | Do not block crawling | Whether crawlers can fetch noindex |
| Protect admin screens and customer data | Authentication, authorization, MFA and network restrictions | Does not provide protection | Whether anonymous access is actually denied |
| Temporarily hide a result in an emergency | Take content offline, noindex, Search Console removal tools, and similar measures | Not sufficient on its own | Long-term controls and recrawl requirements |
Allow crawling where noindex must be read
Serve a robots meta directive for HTML or X-Robots-Tag for PDFs and similar files, and allow crawlers to fetch it. A recrawl is needed for the change to take effect.
Disallow hides secrets and removes search results
Robots.txt is public. A URL discovered through an external link may appear in search results without its content being fetched.
Scope is determined by protocol, host and port
Robots.txt is fetched from /robots.txt at the root of its origin. An HTTPS file is not automatically inherited by HTTP, another subdomain or a nonstandard port. An omitted standard port is equivalent, while nonstandard ports are managed separately.
| Robots.txt URL | Applies to | Does not apply to | Operational action |
|---|---|---|---|
| https://example.com/robots.txt | https://example.com/page | http://example.com/ https://www.example.com/ | Also check HTTP and www if they are public |
| https://shop.example.com/robots.txt | https://shop.example.com/cart | https://example.com/ https://cdn.example.com/ | Manage an owner and rules for each subdomain |
| https://example.com:8181/robots.txt | https://example.com:8181/admin/ | https://example.com/ (443) | First review whether public access is needed |
| https://example.com/help/robots.txt | Not a valid location | The entire site | Move it to /robots.txt at the root |
Test case, trailing slashes, encoding and query strings against real URLs./File/ and /file/ are different paths. A URL fragment such as #section is not sent to the server, so robots.txt cannot distinguish it.
Determine the user-agent group and most specific path rule
The standard centers on User-agent, Allow, Disallow. Sitemap is an additional record supported by major search engines. Field names are case-insensitive, but path values are case-sensitive.
User-agent
Specify the target crawler's product token. * applies to crawlers without a more specific matching group.
Disallow
This rule disallows fetching matching URLs. An empty value imposes no restriction; / matches the entire site.
Allow
Allow a more specific URL within a broad Disallow. A rule does not win merely because it appears later.
Sitemap
List the full sitemap URL. It is not an Allow or Disallow rule within a user-agent group.
Wildcards
For Google, * represents zero or more characters, and $ marks the end of the URL. Check the official specification for other crawlers.
Comments
# and the rest of the line are ignored. You may include reasons and change-ticket numbers, but not secrets or internal architecture.
User-agent: *
Disallow: /search/
Allow: /search/help/
Sitemap: https://example.com/sitemap.xmlChoose the matching group, then the longest matching rule
Disallow: /catalog/ and Allow: /catalog/help/ are present, /catalog/item/1 is disallowed, while /catalog/help/size is allowed by the more specific Allow rule.Write robots.txt safely: configuration examples
Do not copy examples unchanged. Check the target origin, real URLs, search value, noindex and rendering resources. Fewer rules are easier to review and faster to restore after an incident.
Allow all crawlers and advertise the sitemap
For a small site with no URLs to block. An empty Disallow also means no restriction, but Allow: / can make the intent easier to read.
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlBlock site search but allow help pages
For a site that generates many search-result URLs. If help pages share the directory, create an exception with a more specific Allow.
User-agent: *
Disallow: /search/
Allow: /search/help/
Sitemap: https://example.com/sitemap.xmlRestrict a specific query pattern
A Google-specific example for cases where sorting or tracking parameters produce an almost infinite URL space. Test representative URLs to avoid blocking necessary landing pages.
User-agent: Googlebot
Disallow: /*?sort=
Disallow: /*&sort=
Disallow: /*?session=Disallow: / is a stop condition. Automatically detect staging files, environment-variable errors, CDN cache issues and wrong deployment targets. Stop the release if important URLs are blocked site-wide.Manage AI search and training crawlers by purpose in 2026
AI-related user-agents are not all the same. Crawlers used for search results, crawlers used for foundation-model training and agents fetching pages at a user's request have different purposes and may treat robots.txt differently.
| User-agent | Main purpose | What a control changes | Considerations |
|---|---|---|---|
| Google-Extended | Gemini model training and grounding for Gemini and Vertex AI | Controls whether Google-crawled site content is used for these purposes | A usage-control token, not a separate HTTP crawler; it does not affect Google Search inclusion or ranking |
| OAI-SearchBot | Display websites in ChatGPT search | Allowing it preserves the possibility of appearing in ChatGPT search results | Independent of GPTBot; changes may take time to take effect |
| GPTBot | Crawl content that may be used to train generative AI foundation models | Use Disallow to indicate that training use is not desired | You can allow OAI-SearchBot while blocking GPTBot |
| ChatGPT-User | A user asks ChatGPT or a custom GPT to fetch a page | Not used to control automated search crawling | Robots.txt may not apply because the request is user-initiated |
# Preserve opportunities to appear in ChatGPT search
User-agent: OAI-SearchBot
Allow: /
# Indicate that content should not be used for model training
User-agent: GPTBot
Disallow: /
# Control Gemini-related use without affecting Google Search
User-agent: Google-Extended
Disallow: /Use robots.txt and noindex for their distinct SEO purposes
Robots.txt communicates whether a URL may be fetched; noindex communicates whether it may remain in a search index. Since noindex appears in the page or response headers, a crawler must be able to access the URL to read it.
| Comparison | robots.txt | noindex / X-Robots-Tag | Authentication and authorization |
|---|---|---|---|
| Main purpose | Manage crawl scope | Exclude from search results | Control access to content |
| Location | The origin's /robots.txt | HTML meta tag or HTTP header | Application, server or identity system |
| Crawler access | Prevent fetching the target URL | Fetching is needed to read the directive | Access is denied without permission |
| Can it protect secrets? | No | No | Yes, when implemented correctly |
| Can the URL remain in search results? | Yes | Excluded after a recrawl | Address separately based on public availability and the existing index |
How to remove a URL already in search results
- Remove the robots.txt block
Allow the crawler to fetch the page or response header.
- Serve noindex or X-Robots-Tag
Choose a method appropriate for HTML, PDF, images or the relevant format.
- Check the recrawl with URL Inspection and logs
Check that the live response matches what Google fetched.
- Choose a permanent state
Keep noindex if the content remains public, remove it with 404 or 410 if no longer needed, or require authentication if it is confidential.
Treat dangerous settings as release stop conditions
Correct syntax can still cause an SEO incident if target URLs or objectives are wrong. Detect the following through change review and automated checks.
Disallow: / in production
This blocks the entire site. Prioritize detecting staging settings and CDN cache contamination.
Blocking URLs that need noindex
Crawlers cannot fetch noindex or X-Robots-Tag, which can prevent removal from search results.
Listing sensitive paths
/backup/ and administrative URLs are exposed. Use authentication and authorization for access control.
Blocking all CSS, JavaScript and images
Blocking resources needed for rendering and page understanding can prevent search engines from evaluating content properly.
Using 401 or 403 as a block
Google treats 4xx responses other than 429 as if no valid robots.txt exists, assuming no crawl restrictions.
Long redirect chains and HTML responses
Check the body and Content-Type for login screens, WAF pages, 404 HTML or language redirects.
An oversized file
Google ignores content beyond 500 KiB. Organize URL design and consolidate rules.
Relying on crawl-delay
Google does not support crawl-delay. Diagnose load using logs, URL space, performance and errors.
Guessed crawler names
Do not block groups of similar user-agent strings by guesswork. Use official product tokens and IP verification methods.
Monitor HTTP responses, cache and file format as closely as rules
The same body can be interpreted differently depending on status code, Content-Type, encoding, redirects and cache. Record Google's behavior separately from RFC 9309.
| Response | Google's main behavior | Operational checks |
|---|---|---|
| 2xx | Parse the fetched rules | Check 200, UTF-8, plain text, the body and Cache-Control |
| 3xx | Follow at least five redirects, then treat the file as a 404 | Serve a direct 200 where possible; redirects should lead to the intended /robots.txt |
| 4xx except 429 | Assume no crawl restrictions as if no valid robots.txt exists | Do not use 401 or 403 to stop crawling; distinguish an absent file from a delivery failure |
| 429 | Handled differently from ordinary 4xx; investigate as a rate-limit or availability issue | Check excessive restrictions, WAF, CDN, origin load and retries |
| 5xx and network failures | Stop crawling for the first 12 hours. Then use the last good version for up to about 30 days while retrying; if no good version is cached, assume no crawl restrictions | Monitor the last good version, recovery time, DNS, TLS, timeouts and server logs |
Create a URL decision matrix and deploy minimal rules
Before writing robots.txt, list target URLs and crawlers, expected results, alternative controls and verification methods. Make important URLs and rendering resources mandatory allowed test cases.
- List the origins in scope
List HTTP, HTTPS, www, subdomains, CDNs, image hosts and nonstandard ports, and assign an owner for each /robots.txt.
- Classify representative URLs as allowed, disallowed or separately controlled
Include the home page, important subpages, search and filter URLs, admin screens, APIs, CSS, JavaScript, images, PDFs, sitemaps and nonexistent URLs.
- Choose the right mechanism
Separate robots.txt for crawl management, noindex for search exclusion and authentication for secrets.
- Write the minimum groups and rules
Avoid duplicate groups and overly broad wildcards; keep only necessary Disallow rules and Allow exceptions.
- Automate URL decisions locally and in staging
Test user-agents, trailing slashes, case, queries and encoding.
- Choose the authoritative delivery layer and cache behavior
Decide whether the application, web server or CDN owns the canonical file, and prevent environment-specific files from being mixed up.
- Prepare the previous version, stop conditions and owner before release
Be ready to restore the previous version if a global block, important URL block, HTML response or 5xx is detected.
Verify file fetching and URL decisions separately
/robots.txt opening in a browser does not prove that the crawler and URL decisions are correct. Check the HTTP response, syntax, URL decisions, Search Console and logs in stages.
First check delivery with curl
curl -i https://example.com/robots.txtcurl -IL --max-redirs 5 https://example.com/robots.txtcurl -sS https://example.com/robots.txt | wc -ccurl -sSI https://example.com/robots.txt | grep -i content-typeThen evaluate user-agents and representative URLs
- The home page, important categories, articles, products, CSS, JavaScript and images are allowed
- Only intended search, filter and duplicate URLs are disallowed
- Tests include differences in case, trailing slashes, queries and encoding
- Check Googlebot, image and video crawlers, and AI user-agents by purpose
- Rules do not conflict with noindex targets, sitemaps, canonical URLs or internal links
Check the fetched version in the current Search Console robots.txt report
The report shows files Google found on the main hosts, last fetch times, sizes, warnings, errors and fetched versions from the past 30 days. You can request a recrawl after an important correction, but this does not guarantee an immediate recrawl of the URLs themselves.
Fetch status of the published file
Review the latest fetched version, problem lines, previous versions and recrawl requests. Use URL Inspection to check whether a specific URL is blocked.
Google's public robots.txt library
Use the open-source implementation also used by Google Search for automated tests across many URLs and regression checks on changes.
Manage the file, cache and search controls together during release and recovery
Restoring robots.txt alone may leave CDN caches and simultaneous noindex, X-Robots-Tag, canonical, sitemap or internal-link changes in place. Record related controls and recovery order in the change ticket.
Save the last good version before release
Save the body, response headers, hash, fetch time, URL decision matrix and Search Console's fetched version.
Monitor stop conditions
Detect important URLs being disallowed, global blocks, 4xx and 5xx responses, HTML responses and redirects to unexpected hosts.
Restore the previous version at the authoritative layer
Restore the canonical file instead of layering emergency rules elsewhere, and invalidate only the affected CDN cache.
Restore conflicting SEO controls too
Return noindex, X-Robots-Tag, canonical, sitemaps and internal links to the same approved version.
Confirm refetching and URL decisions
Confirm recovery using curl, the URL decision matrix, Search Console and server logs; record the cause and prevention measures.
Robots.txt change completion checklist
Purpose and scope
- ✓Separated crawl management, noindex and confidentiality
- ✓Listed protocols, hosts, subdomains and ports
- ✓Checked official product tokens for target crawlers
- ✓Recorded expected results for representative URLs
Rules
- ✓Checked user-agent groups and longest matches
- ✓Tested case, trailing characters, queries and encoding
- ✓Checked CSS, JavaScript, images, PDFs and sitemaps
- ✓No global block, exposed sensitive paths or unsupported extensions
Delivery and verification
- ✓/robots.txt returns 200 and UTF-8
- ✓No unexpected redirect, WAF, authentication or CDN differences
- ✓Checked Search Console's fetched version and logs
- ✓Added regression tests for important URLs to CI
Release and recovery
- ✓Saved the current and proposed versions, hashes and release time
- ✓Set stop conditions, owner and contacts
- ✓Tested restoration and targeted cache invalidation
- ✓Checked consistency with noindex, canonical URLs and sitemaps
Robots.txt frequently asked questions
Does Disallow in robots.txt remove a URL from search results?
Not necessarily. If a URL is discovered through external links or other means, it may appear in search results without its content being fetched. To remove it, serve noindex or X-Robots-Tag where a crawler can fetch the directive.
Does disallowing an admin page make it secure?
No. Robots.txt is public and some clients do not obey it. Protect administrative screens and confidential data with authentication, authorization, multi-factor authentication, network restrictions and audit logs.
Does Allow take precedence when written after Disallow?
Not simply by order. RFC 9309 uses the most specific matching rule and recommends Allow for equivalent Allow and Disallow rules. Google selects the least restrictive rule when rules conflict.
What happens when Disallow is empty?
That rule imposes no crawl restriction. Disallow: / blocks the entire site. An empty value and a single slash have opposite meanings, so always check the difference before release.
Can crawl-delay slow Googlebot?
Google does not support crawl-delay in robots.txt. Address load through a combination of URL-space cleanup, caching, server performance, error correction, supported crawl settings and rate controls where needed.
Can AI crawlers be controlled individually?
You can target compliant providers using their published user-agent tokens. Google-Extended, OAI-SearchBot and GPTBot have different purposes, so do not treat search appearance and training use as the same setting; check each provider's latest documentation.
Where is the old Search Console robots.txt tester?
Use the current robots.txt report to inspect fetched files, warnings, errors and previous versions, and URL Inspection to check whether an individual URL is blocked. Google's public robots.txt library can also automate development checks.
Is there a maximum robots.txt size?
Google limits processing to 500 KiB and ignores the remainder. Instead of expanding an enormous rule list, organize URL design and directories so fewer rules are needed.
Primary and standards sources checked for this article
The recorded check date for the main sources is July 25, 2026. We rechecked Google's handling of 5xx responses and parts of the RFC on November 8, 2026. Consult each provider's current documentation for AI crawlers and other details.
- RFC 9309: Robots Exclusion Protocol
- Google: how robots.txt is interpreted
- Google: create, configure and submit robots.txt
- Google Search Central: an introduction to robots.txt and its limitations
- Search Console: robots.txt report
- Google: common crawlers and Google-Extended usage controls
- OpenAI: OAI-SearchBot, GPTBot and ChatGPT-User
- Google Search Central: robots meta and X-Robots-Tag