ServerField Server operations and website growth
Technical SEO guideIncludes configuration examplesUpdated November 2026

What is robots.txt?
Rules and SEO impact
Examples and verification

Scope, User-agent, Allow and Disallow, noindex, HTTP errors, Search Console and AI crawlers: follow a decision process that prevents accidental blocks instead of copying a configuration blindly.

FFWritten and checked by: Finite Field Editorial TeamPublished: January 8, 2025Last updated: November 8, 2026
A diagram of controlling crawler access with robots.txt A robots.txt file uses User-agent, Allow and Disallow rules to control search crawlers, AI search and training crawlers according to their purposes. ROBOTS EXCLUSION PROTOCOL A public file that indicates which URLs may be fetched https://example.com/robots.txt User-agent:* Disallow:/search/ Allow:/search/help/ User-agent:GPTBot Disallow:/ Sitemap:.../sitemap.xml Crawl management Specifies allowed URLs Noindex is separate Signals removal from search Not for protecting secrets Authenticate and authorize Verify individual URLs

Robots.txt is a public file anyone can read. Use it to communicate fetching rules to compliant crawlers, not to hide sensitive paths.

July 29, 2026 update: We compared RFC 9309, Google's current robots.txt specification and Search Console report, Google-Extended, and OpenAI's OAI-SearchBot and GPTBot, then added settings for AI search and training alongside conventional SEO.

The short answer: robots.txt is a public set of rules telling crawlers whether they may fetch URLs. It does not remove pages from search results or protect admin screens and confidential files. The safest approach is to distinguish the purpose and create a decision matrix for each origin and URL.

The answer in 30 seconds

Use robots.txt for crawling and separate tools for search exclusion and access protection

Robots.txt can reduce crawling of unnecessary search-result pages and duplicate URLs. Use noindex to remove pages from search results and authentication and authorization to protect secrets. Before and after release, verify the file's HTTP response and representative URL decisions separately.

Managing crawlingControl crawling of unwanted URLs and manage specific crawlers
Not suitable forRemoving search results, protecting secrets or preventing unauthorized access
Required checks before releaseOrigins, HTTP responses, user-agents, representative URLs and recovery version
What is robots.txt?Public text telling compliant crawlers whether they may fetch URLs on the same protocol, host and port
LocationThe lowercase file at each origin's root: /robots.txt
Core directivesUser-agent, Allow, Disallow, with the additional record Sitemap
SEO considerationsA disallowed URL can remain in search results. Do not block URLs where noindex must be read.
Key Google-specific limitsProcesses up to 500 KiB, usually caches for up to about 24 hours and follows at least five redirects
Recorded check date for main sourcesJuly 25, 2026

What is robots.txt? Understand its role and limits first

robots.txt tells automated clients that follow the Robots Exclusion Protocol which URLs they may fetch. It can adjust the crawl scope of compliant crawlers, but does not grant or revoke access permissions.

Search engines fetch a URL before understanding a page. Robots.txt manages this entry point, so its role differs from noindex in a page's robots meta tag or HTTP header.

Robots.txt is most useful for large, low-value URL spaces such as site search, filters, session URLs and nearly infinite calendars. It does not solve URL-design problems alone: also review internal links, canonical URLs, sitemaps, parameters and server performance.

Robots.txt is a request, not a security boundary.
Use authentication, authorization, WAF, network restrictions and rate controls to stop malicious clients or unsupported crawlers.

Do not confuse crawl control, search exclusion and access protection

Before configuring rules, classify URLs by whether you want to reduce fetching, remove them from search results or stop public access. Writing Disallow without this distinction can prevent noindex from being read, block important pages or expose sensitive paths.

PurposeMain approachRole of robots.txtWhat to check
Reduce crawling of unnecessary URLsRobots.txt, URL design and internal-link cleanupSuitableWhether important pages or rendering resources are affected
Remove a page from search resultsRobots meta noindex, X-Robots-Tag, removal or 404/410Do not block crawlingWhether crawlers can fetch noindex
Protect admin screens and customer dataAuthentication, authorization, MFA and network restrictionsDoes not provide protectionWhether anonymous access is actually denied
Temporarily hide a result in an emergencyTake content offline, noindex, Search Console removal tools, and similar measuresNot sufficient on its ownLong-term controls and recrawl requirements
The right approach

Allow crawling where noindex must be read

Serve a robots meta directive for HTML or X-Robots-Tag for PDFs and similar files, and allow crawlers to fetch it. A recrawl is needed for the change to take effect.

A dangerous assumption

Disallow hides secrets and removes search results

Robots.txt is public. A URL discovered through an external link may appear in search results without its content being fetched.

Scope is determined by protocol, host and port

Robots.txt is fetched from /robots.txt at the root of its origin. An HTTPS file is not automatically inherited by HTTP, another subdomain or a nonstandard port. An omitted standard port is equivalent, while nonstandard ports are managed separately.

Robots.txt scopeThe HTTPS example.com robots.txt applies only to the same HTTPS host and standard port, not HTTP, the www subdomain or another port. https://example.com/robots.txt Applies only to the same protocol, host and port ✓ Covered https://example.com/products/1 https://example.com:443/help/ ! Not covered http://example.com/ https://www.example.com/ https://shop.example.com/ https://example.com:8181/ https://example.com/help/robots.txt Operational rule List HTTP and HTTPS, www, subdomains, CDNs, image hosts and nonstandard ports #section is not included in HTTP requests
Robots.txt scope is not determined by domain name alone. List every public origin and check what each one serves.
Robots.txt URLApplies toDoes not apply toOperational action
https://example.com/robots.txthttps://example.com/pagehttp://example.com/
https://www.example.com/
Also check HTTP and www if they are public
https://shop.example.com/robots.txthttps://shop.example.com/carthttps://example.com/
https://cdn.example.com/
Manage an owner and rules for each subdomain
https://example.com:8181/robots.txthttps://example.com:8181/admin/https://example.com/ (443)First review whether public access is needed
https://example.com/help/robots.txtNot a valid locationThe entire siteMove it to /robots.txt at the root

Test case, trailing slashes, encoding and query strings against real URLs.
/File/ and /file/ are different paths. A URL fragment such as #section is not sent to the server, so robots.txt cannot distinguish it.

Determine the user-agent group and most specific path rule

The standard centers on User-agent, Allow, Disallow. Sitemap is an additional record supported by major search engines. Field names are case-insensitive, but path values are case-sensitive.

UA

User-agent

Specify the target crawler's product token. * applies to crawlers without a more specific matching group.

−

Disallow

This rule disallows fetching matching URLs. An empty value imposes no restriction; / matches the entire site.

+

Allow

Allow a more specific URL within a broad Disallow. A rule does not win merely because it appears later.

XML

Sitemap

List the full sitemap URL. It is not an Allow or Disallow rule within a user-agent group.

*

Wildcards

For Google, * represents zero or more characters, and $ marks the end of the URL. Check the official specification for other crawlers.

#

Comments

# and the rest of the line are ignored. You may include reasons and change-ticket numbers, but not secrets or internal architecture.

Robots.txt: a basic example
User-agent: *
Disallow: /search/
Allow: /search/help/

Sitemap: https://example.com/sitemap.xml

Choose the matching group, then the longest matching rule

1. Choose a user-agent groupSelect the group most specific to the crawler. Google combines multiple groups with the same specific token.
2. Match rules to the URLFind Allow and Disallow rules matching the path and any query portion handled by the implementation.
3. Use the longest matchGoogle selects the least restrictive rule when rules conflict. RFC 9309 recommends Allow when Allow and Disallow are equivalent.
For example, if Disallow: /catalog/ and Allow: /catalog/help/ are present, /catalog/item/1 is disallowed, while /catalog/help/size is allowed by the more specific Allow rule.
500KiBGoogle's robots.txt processing limit
About 24 hoursGoogle's usual cache duration
At least fiveRedirects Google follows

Write robots.txt safely: configuration examples

Do not copy examples unchanged. Check the target origin, real URLs, search value, noindex and rendering resources. Fewer rules are easier to review and faster to restore after an incident.

01

Allow all crawlers and advertise the sitemap

For a small site with no URLs to block. An empty Disallow also means no restriction, but Allow: / can make the intent easier to read.

robots.txt
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml
02

Block site search but allow help pages

For a site that generates many search-result URLs. If help pages share the directory, create an exception with a more specific Allow.

robots.txt
User-agent: *
Disallow: /search/
Allow: /search/help/

Sitemap: https://example.com/sitemap.xml
03

Restrict a specific query pattern

A Google-specific example for cases where sorting or tracking parameters produce an almost infinite URL space. Test representative URLs to avoid blocking necessary landing pages.

robots.txt
User-agent: Googlebot
Disallow: /*?sort=
Disallow: /*&sort=
Disallow: /*?session=
In production, Disallow: / is a stop condition. Automatically detect staging files, environment-variable errors, CDN cache issues and wrong deployment targets. Stop the release if important URLs are blocked site-wide.

Manage AI search and training crawlers by purpose in 2026

AI-related user-agents are not all the same. Crawlers used for search results, crawlers used for foundation-model training and agents fetching pages at a user's request have different purposes and may treat robots.txt differently.

User-agentMain purposeWhat a control changesConsiderations
Google-ExtendedGemini model training and grounding for Gemini and Vertex AIControls whether Google-crawled site content is used for these purposesA usage-control token, not a separate HTTP crawler; it does not affect Google Search inclusion or ranking
OAI-SearchBotDisplay websites in ChatGPT searchAllowing it preserves the possibility of appearing in ChatGPT search resultsIndependent of GPTBot; changes may take time to take effect
GPTBotCrawl content that may be used to train generative AI foundation modelsUse Disallow to indicate that training use is not desiredYou can allow OAI-SearchBot while blocking GPTBot
ChatGPT-UserA user asks ChatGPT or a custom GPT to fetch a pageNot used to control automated search crawlingRobots.txt may not apply because the request is user-initiated
Example: allow AI search while controlling training separately
# Preserve opportunities to appear in ChatGPT search
User-agent: OAI-SearchBot
Allow: /

# Indicate that content should not be used for model training
User-agent: GPTBot
Disallow: /

# Control Gemini-related use without affecting Google Search
User-agent: Google-Extended
Disallow: /
Operational note: User-agent names, purposes, IP ranges and robots.txt behavior can change. Do not block guessed log strings; recheck each provider's official documentation when making changes.

Use robots.txt and noindex for their distinct SEO purposes

Robots.txt communicates whether a URL may be fetched; noindex communicates whether it may remain in a search index. Since noindex appears in the page or response headers, a crawler must be able to access the URL to read it.

Comparisonrobots.txtnoindex / X-Robots-TagAuthentication and authorization
Main purposeManage crawl scopeExclude from search resultsControl access to content
LocationThe origin's /robots.txtHTML meta tag or HTTP headerApplication, server or identity system
Crawler accessPrevent fetching the target URLFetching is needed to read the directiveAccess is denied without permission
Can it protect secrets?NoNoYes, when implemented correctly
Can the URL remain in search results?YesExcluded after a recrawlAddress separately based on public availability and the existing index

How to remove a URL already in search results

  1. Remove the robots.txt block

    Allow the crawler to fetch the page or response header.

  2. Serve noindex or X-Robots-Tag

    Choose a method appropriate for HTML, PDF, images or the relevant format.

  3. Check the recrawl with URL Inspection and logs

    Check that the live response matches what Google fetched.

  4. Choose a permanent state

    Keep noindex if the content remains public, remove it with 404 or 410 if no longer needed, or require authentication if it is confidential.

Treat dangerous settings as release stop conditions

Correct syntax can still cause an SEO incident if target URLs or objectives are wrong. Detect the following through change review and automated checks.

/

Disallow: / in production

This blocks the entire site. Prioritize detecting staging settings and CDN cache contamination.

NO

Blocking URLs that need noindex

Crawlers cannot fetch noindex or X-Robots-Tag, which can prevent removal from search results.

Listing sensitive paths

/backup/ and administrative URLs are exposed. Use authentication and authorization for access control.

CSS

Blocking all CSS, JavaScript and images

Blocking resources needed for rendering and page understanding can prevent search engines from evaluating content properly.

403

Using 401 or 403 as a block

Google treats 4xx responses other than 429 as if no valid robots.txt exists, assuming no crawl restrictions.

↪

Long redirect chains and HTML responses

Check the body and Content-Type for login screens, WAF pages, 404 HTML or language redirects.

500K

An oversized file

Google ignores content beyond 500 KiB. Organize URL design and consolidate rules.

Relying on crawl-delay

Google does not support crawl-delay. Diagnose load using logs, URL space, performance and errors.

?

Guessed crawler names

Do not block groups of similar user-agent strings by guesswork. Use official product tokens and IP verification methods.

Monitor HTTP responses, cache and file format as closely as rules

The same body can be interpreted differently depending on status code, Content-Type, encoding, redirects and cache. Record Google's behavior separately from RFC 9309.

ResponseGoogle's main behaviorOperational checks
2xxParse the fetched rulesCheck 200, UTF-8, plain text, the body and Cache-Control
3xxFollow at least five redirects, then treat the file as a 404Serve a direct 200 where possible; redirects should lead to the intended /robots.txt
4xx except 429Assume no crawl restrictions as if no valid robots.txt existsDo not use 401 or 403 to stop crawling; distinguish an absent file from a delivery failure
429Handled differently from ordinary 4xx; investigate as a rate-limit or availability issueCheck excessive restrictions, WAF, CDN, origin load and retries
5xx and network failuresStop crawling for the first 12 hours. Then use the last good version for up to about 30 days while retrying; if no good version is cached, assume no crawl restrictionsMonitor the last good version, recovery time, DNS, TLS, timeouts and server logs
UTF-8Google expects UTF-8 plain text when parsing.
Usually up to 24 hoursRobots.txt is cached, so corrections may not take effect immediately.
HTML may still be parsedGoogle may extract valid lines from an HTML response, but correct the delivery rather than allowing an incident to remain hidden.

Create a URL decision matrix and deploy minimal rules

Before writing robots.txt, list target URLs and crawlers, expected results, alternative controls and verification methods. Make important URLs and rendering resources mandatory allowed test cases.

  1. List the origins in scope

    List HTTP, HTTPS, www, subdomains, CDNs, image hosts and nonstandard ports, and assign an owner for each /robots.txt.

  2. Classify representative URLs as allowed, disallowed or separately controlled

    Include the home page, important subpages, search and filter URLs, admin screens, APIs, CSS, JavaScript, images, PDFs, sitemaps and nonexistent URLs.

  3. Choose the right mechanism

    Separate robots.txt for crawl management, noindex for search exclusion and authentication for secrets.

  4. Write the minimum groups and rules

    Avoid duplicate groups and overly broad wildcards; keep only necessary Disallow rules and Allow exceptions.

  5. Automate URL decisions locally and in staging

    Test user-agents, trailing slashes, case, queries and encoding.

  6. Choose the authoritative delivery layer and cache behavior

    Decide whether the application, web server or CDN owns the canonical file, and prevent environment-specific files from being mixed up.

  7. Prepare the previous version, stop conditions and owner before release

    Be ready to restore the previous version if a global block, important URL block, HTML response or 5xx is detected.

Verify file fetching and URL decisions separately

/robots.txt opening in a browser does not prove that the crawler and URL decisions are correct. Check the HTTP response, syntax, URL decisions, Search Console and logs in stages.

First check delivery with curl

Response headers and bodycurl -i https://example.com/robots.txt
Redirect chaincurl -IL --max-redirs 5 https://example.com/robots.txt
File sizecurl -sS https://example.com/robots.txt | wc -c
Check Content-Type onlycurl -sSI https://example.com/robots.txt | grep -i content-type

Then evaluate user-agents and representative URLs

  • The home page, important categories, articles, products, CSS, JavaScript and images are allowed
  • Only intended search, filter and duplicate URLs are disallowed
  • Tests include differences in case, trailing slashes, queries and encoding
  • Check Googlebot, image and video crawlers, and AI user-agents by purpose
  • Rules do not conflict with noindex targets, sitemaps, canonical URLs or internal links

Check the fetched version in the current Search Console robots.txt report

The report shows files Google found on the main hosts, last fetch times, sizes, warnings, errors and fetched versions from the past 30 days. You can request a recrawl after an important correction, but this does not guarantee an immediate recrawl of the URLs themselves.

Search Console

Fetch status of the published file

Review the latest fetched version, problem lines, previous versions and recrawl requests. Use URL Inspection to check whether a specific URL is blocked.

Development and CI

Google's public robots.txt library

Use the open-source implementation also used by Google Search for automated tests across many URLs and regression checks on changes.

The Search Console report may differ from the currently served body. Account for caching, failed fetches and CDN differences, and compare the fetch time with server logs.

Manage the file, cache and search controls together during release and recovery

Restoring robots.txt alone may leave CDN caches and simultaneous noindex, X-Robots-Tag, canonical, sitemap or internal-link changes in place. Record related controls and recovery order in the change ticket.

01

Save the last good version before release

Save the body, response headers, hash, fetch time, URL decision matrix and Search Console's fetched version.

02

Monitor stop conditions

Detect important URLs being disallowed, global blocks, 4xx and 5xx responses, HTML responses and redirects to unexpected hosts.

03

Restore the previous version at the authoritative layer

Restore the canonical file instead of layering emergency rules elsewhere, and invalidate only the affected CDN cache.

04

Restore conflicting SEO controls too

Return noindex, X-Robots-Tag, canonical, sitemaps and internal links to the same approved version.

05

Confirm refetching and URL decisions

Confirm recovery using curl, the URL decision matrix, Search Console and server logs; record the cause and prevention measures.

Robots.txt change completion checklist

Purpose and scope

  • ✓Separated crawl management, noindex and confidentiality
  • ✓Listed protocols, hosts, subdomains and ports
  • ✓Checked official product tokens for target crawlers
  • ✓Recorded expected results for representative URLs

Rules

  • ✓Checked user-agent groups and longest matches
  • ✓Tested case, trailing characters, queries and encoding
  • ✓Checked CSS, JavaScript, images, PDFs and sitemaps
  • ✓No global block, exposed sensitive paths or unsupported extensions

Delivery and verification

  • ✓/robots.txt returns 200 and UTF-8
  • ✓No unexpected redirect, WAF, authentication or CDN differences
  • ✓Checked Search Console's fetched version and logs
  • ✓Added regression tests for important URLs to CI

Release and recovery

  • ✓Saved the current and proposed versions, hashes and release time
  • ✓Set stop conditions, owner and contacts
  • ✓Tested restoration and targeted cache invalidation
  • ✓Checked consistency with noindex, canonical URLs and sitemaps

Robots.txt frequently asked questions

Does Disallow in robots.txt remove a URL from search results?

Not necessarily. If a URL is discovered through external links or other means, it may appear in search results without its content being fetched. To remove it, serve noindex or X-Robots-Tag where a crawler can fetch the directive.

Does disallowing an admin page make it secure?

No. Robots.txt is public and some clients do not obey it. Protect administrative screens and confidential data with authentication, authorization, multi-factor authentication, network restrictions and audit logs.

Does Allow take precedence when written after Disallow?

Not simply by order. RFC 9309 uses the most specific matching rule and recommends Allow for equivalent Allow and Disallow rules. Google selects the least restrictive rule when rules conflict.

What happens when Disallow is empty?

That rule imposes no crawl restriction. Disallow: / blocks the entire site. An empty value and a single slash have opposite meanings, so always check the difference before release.

Can crawl-delay slow Googlebot?

Google does not support crawl-delay in robots.txt. Address load through a combination of URL-space cleanup, caching, server performance, error correction, supported crawl settings and rate controls where needed.

Can AI crawlers be controlled individually?

You can target compliant providers using their published user-agent tokens. Google-Extended, OAI-SearchBot and GPTBot have different purposes, so do not treat search appearance and training use as the same setting; check each provider's latest documentation.

Where is the old Search Console robots.txt tester?

Use the current robots.txt report to inspect fetched files, warnings, errors and previous versions, and URL Inspection to check whether an individual URL is blocked. Google's public robots.txt library can also automate development checks.

Is there a maximum robots.txt size?

Google limits processing to 500 KiB and ignores the remainder. Instead of expanding an enormous rule list, organize URL design and directories so fewer rules are needed.

Primary and standards sources checked for this article

FF

Finite Field Editorial TeamWe compare RFC 9309, Google's robots.txt implementation, the Search Console report, the robots meta specification and official AI crawler documentation, distinguishing the standard, search-engine-specific behavior and security controls.

Website Production

Ask Finite Field
to build your website

Discuss more than SEO article rewrites: we can bring together information architecture, UI design, speed, multilingual support, WordPress or Go implementation, and post-launch improvements.

Contact usStarting with requirementsWordPress and Go
  • Page structures that support search intent and user journeys
  • Design for crawling, indexing and structured data
  • Check CSS, image paths and responsive display
Contact

Tell us about your current pages and challenges

Even if your redesign direction is undecided, we can start by defining priorities and the scope of improvements.

Start a conversation about your website.