← The Agentic Web · Stage 1: Agent Discoverable

File Deep Dive

robots.txt

From search crawlers to AI agents — the small file at your front door has a much bigger job than it used to. Here is what it does, and how we evaluate yours.

What It Is

A public note at your front door — not a lock on it.

robots.txt is a small, publicly readable text file that lives at the root of your website — yourdomain.com/robots.txt. It tells automated visitors — robots, bots, crawlers — which parts of your site they are asked to visit or avoid.

The idea dates to 1994, when nearly every crawler worked for a search engine and site owners simply needed a way to steer them away from unimportant or resource-hungry corners of their sites. It worked so well that it became an official Internet standard in 2022 — RFC 9309.

One thing it has never been: a security system. robots.txt is a published request that responsible crawlers honor — not a lock that physically prevents access. It cannot keep confidential information private, guarantee a page stays out of an index, or stop a malicious bot from visiting a public URL (Google’s robots.txt guidance). Anything truly sensitive still needs real protection — logins, authorization, access controls.

Why It Matters Now

One file. Many different visitors. Very different intentions.

The bots reading your site are no longer just search engines. A crawler arriving today may be gathering information for any of these purposes:

Traditional search indexingSearch-result summaries and AI-generated answersUser-directed AI retrievalAI model trainingCommercial data collectionWebsite monitoring and analysisAutomated transactions and other agent-assisted activities

These purposes are not interchangeable

You might want search engines indexing your public pages and AI assistants answering customer questions — while declining to hand the same content to model training or unrestricted commercial scraping. Crawler operators are starting to respect that distinction. OpenAI, for example, runs separate crawlers for ChatGPT search (OAI-SearchBot), potential model training (GPTBot), and user-requested actions (ChatGPT-User) — so you can say yes to one and no to another (OpenAI crawler documentation).

A commercial layer is emerging

Cloudflare’s AI Crawl Control helps owners identify AI crawlers, monitor whether they follow robots.txt, and allow or block them. Its Pay Per Crawl feature (currently in closed beta) adds a third option: charge an AI company each time its crawler accesses your content. Your content no longer has to be either free or invisible — some systems get access, some are blocked, and some may pay. It doesn’t guarantee buyers will emerge or set your content’s price, but it lays an early foundation for licensing and monetizing automated access to your digital assets.

Here is the challenge: most robots.txt files were written for the search-engine era. They may technically work — while saying almost nothing about what the owner wants today.

A Modern Example

What a deliberate robots.txt looks like.

This file tells one consistent story: public pages are open to everyone, operational areas are closed, AI search and assistants are welcomed by name, and model training is politely declined.

# --- Everyone: public pages open, private areas closed ---
User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /account/
Disallow: /search

# --- AI search & assistants: welcome ---
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

# --- AI model training: declined ---
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

# --- Usage policy, stated in machine-readable form ---
Content-Signal: search=yes, ai-input=yes, ai-train=no

# --- The map of everything worth finding ---
Sitemap: https://www.example.com/sitemap.xml

Reading it section by section

  • Everyone (User-agent: *) — all bots may read public pages, but admin, cart, account, and internal-search areas are off-limits. Closing operational corners keeps crawlers focused on the pages that matter.
  • AI search & assistants — retrieval crawlers like OAI-SearchBot, ChatGPT-User, and PerplexityBot are explicitly welcomed. These are the systems answering your buyers’ questions.
  • AI model training — GPTBot, Google-Extended, and CCBot are declined. Same content, different purpose, different answer. That is intent clarity.
  • Content-Signal — the same policy stated as a machine-readable summary: search yes, AI answers yes, AI training no. It matches the rules above it — consistency agents can verify.
  • Sitemap — points every welcome crawler to the complete map of pages you want found.

The Bigger Picture

The first layer of your agent policy.

AI agents are the next stage of this story. They are becoming capable of finding information, comparing options, interpreting pages, interacting with websites, and completing permitted tasks on a buyer’s behalf (Google’s agentic website guidance). That raises a question every website owner will eventually have to answer:

Which automated systems should be able to access which information, for what purpose, and under what conditions?

robots.txt cannot answer or enforce all of that by itself. What it can do is provide the clear starting point: a deliberate robots.txt policy translates your intentions into an initial, machine-readable set of instructions. That policy then becomes the blueprint for stronger controls:

Authentication and authorizationFirewalls and bot-management systemsRate limits and server rulesPaywalls and membership controlsLicensing and contractual restrictionsMonitoring, logging, and enforcement

Seen this way, robots.txt is more than a technical file. It is the first layer of a framework for deciding how your website interacts with search engines, AI systems, and autonomous agents. It will not protect intellectual property on its own — and it should never be sold as if it does. Its value is making your intentions explicit, testable, and ready to be translated into stronger infrastructure as the automated web develops.

A well-constructed robots.txt file is not the finish line — it is a low-cost first step toward understanding, governing, and building an agent-ready website.

How We Score It

Five categories. One honest conclusion.

We evaluate a robots.txt file across five categories. Validity is pass or fail; the rest are scored 1–10, with N/A where a fair score would require knowing the owner’s intent. The free Website Analyzer runs all five categories automatically from your file’s public evidence — honestly marking N/A wherever a fair score would need your input.

  1. 1

    Validity

    Pass / Fail

    Before anything else: does the file actually work?

    • Can the file be found and read?
    • Is it correctly formatted?
    • Will compliant bots interpret it properly?

    Ground rule: A file that fails validity cannot receive an overall passing assessment.

  2. 2

    Completeness

    1–10

    Does the file cover everything it needs to — and nothing it doesn't?

    • Are any rules needed to achieve the owner's goals missing?
    • Are all relevant website hostnames considered?
    • Are important public and operational areas addressed where necessary?
    • Are the appropriate XML, RSS, image, video, or other sitemaps declared and accessible?

    Ground rule: No points deducted merely because specific crawlers are not named — a simple wildcard policy may be complete.

  3. 3

    Intent Clarity

    1–10 or N/A

    Does the file say what the owner actually means?

    • Does the file clearly reflect what the owner wants bots to access or avoid?
    • Do the Allow and Disallow rules actually produce the intended outcome?
    • Does the policy treat search, AI retrieval, AI training, and monitoring differently where the owner wants different treatment?
    • Does the file deliberately allow or decline model-training crawlers — or is the owner's preference still unknown?
    • Does the policy agree with sitemaps, canonical URLs, noindex, authentication, and CDN or bot-management settings?

    Ground rule: Scored N/A when owner intent is unknown. We never guess what the owner wants.

  4. 4

    Search-Engine Friendliness

    1–10

    Can search engines still do their job well?

    • Can search engines crawl the pages the owner wants people to find?
    • Can crawlers access the images, scripts, and stylesheets needed to understand public pages?
    • Are any important public pages or resources blocked unintentionally?
    • Are bots wasting resources on duplicate, internal, parameterized, or low-value URLs?
    • Are administrative, account, checkout, preview, staging, and internal-search areas handled appropriately?
  5. 5

    AI Access & Readiness

    1–10 or N/A

    Can the AI systems your buyers use actually reach you?

    • Can verified AI search and retrieval systems access the public pages needed to answer questions about the business, its services, products, and people?
    • Can relevant systems reach useful discovery resources — XML sitemaps, RSS feeds, public documentation, structured content, intentionally public APIs?
    • Are AI-related rules clear enough that retrieval crawlers are not accidentally blocked or treated the same as training crawlers?

    Ground rule: We score actual access, not the number of AI crawlers named.

    Ground rule: Redundant Allow: / rules earn nothing.

    Ground rule: llms.txt and special AI wording are not assumed to improve visibility — machine-readable resources are supporting evidence, not guaranteed AI visibility.

Score colors

8–10 · Green

Good. No significant changes needed.

5–7 · Amber

Functional, but meaningful improvements are available.

1–4 · Red

Material problems or harmful rules.

N/A · Gray

Cannot evaluate without evidence or owner input.

The overall conclusion

After the five categories, the evaluation ends in one plain-language verdict:

  • Sufficient as written
  • Sufficient, with minor improvements recommended
  • Not sufficient for the owner's stated goals
  • Cannot determine without owner input

The conclusion answers one question: would changing the file create a real behavioral improvement — not merely make it longer or more explicit?

What does your robots.txt say about you?

Run the free analyzer for an instant read on your file’s public signals — the first step of the full evaluation.

Run the Website AnalyzerBack to the Journey